AMD Radeon AI PRO R9700 32GB reports

generation:

70.0 tokens/s

quant:

MXFP4-FP8-GPTQ (GPTQ)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at around 70 t/s for a single session on 4x Radeon AI PRO R9700 32GB. Setup is vLLM with MXFP4-FP8-GPTQ weights, fp8 KV cache, tensor-parallel-size 4, 262144 max model length, and MTP speculative decoding with 3 tokens. The user is troubleshooting a decode performance cap hit with newer vLLM image tags starting from 29.06.19, and notes the BetterBench report incorrectly labels the GPU as RTX 4000 Ada.

generation:

36.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at ~36 tok/s single stream on a single Radeon AI PRO R9700 32GB with 64GB system RAM. Setup is llama.cpp with stew675's RDNA4 patches (r30), AtomicChat Q4_K_M quant, q8_0 KV cache, flash attention, 2 slots x 131k context, -ncmoe 41 and MOE_EXPERT_CACHE_MIB=4096, GPU capped at 220W. Two streams combined reach ~50 tok/s and prompt reading is 526-589 tok/s. The 27B Q5_K_XL is ~1.7x faster per turn and reads prompts 1.6-2x as fast, but scores lower on tool-eval-bench Hard Mode (79.8 vs 93.9).

generation:

57.3 tokens/s

prompt processing (prefill):

1166.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User compares R9V and Strata running Qwen3.8 Flash-Next UD-IQ4_XS on a dual Radeon AI PRO R9700 system with 96GB DDR4 and an AMD 3900X. R9V is 15-18% faster on prefill at every context size, while decode is roughly level between the two engines. Setup is R9V v0.4.4 with UD-IQ4_XS GGUF shards and MTP, single-session requests at 16k, 32k and 64k context with 256 output tokens. Strata v0.1.39 ran on a single GPU because the two-GPU split needed about 118 GB of RAM; its expert cache held 10,774 experts in VRAM with a 98.7% decode cache hit rate. R9V measured 1166 tok/s prefill and 57.3 tok/s decode at 16k, 1156/58.8 at 32k, and 1109/53.8 at 64k. Strata measured 980/53.3 at 16k, 1003/53.8 at 32k, and 986/54.6 at 64k. The host has 92 GiB of RAM against the R9V README's 128 GiB, and the README's claimed 1.6-1.7k prefill was not reached.

generation:

244.5 tokens/s

prompt processing (prefill):

10858.0 tokens/s

quant:

MXFP4 (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Ornith-1.5-35B-A3B at 244.5 tok/s decode on an AMD Radeon AI PRO R9700 32GB, compared against llama.cpp b11214 at 118.8 tok/s. Setup is WHIRL v0.1.0 with an MXFP4 GGUF quant (4.46 bpw, 18.45 GiB loaded) and F16 KV cache; the R9700 is connected as a USB4 eGPU. The 244.5 tok/s figure uses MTP + n-gram speculative decoding on 7 Chinese coding prompts (800 tokens, server); plain decoding without MTP gives 166.7 tok/s. Prefill at 8,192 tokens is 10,858 tok/s versus 4,637 tok/s in llama.cpp. Four concurrent requests aggregate 396.7 tok/s versus 191.9 tok/s. Perplexity on a mixed Chinese/English code corpus is 5.5192 ± 0.0653 versus 5.3206 ± 0.0619 for the official Q4_K_M, 3.7% higher.

generation:

179.7 tokens/s

quant:

W3A4 (safetensors)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 179.7 tok/s single-stream decode on a single AMD Radeon AI PRO R9700 32GB. Setup is stock vLLM with a plugin, 3-bit rotated GPTQ weights (W3A4) with FP8 KV cache, 262,144-token context, speculative decoding, and a 569,878-token prefix cache. Aggregate throughput is 488.5 tok/s at 8 concurrent requests. Accuracy: GSM8K 95.22, HumanEval 94.51, MMLU-Pro 60.57. The 3-bit weights are a 9.55 GB add-on; MXFP4 remains available at 156.1 tok/s single-stream.

generation:

21.0 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 21 t/s on llama.cpp with IQ3_XXS, running on system RAM plus a Radeon AI PRO R9700 32GB. Setup is llama.cpp with IQ3_XXS, model partially offloaded to system RAM (64GB DDR5). User also reports ~60 t/s with the same quant on Strata, and 35 t/s for Qwen3.8-27B Q6 on the R9700. System RAM usage reaches 96% with Flash-Next loaded, preventing simultaneous ComfyUI workloads.

generation:

37.5 tokens/s

quant:

MXFP4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B fine-tune Swift-1.5 at 37.5 t/s decode without speculation on a Radeon AI PRO R9700 32GB. Setup is WHIRL with MXFP4 GGUF weights, one GPU, Windows 11, Adrenalin 26.8.1. With MTP and n-gram speculation the same model reaches 107.5 t/s on Chinese coding prompts and 328 t/s on file-edit prompts; prefill is 3,278 t/s at 8k and 2,595 t/s at 32k. The server serves 199 t/s aggregate across 4 concurrent users. User notes the R9700 is connected as a USB4 eGPU, which slows model loading and cache restores but not prefill or decode. Plain decode on the dense models is only 1.1-1.18x faster than llama.cpp; most of the decode gap comes from MTP and n-gram speculation.

generation:

74-143 tokens/s

quant:

FP8 (FP8)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at FP8 running tensor-parallel across two Radeon AI PRO R9700s, with roughly 1.6k tokens/s prefill and 74-143 tokens/s decode single stream depending on content. Setup is vLLM 0.27.1 in a stilldeadcode/vllm-radiance image with FP8 weights and fp8 KV cache, 262,144 context, 422,964 tokens of KV pool, and RCCL 2.27.7 replacing the image's newer RCCL because one card sits behind the chipset and cannot do PCIe atomics. The decode figure is given as a range, so no single generationTps is recorded. The bundled P2P all-reduce is disabled (RADIANCE_USE_R4D_AR=0) and TP=2 falls back to host-staged collectives; the user notes two CPU-attached cards would keep the P2P path and get better prefill.

generation:

29.9 tokens/s

quant:

Q8 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 29.92 t/s on a mixed GPU pool of R9700, RTX 5070 Ti, and RTX 4070 using llama.cpp RPC. Setup is llama.cpp with Q8 quant and 256k context, using MTP speculative decoding. The RPC pool runs each card on its native driver (ROCm for R9700, CUDA for Nvidias). User compares this to Vulkan solo on the R9700 (26.48 t/s) and their daily driver Q6_K_XL at 131k context (59.08 t/s). RPC showed tighter consistency (spread 7.4 vs 16.3). Flash Next saw no improvement.

generation:

34.7 tokens/s

prompt processing (prefill):

863.0 tokens/s

quant:

exl3-5.05bpw (EXL3)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 863 t/s prefill and 34.7 t/s decode on a single Radeon AI PRO R9700 32GB at 230k context. Setup is ExLlamaV3 (phoenixhaxor ROCm fork, commit cbbef08, patched for gfx12) with exl3 5.05 bpw, Q8 KV cache, 262144 context, chunk size 4096, batch 1. 112 of 512 experts per layer run on the GPU and 400 on CPU with 16 threads; the 102GB bf16 n-gram table is streamed from NVMe. MTP drafting is on with around 53-54% acceptance. The prefill figure covers only the new 101k tokens, the rest came from the prefix cache. User asks whether this is normal for the card or if there is room to tune.

generation:

150.0 tokens/s

quant:

MXFP4-FP8 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 150+ t/s single-stream generation on 4x Radeon AI PRO R9700 32GB. Setup uses the MXFP4-FP8 quant from tcclaviger, tested with agentic coding use-cases in a pi harness. User also gets 3-5 concurrent streams at ~100 t/s generation each, and prefill of 10k+ t/s. Speed and quality both exceeded expectations.

generation:

149.2 tokens/s

prompt processing (prefill):

3075.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.5-35B-A3B (MoE, 35B total, ~3.5B active) at 149.2 t/s decode and 3075 t/s prefill on an AMD Radeon AI PRO R9700 32GB. Setup is llama.cpp (Vulkan, RADV driver) with UD-Q4_K_XL quant, flash attention on, ubatch 2048, and a one-line rm_kq=1 patch; single GPU. The figures are the best RADV configuration (rm_kq=1 + ub2048); stock RADV gives 147.8 t/s decode and 2381 t/s prefill. AMDVLK with rm_kq=1 reaches 156.3 t/s decode but only 1823 t/s prefill. WikiText-2 perplexity is 6.9472 ± 0.046 across all configurations.

generation:

40-50 tokens/s

prompt processing (prefill):

1260-1360 tokens/s

quant:

2.875bpw

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4-Flash-0731 at ~40–50 tok/s on long structured answers and ~31–35 tok/s on free-form prose and reasoning on two Radeon AI PRO R9700 32GB cards with 192 GB system RAM. Setup is the affinity engine with a 2.875 bpw quant, 262K context fully allocated at load, and a DeepSeek-native speculative draft. Prefill is ~1,260–1,360 tok/s at 4K–64K context. Needle-in-a-haystack passed 3/3 at 4K/16K/64K, and a 15-turn agentic task passed every check. The user notes RDNA4 (gfx1201) only, two cards max, and ~64 GB+ RAM needed for experts that don't fit on the cards. An experimental DeepSeek-V4.1-Flash branch runs at only ~14 tok/s on two cards.

generation:

113.0 tokens/s

quant:

MXFP4 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 113 t/s on 2x Radeon AI PRO R9700 32GB. Setup is vLLM with MXFP4 and tensor parallelism 2, after moving the second card to CPU lanes with an M.2-to-PCIe riser so PCIe atomics and P2P work. User compares Windows LM Studio (31 t/s on 2 cards, 38 t/s on 1), Vulkan with MTP (39-55 t/s), Linux llama.cpp Q8 tensor split (34.4 t/s, up to ~52 on other prompts), vLLM MXFP4 TP=1 (45 t/s) and vLLM FP8 TP=2 (72 t/s). User settled on Qwen3.8 27B Q8 at around 72+ t/s for daily use.

quant:

IQ4_XS (GGUF)

User compares Qwen3.8 Flash-Next IQ4_XS against Qwen3.8 27B FP8 on a system with 64 GB DDR5 and 2x Radeon AI PRO R9700, serving both via vLLM. Both models were run through a reduced quick eval with the same questions: MMLU-Pro 83.8% vs 75.0%, GPQA Diamond 42.5% vs 27.5%, GSM8K 97.5% vs 90.0%, German MGSM 92.5% vs 90.0%, and IFEval 89.6% for both. The 250-case fast trial took 36m 27s on 27B FP8 and 2h 05m 47s on Flash-Next IQ4_XS. The user notes MMLU used only 80 random questions and the evals are not full, and that FP8 27B is around 4x faster than Flash-Next under concurrency.

generation:

184.9 tokens/s

prompt processing (prefill):

3948.0 tokens/s

quant:

MXFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 184.9 t/s generation and 3,948 t/s prompt processing on 2x Radeon AI PRO R9700 32GB. Setup is vLLM with MXFP4 weights at 16,384 tokens of context; a 256K configuration is mentioned separately. The same model on the same machine reached 19.2 t/s generation and 590 t/s prompt processing with default llama.cpp Q8_0, and 29.7 t/s generation and 1,458 t/s prompt processing with llama.cpp Q8_0 using gfx1201 and tensor split.

generation:

74.7 tokens/s

prompt processing (prefill):

3600.0 tokens/s

quant:

IQ4_XS (GGUF)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash Next Uncensored at 74.7 t/s decode and 3,600 t/s prefill at 64k context on 2x Radeon AI PRO R9700 with 128GB DDR5. Setup is the R9V engine with an IQ4_XS GGUF plus MTP and a KVA projector; concurrent throughput is 70.3 t/s at 8 streams, TTFT P50 338 ms, update P99 51.5 ms. KVA projectors give 1.45-1.85x prefill speedup (1700 to 2500-3150 t/s) for +2.6% to +8% perplexity depending on start layer; a multi-layer variant reaches 1.55x at +2% perplexity. RAM use floats around 50GB for expert storage.

generation:

24.1 tokens/s

prompt processing (prefill):

154.7 tokens/s

quant:

UD-IQ4_NL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User compares ROCm and Vulkan performance for DeepSeek V4 Flash 0731 and Qwen 3.8 Flash Next on a Radeon AI PRO R9700 32GB plus Strix Halo in llama.cpp. DeepSeek V4 Flash 0731 with UD-IQ4_NL quant and a DSpark Q2K-Q4K drafter on ROCm reached 154.67 t/s prompt and 24.07 t/s generation on a 40,180-token needle prompt; Vulkan reached 131.97 t/s prompt and 20.44 t/s generation. Qwen 3.8 Flash Next with UD-Q4_K-XL and no speculative decoding on ROCm reached 566.10 t/s prompt and 17.96 t/s generation; Vulkan reached 477.33 t/s prompt and 18.99 t/s generation. User concludes ROCm still wins for DeepSeek while Vulkan has closed the gap, and for Qwen ROCm leads on prompt processing but Vulkan leads on token generation.

generation:

37.0 tokens/s

prompt processing (prefill):

1069.0 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 37 t/s generation and 1069 t/s prompt processing on two Radeon AI PRO R9700 32GB cards in a GMKtec EVO-X2 Strix Halo machine. Setup is llama.cpp with the ROCm build, RCCL enabled via -DGGML_HIP_RCCL=ON, and the UD-Q4_K_XL GGUF. The GPUs connect over NVMe-to-USB4 enclosures and run at PCIe 4.0 x4. Tensor parallelism was slower than a single card under the Vulkan build; the ROCm build restored a decode speedup, and RCCL restored full prompt processing. Single card measured 28 t/s generation and 1070 t/s prompt processing.

generation:

276.0 tokens/s

prompt processing (prefill):

5809.0 tokens/s

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 5,809 t/s prefill and 276 t/s decode on an AMD Radeon AI PRO R9700 32GB. Setup is a custom vLLM MXFP4 build with online NVFP4 to MXFP4 conversion, using the Unsloth Dynamic v3 NVFP4 quant. Results come from BetterBench.