Local model performance on your hardware

prompt processing (prefill):

2750-2970 tokens/s

quant:

MXFP4 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 125–134 tok/s decode (greedy) on a single AMD Radeon AI PRO R9700 32GB, running on Windows 11 + WSL2 with a Ryzen 9 9950X. Setup is vLLM with AMD's official MXFP4 checkpoint (amd/Qwen3.8-27B-Quark-AWQ-MXFP4), DFlash2 speculative decoding with a FP8 drafter proposing 7 tokens per step (~60% acceptance, ~5.3 tokens per forward pass), and RDNA4 kernels for FP8 paged attention and gated-delta-net layers. Default context window is ~216k tokens, extendable to ~260k with -Long. Prefill is ~2,750–2,970 tok/s with 0.7 s time to first token on a 1.9k-token prompt. Decode deep in context is 165 tok/s at 32k, 136 at 98k, and 115 at 258k. Long-context recall retrieved 8/8 planted facts from a 258k-token prompt. A comparison llama.cpp setup (IQ4_XS GGUF + speculative decoding) does about 52–68 tok/s on the same card, though measured with a different prompt. Spilling into shared memory drops performance to ~10 tok/s.

generation:

70.0 tokens/s

quant:

MXFP4-FP8-GPTQ (GPTQ)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at around 70 t/s for a single session on 4x Radeon AI PRO R9700 32GB. Setup is vLLM with MXFP4-FP8-GPTQ weights, fp8 KV cache, tensor-parallel-size 4, 262144 max model length, and MTP speculative decoding with 3 tokens. The user is troubleshooting a decode performance cap hit with newer vLLM image tags starting from 29.06.19, and notes the BetterBench report incorrectly labels the GPU as RTX 4000 Ada.

generation:

179.7 tokens/s

quant:

W3A4 (safetensors)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 179.7 tok/s single-stream decode on a single AMD Radeon AI PRO R9700 32GB. Setup is stock vLLM with a plugin, 3-bit rotated GPTQ weights (W3A4) with FP8 KV cache, 262,144-token context, speculative decoding, and a 569,878-token prefix cache. Aggregate throughput is 488.5 tok/s at 8 concurrent requests. Accuracy: GSM8K 95.22, HumanEval 94.51, MMLU-Pro 60.57. The 3-bit weights are a 9.55 GB add-on; MXFP4 remains available at 156.1 tok/s single-stream.

generation:

74-143 tokens/s

quant:

FP8 (FP8)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at FP8 running tensor-parallel across two Radeon AI PRO R9700s, with roughly 1.6k tokens/s prefill and 74-143 tokens/s decode single stream depending on content. Setup is vLLM 0.27.1 in a stilldeadcode/vllm-radiance image with FP8 weights and fp8 KV cache, 262,144 context, 422,964 tokens of KV pool, and RCCL 2.27.7 replacing the image's newer RCCL because one card sits behind the chipset and cannot do PCIe atomics. The decode figure is given as a range, so no single generationTps is recorded. The bundled P2P all-reduce is disabled (RADIANCE_USE_R4D_AR=0) and TP=2 falls back to host-staged collectives; the user notes two CPU-attached cards would keep the P2P path and get better prefill.

generation:

113.0 tokens/s

quant:

MXFP4 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 113 t/s on 2x Radeon AI PRO R9700 32GB. Setup is vLLM with MXFP4 and tensor parallelism 2, after moving the second card to CPU lanes with an M.2-to-PCIe riser so PCIe atomics and P2P work. User compares Windows LM Studio (31 t/s on 2 cards, 38 t/s on 1), Vulkan with MTP (39-55 t/s), Linux llama.cpp Q8 tensor split (34.4 t/s, up to ~52 on other prompts), vLLM MXFP4 TP=1 (45 t/s) and vLLM FP8 TP=2 (72 t/s). User settled on Qwen3.8 27B Q8 at around 72+ t/s for daily use.

quant:

IQ4_XS (GGUF)

User compares Qwen3.8 Flash-Next IQ4_XS against Qwen3.8 27B FP8 on a system with 64 GB DDR5 and 2x Radeon AI PRO R9700, serving both via vLLM. Both models were run through a reduced quick eval with the same questions: MMLU-Pro 83.8% vs 75.0%, GPQA Diamond 42.5% vs 27.5%, GSM8K 97.5% vs 90.0%, German MGSM 92.5% vs 90.0%, and IFEval 89.6% for both. The 250-case fast trial took 36m 27s on 27B FP8 and 2h 05m 47s on Flash-Next IQ4_XS. The user notes MMLU used only 80 random questions and the evals are not full, and that FP8 27B is around 4x faster than Flash-Next under concurrency.

generation:

184.9 tokens/s

prompt processing (prefill):

3948.0 tokens/s

quant:

MXFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 184.9 t/s generation and 3,948 t/s prompt processing on 2x Radeon AI PRO R9700 32GB. Setup is vLLM with MXFP4 weights at 16,384 tokens of context; a 256K configuration is mentioned separately. The same model on the same machine reached 19.2 t/s generation and 590 t/s prompt processing with default llama.cpp Q8_0, and 29.7 t/s generation and 1,458 t/s prompt processing with llama.cpp Q8_0 using gfx1201 and tensor split.

generation:

276.0 tokens/s

prompt processing (prefill):

5809.0 tokens/s

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 5,809 t/s prefill and 276 t/s decode on an AMD Radeon AI PRO R9700 32GB. Setup is a custom vLLM MXFP4 build with online NVFP4 to MXFP4 conversion, using the Unsloth Dynamic v3 NVFP4 quant. Results come from BetterBench.

generation:

120.0 tokens/s

prompt processing (prefill):

3619.0 tokens/s

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B NVFP4 at 120 tok/s at concurrency 1 on a single AMD Radeon R9700, with prefill at 3,619 tok/s at 16000 depth. Setup is a vLLM fork (radiance-vllm-mxfp4) with Unsloth's Qwen3.8 27b NVFP4 weights on one card. Median decode varies by category from 67.1 tok/s on chat to 153.1 tok/s on json, while update p50 stays between 34.2 and 42.5 ms. Concurrency scales to 471 tok/s at 8 concurrent requests.

generation:

90.8 tokens/s

quant:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 90.8 t/s on a Radeon AI Pro R9700. Setup is vLLM with a Q8 quant, single stream. User says their vLLM numbers slightly beat the linked run, which wins on prefill by a lot, and recommends the llama-cpp-rdna-boosts repo for R9700 owners.

generation:

61.4 tokens/s

prompt processing (prefill):

7800.0 tokens/s

quant:

INT4 (W4A16)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.6 35B-A3B MoE on a single R9700 with vLLM, using INT4 weights from Avesed. The user also tests a 27B dense model with MTP spec=4. Prefill and decode speeds are measured at various context depths, and the user is happy with the results.

generation:

120.0 tokens/s

prompt processing (prefill):

12000.0 tokens/s

quant:

MXFP4-FP8

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports 80-120 t/s generation and 12k t/s prefill for a single request on 4x R9700. Setup is an optimized vLLM image with tcclaviger's MXFP4-FP8 quant, tensor-parallel-size 4, fp8 KV cache, and speculative decoding with MTP.

generation:

56.6 tokens/s

prompt processing (prefill):

198.5 tokens/s

quant:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B-FP8 at roughly 50-56 t/s generation on two Radeon AI PRO R9700 GPUs in TP=2 on PCIe Gen5 x8, with an AMD Ryzen 5 7400 CPU and 64 GB of DDR5-6000. Setup uses MTP3 speculative decoding, with accepted throughput of roughly 30-37 t/s. User asks whether these numbers are reasonable and where to look for bottlenecks.

generation:

75.2 tokens/s

prompt processing (prefill):

671.9 tokens/s

quant:

FP8

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks dual R9700 GPUs with tensor parallelism, comparing PCIe gen4 x16+x4 against gen5 x8/x8. Setup is vLLM with stilldeadcode/vllm-radiance:0.7.4 on a 9950X with 64GB DDR5 6000. The gen5 x8/x8 configuration shows 18-32% improvements, with results given for concurrency 1, 2, and 4.

generation:

280.0 tokens/s

prompt processing (prefill):

3831.0 tokens/s

quant:

MXFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 280 t/s decode on dual R9700s. Setup is MXFP4 quantization. Prompt processing at 128k context runs at 3831 t/s. The user notes the performance came from collaboration.

generation:

87.6 tokens/s

prompt processing (prefill):

4134.0 tokens/s

quant:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B FP8 on 2x R9700 with vLLM Radiance TP2. User also tests MXFP4 at 111.4 t/s and Qwen3.8 Flash Next at 35.4 t/s.