Local model performance on your hardware

generation:

70.0 tokens/s

quant:

MXFP4-FP8-GPTQ (GPTQ)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at around 70 t/s for a single session on 4x Radeon AI PRO R9700 32GB. Setup is vLLM with MXFP4-FP8-GPTQ weights, fp8 KV cache, tensor-parallel-size 4, 262144 max model length, and MTP speculative decoding with 3 tokens. The user is troubleshooting a decode performance cap hit with newer vLLM image tags starting from 29.06.19, and notes the BetterBench report incorrectly labels the GPU as RTX 4000 Ada.

generation:

179.7 tokens/s

quant:

W3A4 (safetensors)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 179.7 tok/s single-stream decode on a single AMD Radeon AI PRO R9700 32GB. Setup is stock vLLM with a plugin, 3-bit rotated GPTQ weights (W3A4) with FP8 KV cache, 262,144-token context, speculative decoding, and a 569,878-token prefix cache. Aggregate throughput is 488.5 tok/s at 8 concurrent requests. Accuracy: GSM8K 95.22, HumanEval 94.51, MMLU-Pro 60.57. The 3-bit weights are a 9.55 GB add-on; MXFP4 remains available at 156.1 tok/s single-stream.

generation:

113.0 tokens/s

quant:

MXFP4 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 113 t/s on 2x Radeon AI PRO R9700 32GB. Setup is vLLM with MXFP4 and tensor parallelism 2, after moving the second card to CPU lanes with an M.2-to-PCIe riser so PCIe atomics and P2P work. User compares Windows LM Studio (31 t/s on 2 cards, 38 t/s on 1), Vulkan with MTP (39-55 t/s), Linux llama.cpp Q8 tensor split (34.4 t/s, up to ~52 on other prompts), vLLM MXFP4 TP=1 (45 t/s) and vLLM FP8 TP=2 (72 t/s). User settled on Qwen3.8 27B Q8 at around 72+ t/s for daily use.

generation:

184.9 tokens/s

prompt processing (prefill):

3948.0 tokens/s

quant:

MXFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 184.9 t/s generation and 3,948 t/s prompt processing on 2x Radeon AI PRO R9700 32GB. Setup is vLLM with MXFP4 weights at 16,384 tokens of context; a 256K configuration is mentioned separately. The same model on the same machine reached 19.2 t/s generation and 590 t/s prompt processing with default llama.cpp Q8_0, and 29.7 t/s generation and 1,458 t/s prompt processing with llama.cpp Q8_0 using gfx1201 and tensor split.

generation:

276.0 tokens/s

prompt processing (prefill):

5809.0 tokens/s

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 5,809 t/s prefill and 276 t/s decode on an AMD Radeon AI PRO R9700 32GB. Setup is a custom vLLM MXFP4 build with online NVFP4 to MXFP4 conversion, using the Unsloth Dynamic v3 NVFP4 quant. Results come from BetterBench.

generation:

120.0 tokens/s

prompt processing (prefill):

3619.0 tokens/s

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B NVFP4 at 120 tok/s at concurrency 1 on a single AMD Radeon R9700, with prefill at 3,619 tok/s at 16000 depth. Setup is a vLLM fork (radiance-vllm-mxfp4) with Unsloth's Qwen3.8 27b NVFP4 weights on one card. Median decode varies by category from 67.1 tok/s on chat to 153.1 tok/s on json, while update p50 stays between 34.2 and 42.5 ms. Concurrency scales to 471 tok/s at 8 concurrent requests.

generation:

90.8 tokens/s

quant:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 90.8 t/s on a Radeon AI Pro R9700. Setup is vLLM with a Q8 quant, single stream. User says their vLLM numbers slightly beat the linked run, which wins on prefill by a lot, and recommends the llama-cpp-rdna-boosts repo for R9700 owners.

generation:

61.4 tokens/s

prompt processing (prefill):

7800.0 tokens/s

quant:

INT4 (W4A16)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.6 35B-A3B MoE on a single R9700 with vLLM, using INT4 weights from Avesed. The user also tests a 27B dense model with MTP spec=4. Prefill and decode speeds are measured at various context depths, and the user is happy with the results.

generation:

120.0 tokens/s

prompt processing (prefill):

12000.0 tokens/s

quant:

MXFP4-FP8

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports 80-120 t/s generation and 12k t/s prefill for a single request on 4x R9700. Setup is an optimized vLLM image with tcclaviger's MXFP4-FP8 quant, tensor-parallel-size 4, fp8 KV cache, and speculative decoding with MTP.

generation:

56.6 tokens/s

prompt processing (prefill):

198.5 tokens/s

quant:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B-FP8 at roughly 50-56 t/s generation on two Radeon AI PRO R9700 GPUs in TP=2 on PCIe Gen5 x8, with an AMD Ryzen 5 7400 CPU and 64 GB of DDR5-6000. Setup uses MTP3 speculative decoding, with accepted throughput of roughly 30-37 t/s. User asks whether these numbers are reasonable and where to look for bottlenecks.

generation:

75.2 tokens/s

prompt processing (prefill):

671.9 tokens/s

quant:

FP8

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks dual R9700 GPUs with tensor parallelism, comparing PCIe gen4 x16+x4 against gen5 x8/x8. Setup is vLLM with stilldeadcode/vllm-radiance:0.7.4 on a 9950X with 64GB DDR5 6000. The gen5 x8/x8 configuration shows 18-32% improvements, with results given for concurrency 1, 2, and 4.

generation:

280.0 tokens/s

prompt processing (prefill):

3831.0 tokens/s

quant:

MXFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 280 t/s decode on dual R9700s. Setup is MXFP4 quantization. Prompt processing at 128k context runs at 3831 t/s. The user notes the performance came from collaboration.

generation:

87.6 tokens/s

prompt processing (prefill):

4134.0 tokens/s

quant:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B FP8 on 2x R9700 with vLLM Radiance TP2. User also tests MXFP4 at 111.4 t/s and Qwen3.8 Flash Next at 35.4 t/s.