AMD Radeon AI PRO R9700 32GB reports

generation:

120.0 tokens/s

prompt processing (prefill):

3619.0 tokens/s

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B NVFP4 at 120 tok/s at concurrency 1 on a single AMD Radeon R9700, with prefill at 3,619 tok/s at 16000 depth. Setup is a vLLM fork (radiance-vllm-mxfp4) with Unsloth's Qwen3.8 27b NVFP4 weights on one card. Median decode varies by category from 67.1 tok/s on chat to 153.1 tok/s on json, while update p50 stays between 34.2 and 42.5 ms. Concurrency scales to 471 tok/s at 8 concurrent requests.

generation:

90.8 tokens/s

quant:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 90.8 t/s on a Radeon AI Pro R9700. Setup is vLLM with a Q8 quant, single stream. User says their vLLM numbers slightly beat the linked run, which wins on prefill by a lot, and recommends the llama-cpp-rdna-boosts repo for R9700 owners.

generation:

100.0 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash Next at ~100 t/s generation on 2x Radeon AI PRO R9700 with 128GB system RAM. Setup uses the R9V engine with an IQ4_XS GGUF and n-gram SSD streaming, which the user says fixed crashes after ~12 hours of torture testing. Q4_K_XL is now supported at 50 t/s generation but the user notes it still needs fine tuning.

generation:

61.4 tokens/s

prompt processing (prefill):

7800.0 tokens/s

quant:

INT4 (W4A16)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.6 35B-A3B MoE on a single R9700 with vLLM, using INT4 weights from Avesed. The user also tests a 27B dense model with MTP spec=4. Prefill and decode speeds are measured at various context depths, and the user is happy with the results.

generation:

120.0 tokens/s

prompt processing (prefill):

12000.0 tokens/s

quant:

MXFP4-FP8

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports 80-120 t/s generation and 12k t/s prefill for a single request on 4x R9700. Setup is an optimized vLLM image with tcclaviger's MXFP4-FP8 quant, tensor-parallel-size 4, fp8 KV cache, and speculative decoding with MTP.

quant:

Q4_K_XL (GGUF)

User asks for real-world sustained t/s on an R9700 with Qwen3.8-27B Q4 at 64K context and above. User cites an AMD blog figure of 51.8 t/s and 5090 benchmarks showing degradation from 75 t/s at 4K to about 26 t/s at 64K. User also asks about MTP speculative decoding stability.

generation:

56.6 tokens/s

prompt processing (prefill):

198.5 tokens/s

quant:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B-FP8 at roughly 50-56 t/s generation on two Radeon AI PRO R9700 GPUs in TP=2 on PCIe Gen5 x8, with an AMD Ryzen 5 7400 CPU and 64 GB of DDR5-6000. Setup uses MTP3 speculative decoding, with accepted throughput of roughly 30-37 t/s. User asks whether these numbers are reasonable and where to look for bottlenecks.

generation:

75.2 tokens/s

prompt processing (prefill):

671.9 tokens/s

quant:

FP8

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks dual R9700 GPUs with tensor parallelism, comparing PCIe gen4 x16+x4 against gen5 x8/x8. Setup is vLLM with stilldeadcode/vllm-radiance:0.7.4 on a 9950X with 64GB DDR5 6000. The gen5 x8/x8 configuration shows 18-32% improvements, with results given for concurrency 1, 2, and 4.

generation:

40.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares Qwen3.8 27B to Qwen3.6 35B-A3B for daily coding use. User reports Qwen3.8 27B is verbose and overthinks, while Qwen3.6 35B-A3B is faster and more concise. The comparison is on a Radeon AI PRO R9700, with an iGPU comparison raised as hypothetical.

generation:

26.6 tokens/s

prompt processing (prefill):

279.3 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports tensor-read-lazy with mmap fitting a Q4 model on a system with an AMD Radeon AI PRO R9700, RX 9070 XT, and Radeon 8060S (Strix Halo). The model is 103.68 GiB with 176.94 B params. Prompt processing reaches 279.25 t/s and generation 26.63 t/s.

generation:

78.1 tokens/s

prompt processing (prefill):

1512.0 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports custom kernels for RDNA4 giving roughly 3x generation speedup and roughly 30x prefill speedup over public vLLM-Radiance. Setup is dual R9700s with 128GB RAM. User also mentions work on the Muse Glimmer 30B dense model.

generation:

51.6 tokens/s

quant:

Q8_K_XL (GGUF)

kv:

F16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 51.56 tokens per second over 5369 generated tokens with MTP on 2x R9700, writing a binary tree in javascript. Setup is llama.cpp built in the ROCm 10 docker image with unsloth's Q8_K_XL GGUF, tensor split across both cards, 255000 context, f16 KV cache and MTP drafting up to 3 tokens, using 27GB on each card. Draft acceptance is 0.72147. The user sees 37-50 tg/s generating text, spiking over 60tg/s when writing code.

generation:

16.0 tokens/s

quant:

Q4_K_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User asks about adding RAM to run a larger quant of Qwen 3.8. User mentions Q4_K_XS ran fine at 16 tps.

generation:

280.0 tokens/s

prompt processing (prefill):

3831.0 tokens/s

quant:

MXFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 280 t/s decode on dual R9700s. Setup is MXFP4 quantization. Prompt processing at 128k context runs at 3831 t/s. The user notes the performance came from collaboration.

generation:

36.6 tokens/s

prompt processing (prefill):

17636.0 tokens/s

quant:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Hermes with Qwen 3.8 27B fp8 at 262k context on a quad R9700 setup, with prefill at 17636 t/s and TG at 36.6 t/s. Setup is vLLM-Radiance, with one GPU on PCIe 4x8 and three on 4x16. Best TG reached 106 t/s with 80% MTP 4 acceptance. vLLM-Radiance works well with the quad setup despite official dual support.

generation:

35.0 tokens/s

prompt processing (prefill):

242.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports that 2 GPUs outperform 3 due to a slow lane. Setup is Qwen3.8 Flash Next with the Q4_K_M quant, variant AD-4.27bpw.

generation:

29.0 tokens/s

quant:

Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B Q4_K_XL on a Radeon AI PRO R9700 with llama.cpp at 29 TG/s, 76% of a theoretical 38 TG/s. The figure is a theoretical maximum TG/s calculation for dense models. An RTX 5090 with 1.8 TB/s bandwidth is also mentioned.

generation:

87.6 tokens/s

prompt processing (prefill):

4134.0 tokens/s

quant:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B FP8 on 2x R9700 with vLLM Radiance TP2. User also tests MXFP4 at 111.4 t/s and Qwen3.8 Flash Next at 35.4 t/s.

generation:

84.5 tokens/s

prompt processing (prefill):

2000.4 tokens/s

quant:

Q5_K_M (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 84.48 tokens per second generation and 2000.37 tokens per second prompt eval on a Radeon 9700 with 32GB VRAM. Setup is llama.cpp on Vulkan with the UD-Q5_K_M GGUF, -ngl 99, 262144 context, q8_0 KV cache, flash attention on, batch and ubatch size 2048, 8 threads and parallel 1. With MTP, which the user is now testing, generation is around 110 - 130 t/s and prefill drops to 1700 t/s. ROCm gave much faster prefill but slower generation.

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

on

User reports 40-67 t/s decode on 2x Radeon AI PRO R9700 32GB at 131,072 context. Setup uses an F16 KV cache. Prefill throughput is 410-1500 t/s. MTP draft acceptance ranges from 0.33-0.61.