AMD Radeon AI PRO R9700 32GB reports

generation:

154.4 tokens/s

prompt processing (prefill):

3927.0 tokens/s

quant:

MXFP4 (GPTQ)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B (Swift-1.5 MXFP4 GPTQ) at 154.4 tok/s weighted single-stream decode and 3,927 tok/s prefill on one AMD Radeon AI PRO R9700 32GB. Setup is vllm-sly-radiance with MXFP4 weights, FP8 KV cache, 65,536 context, DFlash2 speculative decoding with a W4A16 drafter (k=7), 4,096 prefill chunk, prefix caching off. Prefill falls to 3,370 tok/s at 47k tokens; aggregate throughput is 480 tok/s at 8 concurrent requests and 512 tok/s at 16. Per-category decode ranges from 90.5 tok/s (prose) to 210.8 tok/s (json).

generation:

236.0 tokens/s

prompt processing (prefill):

8300-8900 tokens/s

quant:

fp8-iq4r-moe

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 236 tok/s single-stream aggregate on four R9700 32 GB cards at TP4, with 1,192 tok/s at 16 concurrent streams. Setup is radiance 1.11.0 with the fp8-iq4r-moe container (113.6 GiB, 4-bit experts, int8 trunk, bf16 protected experts), fp8 KV cache, MTP head depth 3, 262144 max model length, all weights in VRAM and the 47.7 GiB n-gram table pinned in host RAM. Decode step is 11.27 ms against a 11.2 ms reference; prefill measured 8.4K t/s at 4.4K and 70K prompt tokens, and single stream with thinking on reached 197 tok/s over 4,096 generated tokens. Concurrency rows are thinking off, 512 tokens each, wall-clock over the whole batch; a 2x TP2 attempt lost.

generation:

160.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 160 tok/s single-stream on two Radeon AI PRO R9700 cards. Setup details, engine, quant and KV cache are not stated in the post. At 8 concurrent users the user reports 470-630 tok/s aggregate. The user made a quant of the model and posted benchmarks and dev notes, and is looking to compare notes with other R9700 users.

generation:

66-71 tokens/s

prompt processing (prefill):

3200.0 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports gemma-4-26B-A4B Q8_0 (26.9 GB) on a Radeon AI PRO R9700 32GB in a Minisforum DEG2 eGPU over USB4, at about 3,200 t/s prefill (pp512) and 66-71 t/s decode, versus 1,387 t/s and 46.6 t/s on the Strix Halo iGPU. Setup is llama.cpp Vulkan/RADV with -ngl 99 -fa 1, a 256 MB BAR, the card pinned to profile_peak, and both GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 and GGML_VK_MMID_F16B=0 set. A 10 h soak ran 6,724/6,724 requests with decode flat from 66.2 to 63.7 tok/s between the first and last quarter. Big model loads wedge the card at the default power level unless profile_peak is set, which idles at 137 W; a wrapper switching it around jobs passed 22/22 image jobs. A full 32 GB BAR works only with the dock powered on at boot, where 27B Q8_0 decode is 17.9 tok/s at a 1k context.

generation:

29.8 tokens/s

prompt processing (prefill):

1287.0 tokens/s

quant:

IQ4_NL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports 29.8 t/s generation and 1287 t/s prompt processing. Source evidence (verbatim): Source: https://akougkas.io/journal/2026/serving-qwen38-27b-on-a-radeon-ai-pro-r9700/ # Serving Qwen3.8-27B on a Radeon AI PRO R9700 is not a one-liner Hardware: Radeon AI PRO R9700 (Navi 48, gfx1201) with 32 GB VRAM and 640 GB/s, Ryzen 7 5700X3D (8 cores), Ubuntu 24.04.4, kernel 6.17 OEM. Engine: llama.cpp b10687 (commit c841aee), HIP build on ROCm 7.2.4. Model: Qwen3.8-27B, Unsloth IQ4_NL (15.21 GiB). Single-stream llama-bench with no speculative decoding: -ngl 99 -fa 1 -ctk q8_0 -ctv q8_0 -p 2048 -n 512 gave 1287 tokens/s prefill (pp2048) and 29.8 tokens/s decode (tg512). The same bench for other quants: UD-Q4_K_XL 1151 / 26.7, UD-Q5_K_M 946 / 24.5, UD-Q6_K 821 / 23.2 (prefill / decode tokens/s).

generation:

53.1 tokens/s

quant:

FP8 (FP8)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 53.1 tok/s single-stream on four Radeon AI PRO R9700 32GB cards in one box. Setup is vLLM-Radiance 1.0.387 (vLLM 0.30.0, ROCm 7.14) with the official FP8 checkpoint, FP8 KV cache, and the model's MTP head with 8 draft tokens, two engines per box at TP=2, 256K max context. At 16 users the box serves 484 tok/s aggregate (about 40 tok/s per user); a random 1024-in/512-out profile gives 623 tok/s aggregate at 837 W. MTP acceptance is 83% on JSON extraction versus 39% on MT-Bench. FP8 scored 98.0/100 versus MXFP4's 95.3/100 on the user's extraction workbench, at 4,477 versus 4,819 validated jobs an hour. Three such boxes were built at about $10,733.91 each.

generation:

17.6 tokens/s

prompt processing (prefill):

616.5 tokens/s

quant:

UD-Q3_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-Flash-Next on a dual AMD Radeon AI PRO R9700 32GB setup (Lenovo P620, Threadripper Pro 3945WX, 128GB DDR4). Setup is llama.cpp with Unsloth UD-Q3_K_XL GGUF, -ncmoe 5, -ngl 49, tensor-split 1.10,0.90, and 3 MTP draft tokens. llama.cpp achieves 17.6 t/s generation and 616.5 t/s prompt processing, compared to 147.2 t/s and 6618.1 t/s on radiance 1.4.0, and 79.1 t/s and 1455.4 t/s on Strata 0.1.42. User notes radiance's prompt processing is unreasonably fast and adds a prefill slope metric. All three are considered equivalent in output quality for the user's purposes.

generation:

79.0 tokens/s

quant:

2.875 bpw

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek-V4-Flash-0731 at 79 tok/s greedy decode on four Radeon AI PRO R9700 cards. Setup is the affinity engine with a prebuilt 2.875 bpw quant, all 11,008 experts resident in VRAM with no host pool allocated, on a Threadripper PRO 9955WX with 128 GB RAM. The same box on two cards gave 36 tok/s with 45% of experts in VRAM and the rest streaming from a 51 GB pinned host pool. Sampled decode at temp 1.0 was 63 tok/s and about 90 tok/s on a chat answer with thinking on. Load takes about a minute, 42 seconds of it pushing 96 GB of experts onto the cards.

generation:

176.0 tokens/s

prompt processing (prefill):

6450.0 tokens/s

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports 176 t/s decode and 6,450 t/s prefill with Qwen3.8 Flash-Next Q4 on 2x R9700 with 64GB DDR5. Setup uses Deadcode's Radiance inference engine, optimized for 2x R9700, with a Q4 quant. Benchmark tool is BetterBench.

generation:

140-155 tokens/s

quant:

FP8+IQ4R

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at approximately 140-155 tok/s on 2x Radeon AI PRO R9700, up from 31 tok/s on Windows. Setup is the Radiance engine with FP8+IQ4R weights and MTP speculative decoding at 200K context. The user also reports Qwen3.8-27B at about 42 tok/s on one R9700 with Vulkan and MTP, Ling 3.0 Tiny at about 183 tok/s on an RTX 5060 Ti 16GB with Q8_0, and Gemma 4 12B at about 90 tok/s on the same card with a QAT UD-Q4_K_XL build and an MTP drafter.

generation:

90.0 tokens/s

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next IQ3_S at approximately 90 tk/s decode on an AMD R9700 32GB with 64GB RAM. Setup is Strata with IQ3_S quant. The user notes this is about 3x faster than a 27B model and was surprised by the quality given the small Q3 quant.

generation:

31.8 tokens/s

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B-Q4 at over 31.8 t/s baseline on one R9700 in a Razer Core X V2 over TB5 connected to a MacBook Pro M5 Max. Setup is the LSE engine with Q4 quant. Around 55 t/s with MTP=3, 54 to 80+ t/s with DFlash2 depending on context and text, and over 1400 PP/s on 1K+ token prompts. DFlash2 reaches over 80 t/s on HumanEval+. The user notes HumanEval+ reports predate the prefill optimization.

generation:

208.1 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B at 208.1 tok/s average on a Radeon AI PRO R9700, with a 227.8 tok/s peak request. Setup is the Lucebox engine with UD-IQ4_XS GGUF weights and a DFlash2 Q8_0 drafter at block size 16, q8_0 KV cache, single card. The figure is a HumanEval average; the repository also lists 110.6 tok/s for Qwen 3.8 27B on an RTX 5090 and 79.7 tok/s on 2x RTX 3090 with NVLink.

generation:

158.5 tokens/s

prompt processing (prefill):

1636.0 tokens/s

quant:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 158.5 tok/s decode on a Radeon AI PRO R9700 in an iPad Pro eGPU enclosure. Setup is LemonSeed Engine with Q4 weights and a Q8 DFlash2 draft, 131k context, speculative decoding on. Baseline decode is 31.2 tok/s and MTP=3 gives 108.7 tok/s; prefill at 4K is 1,636 tok/s and 1,395 tok/s at 32K. On Strix Halo (Radeon 8060S) the same model reaches 64.4 tok/s decode and 517 tok/s prefill at 4K.

generation:

100.0 tokens/s

quant:

MXFP4 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B MXFP4 at over 100 t/s on 2x R9700s, with over 200 t/s for an initial request and around 80 t/s under concurrent requests. Setup is DeepSeek Harness with MXFP4 weights on two R9700 cards, running at 130k context out of a 262k limit. The user is promoting a paid token service built on this setup rather than a personal benchmark.

generation:

40.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash reaching a peak of 40 t/s generation on 8x Radeon AI PRO R9700 under llama.cpp, but the run froze after 200-300 tokens. Setup is llama.cpp on 8x R9700 with 256GB system RAM and roughly 400GB total VRAM; the user notes llama.cpp on many GPUs was very slow, possibly due to ROCm. The user also runs MiniMax M2.7 (MXFP4_16, iq4_nl) on 8x R9700 and Qwen3-27b-BF16 on 4x7900xtx under vLLM, and gave up on a REAP GLM-5.1 50% build due to vLLM errors.

generation:

112.0 tokens/s

quant:

MXFP4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 112 tok/s decode on coding prompts with WHIRL, the engine they wrote, for Swift-1.5 27B MXFP4 on a single Radeon AI PRO R9700, against 61 tok/s for llama.cpp b11214 on the same GGUF. Setup is one Radeon AI PRO R9700 32GB over USB4 on Windows 11, MXFP4 weights, WHIRL v0.1.3 with speculative decoding, single request. Prefill reaches 1,757 tok/s at 128K and 970 at 256K, and 4 concurrent users get 182 tok/s in total. On Ornith-1.5-35B-A3B MXFP4 WHIRL decodes at 258 tok/s and prefills over 11,000 tok/s at 8K.

prompt processing (prefill):

2750-2970 tokens/s

quant:

MXFP4 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 125–134 tok/s decode (greedy) on a single AMD Radeon AI PRO R9700 32GB, running on Windows 11 + WSL2 with a Ryzen 9 9950X. Setup is vLLM with AMD's official MXFP4 checkpoint (amd/Qwen3.8-27B-Quark-AWQ-MXFP4), DFlash2 speculative decoding with a FP8 drafter proposing 7 tokens per step (~60% acceptance, ~5.3 tokens per forward pass), and RDNA4 kernels for FP8 paged attention and gated-delta-net layers. Default context window is ~216k tokens, extendable to ~260k with -Long. Prefill is ~2,750–2,970 tok/s with 0.7 s time to first token on a 1.9k-token prompt. Decode deep in context is 165 tok/s at 32k, 136 at 98k, and 115 at 258k. Long-context recall retrieved 8/8 planted facts from a 258k-token prompt. A comparison llama.cpp setup (IQ4_XS GGUF + speculative decoding) does about 52–68 tok/s on the same card, though measured with a different prompt. Spilling into shared memory drops performance to ~10 tok/s.

generation:

24.9 tokens/s

prompt processing (prefill):

611.0 tokens/s

quant:

Q5_K_M (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Qwen3.6 27B Q5_K_M on dual AMD Radeon AI PRO R9700 GPUs, reporting 24.85 t/s generation and 611.02 t/s prompt processing at 32768 context on a single GPU. Setup is llama.cpp with ROCm, Q5_K_M GGUF weights, q8_0 KV cache, flash attention, and 2 GPUs with layer split. Dual GPUs roughly doubled long-context prefill (1215.87 t/s at 32768) but did not improve single-stream generation (24.31 t/s). The user also tested Gemma 4 31B, Qwen3.6 35B A3B, and Gemma 4 26B A4B, finding sparse-active models much faster but not directly comparable to dense models.

generation:

130.0 tokens/s

prompt processing (prefill):

5000.0 tokens/s

quant:

FP8 (FP8)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B (Swift 1.5) at 130 t/s generation and 5000 t/s prefill on dual Radeon AI PRO R9700 32GB cards. Setup is vllm-radiance with FP8 weights. User asks whether any other RDNA4-specific engine could beat this, and is interested in running Qwen3.8 Flash-Next, which vllm-radiance does not yet support.