Local model performance on your hardware

generation:

154.4 tokens/s

prompt processing (prefill):

3927.0 tokens/s

quant:

MXFP4 (GPTQ)

kv:

FP8

User reports Qwen3.8-27B (Swift-1.5 MXFP4 GPTQ) at 154.4 tok/s weighted single-stream decode and 3,927 tok/s prefill on one AMD Radeon AI PRO R9700 32GB. Setup is vllm-sly-radiance with MXFP4 weights, FP8 KV cache, 65,536 context, DFlash2 speculative decoding with a W4A16 drafter (k=7), 4,096 prefill chunk, prefix caching off. Prefill falls to 3,370 tok/s at 47k tokens; aggregate throughput is 480 tok/s at 8 concurrent requests and 512 tok/s at 16. Per-category decode ranges from 90.5 tok/s (prose) to 210.8 tok/s (json).

generation:

236.0 tokens/s

prompt processing (prefill):

8300-8900 tokens/s

quant:

fp8-iq4r-moe

kv:

fp8

User reports Qwen3.8-Flash-Next at 236 tok/s single-stream aggregate on four R9700 32 GB cards at TP4, with 1,192 tok/s at 16 concurrent streams. Setup is radiance 1.11.0 with the fp8-iq4r-moe container (113.6 GiB, 4-bit experts, int8 trunk, bf16 protected experts), fp8 KV cache, MTP head depth 3, 262144 max model length, all weights in VRAM and the 47.7 GiB n-gram table pinned in host RAM. Decode step is 11.27 ms against a 11.2 ms reference; prefill measured 8.4K t/s at 4.4K and 70K prompt tokens, and single stream with thinking on reached 197 tok/s over 4,096 generated tokens. Concurrency rows are thinking off, 512 tokens each, wall-clock over the whole batch; a 2x TP2 attempt lost.

generation:

160.0 tokens/s

User reports Qwen3.8-Flash-Next at 160 tok/s single-stream on two Radeon AI PRO R9700 cards. Setup details, engine, quant and KV cache are not stated in the post. At 8 concurrent users the user reports 470-630 tok/s aggregate. The user made a quant of the model and posted benchmarks and dev notes, and is looking to compare notes with other R9700 users.

generation:

66-71 tokens/s

prompt processing (prefill):

3200.0 tokens/s

quant:

Q8_0 (GGUF)

User reports gemma-4-26B-A4B Q8_0 (26.9 GB) on a Radeon AI PRO R9700 32GB in a Minisforum DEG2 eGPU over USB4, at about 3,200 t/s prefill (pp512) and 66-71 t/s decode, versus 1,387 t/s and 46.6 t/s on the Strix Halo iGPU. Setup is llama.cpp Vulkan/RADV with -ngl 99 -fa 1, a 256 MB BAR, the card pinned to profile_peak, and both GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 and GGML_VK_MMID_F16B=0 set. A 10 h soak ran 6,724/6,724 requests with decode flat from 66.2 to 63.7 tok/s between the first and last quarter. Big model loads wedge the card at the default power level unless profile_peak is set, which idles at 137 W; a wrapper switching it around jobs passed 22/22 image jobs. A full 32 GB BAR works only with the dock powered on at boot, where 27B Q8_0 decode is 17.9 tok/s at a 1k context.

generation:

29.8 tokens/s

prompt processing (prefill):

1287.0 tokens/s

quant:

IQ4_NL (GGUF)

kv:

q8_0

User reports 29.8 t/s generation and 1287 t/s prompt processing. Source evidence (verbatim): Source: https://akougkas.io/journal/2026/serving-qwen38-27b-on-a-radeon-ai-pro-r9700/ # Serving Qwen3.8-27B on a Radeon AI PRO R9700 is not a one-liner Hardware: Radeon AI PRO R9700 (Navi 48, gfx1201) with 32 GB VRAM and 640 GB/s, Ryzen 7 5700X3D (8 cores), Ubuntu 24.04.4, kernel 6.17 OEM. Engine: llama.cpp b10687 (commit c841aee), HIP build on ROCm 7.2.4. Model: Qwen3.8-27B, Unsloth IQ4_NL (15.21 GiB). Single-stream llama-bench with no speculative decoding: -ngl 99 -fa 1 -ctk q8_0 -ctv q8_0 -p 2048 -n 512 gave 1287 tokens/s prefill (pp2048) and 29.8 tokens/s decode (tg512). The same bench for other quants: UD-Q4_K_XL 1151 / 26.7, UD-Q5_K_M 946 / 24.5, UD-Q6_K 821 / 23.2 (prefill / decode tokens/s).

generation:

53.1 tokens/s

quant:

FP8 (FP8)

kv:

FP8

User reports Qwen3.8-27B at 53.1 tok/s single-stream on four Radeon AI PRO R9700 32GB cards in one box. Setup is vLLM-Radiance 1.0.387 (vLLM 0.30.0, ROCm 7.14) with the official FP8 checkpoint, FP8 KV cache, and the model's MTP head with 8 draft tokens, two engines per box at TP=2, 256K max context. At 16 users the box serves 484 tok/s aggregate (about 40 tok/s per user); a random 1024-in/512-out profile gives 623 tok/s aggregate at 837 W. MTP acceptance is 83% on JSON extraction versus 39% on MT-Bench. FP8 scored 98.0/100 versus MXFP4's 95.3/100 on the user's extraction workbench, at 4,477 versus 4,819 validated jobs an hour. Three such boxes were built at about $10,733.91 each.

generation:

17.6 tokens/s

prompt processing (prefill):

616.5 tokens/s

quant:

UD-Q3_K_XL (GGUF)

User benchmarks Qwen3.8-Flash-Next on a dual AMD Radeon AI PRO R9700 32GB setup (Lenovo P620, Threadripper Pro 3945WX, 128GB DDR4). Setup is llama.cpp with Unsloth UD-Q3_K_XL GGUF, -ncmoe 5, -ngl 49, tensor-split 1.10,0.90, and 3 MTP draft tokens. llama.cpp achieves 17.6 t/s generation and 616.5 t/s prompt processing, compared to 147.2 t/s and 6618.1 t/s on radiance 1.4.0, and 79.1 t/s and 1455.4 t/s on Strata 0.1.42. User notes radiance's prompt processing is unreasonably fast and adds a prefill slope metric. All three are considered equivalent in output quality for the user's purposes.

generation:

79.0 tokens/s

quant:

2.875 bpw

User reports DeepSeek-V4-Flash-0731 at 79 tok/s greedy decode on four Radeon AI PRO R9700 cards. Setup is the affinity engine with a prebuilt 2.875 bpw quant, all 11,008 experts resident in VRAM with no host pool allocated, on a Threadripper PRO 9955WX with 128 GB RAM. The same box on two cards gave 36 tok/s with 45% of experts in VRAM and the rest streaming from a 51 GB pinned host pool. Sampled decode at temp 1.0 was 63 tok/s and about 90 tok/s on a chat answer with thinking on. Load takes about a minute, 42 seconds of it pushing 96 GB of experts onto the cards.

generation:

176.0 tokens/s

prompt processing (prefill):

6450.0 tokens/s

quant:

Q4

User reports 176 t/s decode and 6,450 t/s prefill with Qwen3.8 Flash-Next Q4 on 2x R9700 with 64GB DDR5. Setup uses Deadcode's Radiance inference engine, optimized for 2x R9700, with a Q4 quant. Benchmark tool is BetterBench.

generation:

90.0 tokens/s

quant:

IQ3_S (GGUF)

User reports Qwen3.8-Flash-Next IQ3_S at approximately 90 tk/s decode on an AMD R9700 32GB with 64GB RAM. Setup is Strata with IQ3_S quant. The user notes this is about 3x faster than a 27B model and was surprised by the quality given the small Q3 quant.

generation:

31.8 tokens/s

quant:

Q4

User reports Qwen3.8-27B-Q4 at over 31.8 t/s baseline on one R9700 in a Razer Core X V2 over TB5 connected to a MacBook Pro M5 Max. Setup is the LSE engine with Q4 quant. Around 55 t/s with MTP=3, 54 to 80+ t/s with DFlash2 depending on context and text, and over 1400 PP/s on 1K+ token prompts. DFlash2 reaches over 80 t/s on HumanEval+. The user notes HumanEval+ reports predate the prefill optimization.

generation:

208.1 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

q8_0

User reports Qwen 3.8 27B at 208.1 tok/s average on a Radeon AI PRO R9700, with a 227.8 tok/s peak request. Setup is the Lucebox engine with UD-IQ4_XS GGUF weights and a DFlash2 Q8_0 drafter at block size 16, q8_0 KV cache, single card. The figure is a HumanEval average; the repository also lists 110.6 tok/s for Qwen 3.8 27B on an RTX 5090 and 79.7 tok/s on 2x RTX 3090 with NVLink.

generation:

158.5 tokens/s

prompt processing (prefill):

1636.0 tokens/s

quant:

Q4

User reports Qwen3.8-27B at 158.5 tok/s decode on a Radeon AI PRO R9700 in an iPad Pro eGPU enclosure. Setup is LemonSeed Engine with Q4 weights and a Q8 DFlash2 draft, 131k context, speculative decoding on. Baseline decode is 31.2 tok/s and MTP=3 gives 108.7 tok/s; prefill at 4K is 1,636 tok/s and 1,395 tok/s at 32K. On Strix Halo (Radeon 8060S) the same model reaches 64.4 tok/s decode and 517 tok/s prefill at 4K.

generation:

100.0 tokens/s

quant:

MXFP4 (MXFP4)

User reports Qwen3.8-27B MXFP4 at over 100 t/s on 2x R9700s, with over 200 t/s for an initial request and around 80 t/s under concurrent requests. Setup is DeepSeek Harness with MXFP4 weights on two R9700 cards, running at 130k context out of a 262k limit. The user is promoting a paid token service built on this setup rather than a personal benchmark.

generation:

40.0 tokens/s

User reports DeepSeek V4 Flash reaching a peak of 40 t/s generation on 8x Radeon AI PRO R9700 under llama.cpp, but the run froze after 200-300 tokens. Setup is llama.cpp on 8x R9700 with 256GB system RAM and roughly 400GB total VRAM; the user notes llama.cpp on many GPUs was very slow, possibly due to ROCm. The user also runs MiniMax M2.7 (MXFP4_16, iq4_nl) on 8x R9700 and Qwen3-27b-BF16 on 4x7900xtx under vLLM, and gave up on a REAP GLM-5.1 50% build due to vLLM errors.

generation:

112.0 tokens/s

quant:

MXFP4 (GGUF)

User reports 112 tok/s decode on coding prompts with WHIRL, the engine they wrote, for Swift-1.5 27B MXFP4 on a single Radeon AI PRO R9700, against 61 tok/s for llama.cpp b11214 on the same GGUF. Setup is one Radeon AI PRO R9700 32GB over USB4 on Windows 11, MXFP4 weights, WHIRL v0.1.3 with speculative decoding, single request. Prefill reaches 1,757 tok/s at 128K and 970 at 256K, and 4 concurrent users get 182 tok/s in total. On Ornith-1.5-35B-A3B MXFP4 WHIRL decodes at 258 tok/s and prefills over 11,000 tok/s at 8K.

generation:

24.9 tokens/s

prompt processing (prefill):

611.0 tokens/s

quant:

Q5_K_M (GGUF)

kv:

q8_0

User benchmarks Qwen3.6 27B Q5_K_M on dual AMD Radeon AI PRO R9700 GPUs, reporting 24.85 t/s generation and 611.02 t/s prompt processing at 32768 context on a single GPU. Setup is llama.cpp with ROCm, Q5_K_M GGUF weights, q8_0 KV cache, flash attention, and 2 GPUs with layer split. Dual GPUs roughly doubled long-context prefill (1215.87 t/s at 32768) but did not improve single-stream generation (24.31 t/s). The user also tested Gemma 4 31B, Qwen3.6 35B A3B, and Gemma 4 26B A4B, finding sparse-active models much faster but not directly comparable to dense models.

generation:

130.0 tokens/s

prompt processing (prefill):

5000.0 tokens/s

quant:

FP8 (FP8)

User reports Qwen3.8 27B (Swift 1.5) at 130 t/s generation and 5000 t/s prefill on dual Radeon AI PRO R9700 32GB cards. Setup is vllm-radiance with FP8 weights. User asks whether any other RDNA4-specific engine could beat this, and is interested in running Qwen3.8 Flash-Next, which vllm-radiance does not yet support.

generation:

70.0 tokens/s

quant:

MXFP4-FP8-GPTQ (GPTQ)

kv:

fp8

User reports Qwen3.8-Flash-Next at around 70 t/s for a single session on 4x Radeon AI PRO R9700 32GB. Setup is vLLM with MXFP4-FP8-GPTQ weights, fp8 KV cache, tensor-parallel-size 4, 262144 max model length, and MTP speculative decoding with 3 tokens. The user is troubleshooting a decode performance cap hit with newer vLLM image tags starting from 29.06.19, and notes the BetterBench report incorrectly labels the GPU as RTX 4000 Ada.

generation:

36.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

User reports Qwen3.8-Flash-Next at ~36 tok/s single stream on a single Radeon AI PRO R9700 32GB with 64GB system RAM. Setup is llama.cpp with stew675's RDNA4 patches (r30), AtomicChat Q4_K_M quant, q8_0 KV cache, flash attention, 2 slots x 131k context, -ncmoe 41 and MOE_EXPERT_CACHE_MIB=4096, GPU capped at 220W. Two streams combined reach ~50 tok/s and prompt reading is 526-589 tok/s. The 27B Q5_K_XL is ~1.7x faster per turn and reads prompts 1.6-2x as fast, but scores lower on tool-eval-bench Hard Mode (79.8 vs 93.9).