Speeds are as each source reported them. GPU count, offloading and concurrent requests change them, and prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing two reports.
User reports GLM-5.3-Flash at about 11 tok/s on an RX 9070 XT 16GB, with long prompts read at about 280 tok/s on Linux.
Setup is the user's own engine with IQ4_XS GGUF, experts streamed from NVMe drives with hot experts kept on the GPU and in RAM, on a Core Ultra 7 265KF with 32 GB DDR5 and two NVMe drives.
Windows gives about 10 tok/s and about 250 tok/s prompt reading. Testers measured about 11 tok/s on an Intel Arc Pro B70 32GB and 5.6 tok/s on an RTX 3090 plus RTX 3060; the 12-13 and 6.5-7 tok/s figures are expectations from logs, not measurements. The model needs about 157 GB on disk and 32 GB of RAM.
User reports Qwen3-8B at 68.4 tok/s with speculative decoding on an RX 9070 XT 16GB, versus 42.7 tok/s plain, using a custom engine that packs bf16 weights into a lossless 12-bit format (B12) with a Qwen3-0.6B draft model.
Setup is the user's own engine on Windows with ROCm PyTorch, batch-invariant GEMV kernels and seeded Gumbel-max sampling; speculative output matched plain decoding on 11 of 11 prompts.
On 50 longer prompts of 512 tokens the speculative run reached 73.0 tok/s with 50/50 matches. The user notes llama.cpp Q8_0 with a draft is about 1.4x faster (98.8 tok/s ROCm, 95.0 Vulkan) but lossy, matching plain output on only 7 of 11 prompts.
User benchmarks MoE expert offloading on an RX 9070 XT 16GB with 32GB DDR4, reporting Qwen3.6-35B-A3B at 71.6 t/s decode and 1246 t/s prefill at 128K context.
Setup is ExLlamaV3 with EXL3 4.0 bpw and Q8 KV cache, DFlash2 with 2 drafts, experts offloaded to system RAM.
Prefill drops from 1900 t/s at 16K to 700 t/s at 128K on the 35B models. Ornith 1.5 35B-A3B reached 57.3 t/s decode and 1168 t/s prefill at 128K, and Qwen3.8-Flash-Next Coder on Polystrata with IQ1_M reached 44.0 t/s decode and 1572 t/s prefill at 128K.
User asks which quant to settle on for Qwen3.8 Flash-Next run with Strata on an RX 9070 XT with 64GB DDR5 and a 9800X3D, all at 256k context.
ISTA IQ3_S runs at ~70-75 tokens/s, Unsloth UD-IQ3_XSS (3.1) at ~60-65 tokens/s and Unsloth UD-IQ4_XS (3.1) at ~40-50 tokens/s.
User notices no substantial quality difference between the three and questions whether the ~40% speed penalty of UD-IQ4_XS over IQ3_S is worth it.
User reports Qwen3.8-Flash-Next at 88 t/s generation and 514 t/s prompt processing on an RX 9070 XT 16GB.
Setup is the Avalon llama.cpp fork with IQ3KS at xhigh effort, after upgrading the machine to 64 GB of 4800 MHz DDR5.
The run used 5 symbols over 4 runs plus 1 failure, at 42-45 minutes per run. The user also compares against Anthropic Haiku 5.5 at 207 t/s and other local runs on Strata and llama.cpp.
User reports DeepSeek V4.1 Flash at 1M context running on an RX 9070 XT 16GB with experts spread across VRAM, 64 GB DDR4 and an SSD, at about 7 tok/s generation.
Setup is a patched Strata engine (HIP, gfx1201) with IST-DAS 1.75 bpw weights, plus a 5800X3D and PCIe 3.0 NVMe. The patch adds KV splicing and resident-only expert routing during prompt reads.
Prompt reading for a ~37K-token agent system prompt took about 100 s before the patch; the user claims agent sessions now start 20x faster. A 1.2K-token prompt previously streamed 58 GB off the SSD. The splice is approximate and resident routing changes answers slightly; the user's coding suite scored 12/13 both ways.
User reports Qwen3.5 27B at IQ3_S prefill of 1049.13 t/s (pp512) on an RX 9070 XT with the Lemonade B1356 ROCm build, against 460.03 t/s on B1357.
Setup is llama.cpp with IQ3_S weights (9.72 GiB), q8_0 KV cache, flash attention on, 99 layers offloaded and 512 batch.
Prefill falls from 1049.13 to 460.03 t/s at pp512 between B1356 and B1357, and rebuilding llama.cpp over B1356 returns the slow result. No generation figures are given.
User reports Qwen3.8-27B at about 65 tok/s on a Radeon RX 9070 XT.
Setup is llama.cpp with IQ3_M weights, run under both Vulkan and ROCm 10.1 on a Ryzen 7 7800X3D with 32 GB DDR5.
User asks whether Strata can span two machines to run Qwen3.8-Flash-Next (125B-A6B), and whether llama.cpp RPC over a LAN is realistic; no figures are given for that setup.
User reports ~3700 t/s prefill with Qwen3.8 Flash-Next fp8-iq4r on a single RX 9070 XT, with experts placed in RAM and a ~57GB host pool.
KV cache uses fp8_v4 and --max-num-batched-tokens is set to 4096, which gave the best prefill results. The card is power limited during prefill at over 300W, and PCIe5 x16 is nearly saturated; gen 4 drops prefill to ~2000 t/s.
Prefill stays flat across context while decode decays 18-19% over 100k tokens, with a worst case of ~45 t/s at 240k context. Vision is fully working. The user has 96GB of DDR5 and is still working on decode speed.
User reports Qwen3.8 27B at about 30-50 tok/s on an RX 9070 XT paired with an RTX 3080, with performance dropping noticeably at 64k versus 128k context.
Setup is llama.cpp with Q4_K_M weights and Q8 KV cache, layer split rather than tensor split, MTP enabled, one parallel slot and flash attention on. MTP roughly doubles speed; the other tweaks mainly save VRAM.
The user calls the pairing a bad choice for LLM work, citing the 9070 XT's lower memory bandwidth and weaker ROCm/Vulkan performance, and notes task manager shows only about 60% and 30% GPU usage on the two cards. Monitors must stay on the 9070 XT or performance collapses.
User reports 46.06 t/s generation and 549.27 t/s prompt processing.
Source evidence (verbatim):
Source: https://lemmy.world/post/48144487
# Qwen 3.6 27B running at 46 tok/s on an RX 9070 XT (llama.cpp + MTP Speculative Decoding is basically magic)
Just got my hands on a new AMD Radeon RX 9070 XT (16GB) and wanted to share some inference numbers. I've been messing around with llama.cpp via their official ROCm Docker image, testing out Qwen 3.6 27B (Omnimerge-v4) in IQ3_M.
Honestly, the performance you can squeeze out of a 27B model on a 16GB consumer card right now is blowing my mind.
Here’s the breakdown:
The Setup
- GPU: AMD Radeon RX 9070 XT (RDNA4 / gfx1201) - 16 GB VRAM
- CPU: AMD Ryzen 9 9950X3D
- OS/Backend: Linux via Docker using ghcr.io/ggml-org/llama.cpp:server-rocm. (Props to the devs, it natively supports RDNA4 gfx1201 out of the box!)
- Model: Qwen3.6-27B-Omnimerge-v4-IQ3_M.gguf (~13 GB)
- Context: 16k
Tweaks:
- Set -np 1 since I'm just running it as a single-user chatbot in Open WebUI.
- Slapped on 8-bit KV cache (--cache-type-k q8_0 --cache-type-v q8_0) to save about 50% VRAM.
- Enabled MTP Speculative Decoding (--spec-type draft-mtp).
The Numbers (512-token test)
- Prompt Processing (TTFT): 549.27 tok/s (1220 tokens evaluated in ~2.2s). Latency: 1.82 ms/token.
- Text Gen: 46.06 tok/s (512 tokens generated in ~11.1s). Latency: 21.71 ms/token.
- MTP Stats: Draft acceptance rate was super high at 62.7% (333 drafts accepted / 531 generated). Aggregate speed (including prompt eval) hit roughly 48.97 tok/s.
Memory Footprint
- VRAM: Sitting at 14.46 GB out of 16 GB. This leaves about 1.5 GB of breathing room, which has been totally stable with zero OOM crashes so far.
- System RAM: ~4.3 GB (mostly the host-side prompt cache helping speed up subsequent turns).
RDNA4 is ready: The latest ROCm images and llama.cpp HIP libs support the 9070 XT natively. Didn't even need to mess with HSA_OVERRIDE_GFX_VERSION.
KV Cache quantization is required: Pushing the KV cache to q8_0 is the only reason a 16k context window fits on a 16GB card alongside a 27B model.
If anyone with a 16GB card is looking for the sweet spot, this has to be one of the best price-to-performance setups available right now. Let me know if you want my docker-compose.yml or run scripts...
Edit:
To see how much MTP actually helps, I ran the exact same 512-token prompt test with speculative decoding toggled off:
Standard Inference (No Speculation):
- Speed: 25.88 tokens/sec (Latency: 38.64 ms/token)
- Prompt Eval: 239.46 tokens/sec
MTP Speculative Decoding (MTP):
- Speed: 46.06 tokens/sec (Latency: 21.71 ms/token)
- Prompt Eval: 549.27 tokens/sec
- Draft Acceptance Rate: 62.7%
For more info, here are my logs
::: spoiler Logs
```
ROCm GPU and System Info:
device_info:
- ROCm0 : AMD Radeon RX 9070 XT (16304 MiB, 16200 MiB free)
- ROCm1 : AMD Ryzen 9 9950X3D 16-Core Processor (15589 MiB, 25012 MiB free)
- CPU : AMD Ryzen 9 9950X3D 16-Core Processor (31178 MiB, 31178 MiB free)
I system_info: n_threads = 16 (n_threads_batch = 16) / 32 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 |
AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1
Context Initialization (16k single-slot & 8-bit KV Cache):
I srv load_model: [spec] estimated memory usage of MTP context is 178.02 MiB
W llama_context: n_ctx_seq (16384) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
I srv load_model: initializing slots, n_slots = 1
I slot load_model: id 0 | task -1 | new slot, n_ctx = 16384
I srv load_model: prompt cache is enabled, size limit: 8192 MiB
I srv llama_server: model loaded
I srv llama_server: server is listening on http://0.0.0.0:8080/
Benchmark Request Timing Statistics (Generating 512 tokens):
I slot launch_slot_: id 0 | task 3454 | processing task, is_child = 0
I slot print_timing: id 0 | task 3454 | n_decoded = 103, tg = 37.93 t/s
I slot print_timing: id 0 | task 3454 | n_decoded = 247, tg = 43.16 t/s
I slot print_timing: id 0 | task 3454 | n_decoded = 392, tg = 44.87 t/s
I slot print_timing: id 0 | task 3454 | prompt eval time = 195.88 ms / 42 tokens ( 4.66 ms per token, 214.41 tokens per second)
I slot print_timing: id 0 | task 3454 | eval time = 11116.24 ms / 512 tokens ( 21.71 ms per token, 46.06 tokens per second)
I slot print_timing: id 0 | task 3454 | total time = 11312.12 ms / 554 tokens
I slot print_timing: id 0 | task 3454 | graphs reused = 3560
I slot print_timing: id 0 | task 3454 | draft acceptance = 0.62712 ( 333 accepted / 531 generated)
I statistics draft-mtp: #calls(b,g,a) = 6 3607 3607, #gen drafts = 3607, #acc drafts = 2824, #gen tokens = 10820, #acc tokens = 6605, dur(b,g,a) = 0.008, 31925.089, 2.740 ms
I slot release: id 0 | task 3454 | stop processing: n_tokens = 553, truncated = 0
```
___
:::
User reports Qwen3.8-Flash-Next at around 40 tokens/s on a single RX 9070 XT 16 GB, with experts partly held in 64 GB of system RAM and read from NVMe when not cached.
Setup is Strata 0.1.44.1 with Unsloth UD-IQ4_XS weights, INT8 KV cache and a 292,000-token context using YaRN; the RAM expert budget is 48 GiB and the engine memory limit 57 GiB.
Prompt processing is about 5,364 tokens/s but may reuse cached context, so it is not a cold-prefill figure. Thinking runs about 36.5 tokens/s and some runs reached about 51 tokens/s. A 49 GiB expert budget was OOM-killed on a roughly 161,000-token conversation.
User reports 62 t/s generation.
Source evidence (verbatim):
Source: https://digtvbg.com/blog/llama-server-vulkan-rdna4-vllm-rocm-benchmark/
# Local LLM Inference on AMD RX 9070 XT: Vulkan vs ROCm Benchmarks on RDNA4
Hardware: AMD RX 9070 XT with 16GB VRAM on CachyOS. Model: Qwen3.5-9B-UD-Q6_K_XL (about 6.5 bits per weight). llama-server from llama.cpp built against Vulkan with --n-gpu-layers 999 -c 65536 --batch-size 2048 --ubatch-size 2048 --parallel 1: 62 tokens/second. vLLM on ROCm 7.2: 48 tokens/second. Same prompt, max tokens, temperature 0 and seed 42 for both; 10 measured runs per backend after 2 warmup runs. No speculative decoding is configured.
User reports Qwen3.8-27B at 36.5 t/s decode on a single 16 GB RX 9070 XT, using a custom llama.cpp fork with adaptive KV streaming over PCIe.
Setup is llama.cpp with IQ4_XS weights and Q8_0/Q4_0 KV cache, running the full 262,144-token context with the KV cache held in host RAM (~7.4 GB) and streamed in bounded working sets.
Decode falls from 36.5 t/s at 1k to 9.5 t/s at 262k; with MTP speculative decoding it reaches 71.8-86.5 t/s at 1k-4k and 31.1 t/s at 262k. Prefill is 1368.4 t/s at 1k and 585.8 t/s at 262k. The card runs at a 231W limit with a 3.6% prefill penalty and no decode loss.
User reports Qwen3.8-27B at ~150 tok/s on code at short context on a single RX 9070 XT.
Setup is ExLlamaV3 with EXL3 3.0 bpw quant, DFlash2 speculative decoding, and TabbyAPI, running in a container with RDNA4 patches. Default context is 96k, with 128k available if nothing else uses the card.
At 120k context code generation drops to ~105 tok/s, prose to ~60 tok/s. Prompt processing is ~1,100-1,250 tok/s on short prompts and ~900 tok/s at 128k. Up to 3 concurrent requests share the context. GPQA Diamond scores 88.4% at xhigh effort and ~81% at medium.
User reports GLM-5.2 744B at 0.2414 tok/s (10-run median) on an RX 9070 XT 16GB with colibri's Vulkan backend at its default 320 resident experts.
Setup is colibri's Vulkan backend with COLI_VK_EXPERTS=320, MTP drafting at 2.00 tokens per forward, fmt=4 grouped int4 weights with group size 64, experts streamed from a Gen3 x4 NVMe and 64GB DDR4 system RAM, Resizable BAR on, 40 generated tokens.
The box is storage-bound. The HIP arms (0.3189 tok/s at 6 GB, 0.3331 at 12 GB) ran with MTP disabled by a guard bug, and the author has withdrawn that Vulkan-vs-HIP comparison. 640 resident experts gave 0.2475 tok/s, and with Resizable BAR off the Vulkan run fell to 0.11 tok/s.
User reports Qwen3-8B Q4_K_M at 98.8 t/s decode and 4245 t/s prefill (pp512) on an RX 9070 XT with llama.cpp's HIP backend.
Setup is llama.cpp b10290 official HIP build on one RX 9070 XT 16 GB, Q4_K_M weights, all layers on the GPU, Windows 11, no speculative decoding.
The issue reports a Vulkan backend fault on the same card: Qwen3-8B Q4_K_M drops to 21.0 t/s decode and 791 t/s at pp128 there, and models with hidden size 4096 or more run 4.7-6.7x slower than on HIP, while Qwen3-4B is normal at 183.0 t/s. Qwen3-14B Q4_K_M runs 54.3 t/s on HIP against 8.1 on Vulkan.
User reports Qwen3.5 9B Q4_K_M decoding at 7.89 t/s on an AMD RX 9070 XT under a pinned llama.cpp Vulkan build (b10964), while LM Studio's Vulkan llama.cpp build reaches 29.89-30.46 t/s on the same GPU and byte-identical GGUF.
Setup is llama.cpp with Q4_K_M and q8_0 KV cache at 32768 context, flash attention on, single RX 9070 XT; llama-bench -p 128 -n 64 -r 1 gives 451.45 pp128 t/s and 7.89 tg64 t/s at -ngl 99.
-ngl 0 (CPU) decodes at 8.00 t/s and -ngl 99 at 7.83-7.89 t/s, so GPU offload buys nothing; the iGPU alone gives 3.04 t/s. Through the managed runtime decode is 7.41-7.49 t/s with prompt eval 98-222 t/s. LM Studio loaded the model at 255744 context and was still about 4x faster. Root cause not established.
User reports Qwen3.6 35B A3B at 62 t/s on an AMD RX 9070 XT 16GB, with vLLM on ROCm reaching 48 t/s in the same comparison.
Setup is llama.cpp with the Vulkan backend, 32k context, on a Ryzen 7 9800X3D with 64GB DDR5; the MoE model spills part of its weights to system RAM.
User notes the setup is early and numbers are not settled, that 16GB VRAM rules out big dense models, and that the local model needs supervision.
User reports Qwen 3.8 27B at 42 t/s on an AMD RX 9070 XT 16GB.
Setup is llama.cpp with MTP2 speculative decoding and a 64K context, using about 1 GB of working RAM during evaluation.
MTP2 delivered +41% to +53% throughput depending on the quantization, but the user notes the fastest configuration was not necessarily best for reasoning quality.