Summary
User reports Qwen3-8B at 68.4 tok/s with speculative decoding on an RX 9070 XT 16GB, versus 42.7 tok/s plain, using a custom engine that packs bf16 weights into a lossless 12-bit format (B12) with a Qwen3-0.6B draft model.
Setup is the user's own engine on Windows with ROCm PyTorch, batch-invariant GEMV kernels and seeded Gumbel-max sampling; speculative output matched plain decoding on 11 of 11 prompts.
On 50 longer prompts of 512 tokens the speculative run reached 73.0 tok/s with 50/50 matches. The user notes llama.cpp Q8_0 with a draft is about 1.4x faster (98.8 tok/s ROCm, 95.0 Vulkan) but lossy, matching plain output on only 7 of 11 prompts.
Compare this run
Medians of plain runs: one device, one request, no speculative decoding, the whole model in the device's memory. The report above is one setup, so read it beside them and not as one of them.
Qwen3 8B at 4-bit by hardware
- NVIDIA RTX 508016 GB130.6 t/s · one run
- 104.2 t/s · 2 runs
- 98.8 t/s · one run
- 81.2 t/s · one run
- Intel Arc Pro B7032 GB77.7 t/s · one run
- AMD RX 6700 XT12 GB61.0 t/s · one run
Bars are medians on one scale, 0 to 130.6 tokens/s. Each dot is one run and opens its report. Press ▶ to watch text arrive at a row's speed.
Models most run on the AMD RX 9070 XT 16GB
- Qwen3 8B4-bit98.8 t/s · one run
- Qwen3.5 9B4-bit7.9 t/s · one run
- Qwen3.5 9B6-bit62.0 t/s · one run
Bars are medians on one scale, 0 to 98.8 tokens/s. Press ▶ to watch text arrive at a row's speed.
Everything reported on the AMD RX 9070 XT 16GB →Every Qwen3 report →Estimate Qwen3 on this card →