Local model performance on your hardware

generation:

196.0 tokens/s

quant:

BF16 (safetensors)

User benchmarks Qwen3-Coder-30B-A3B-Instruct in BF16 and FP8 on a single MI300X 192GB, reporting 196 tok/s at 1 concurrent request for BF16 and 136 tok/s for FP8. Setup is vLLM 0.29.1 in the rocm/vllm container on AMD Developer Cloud, 1024 prompt tokens and 256 output tokens, vllm bench serve. The 1-request figures are medians of 3 warm runs. FP8 is 31% slower at 1 request but 35% faster at 64 concurrent (3442 vs 2555 tok/s). Accuracy on GSM8K (250 problems) was 96.4% BF16 vs 94.0% FP8. llama.cpp with Q4_K_M reached 187 tok/s at 1 user. Also ran gpt-oss-120b (72 tok/s at 1, 1423 at 64) and Qwen2.5-72B BF16 (26 tok/s at 1, 421 at 32). Total cost about $5.73.