AMD Instinct MI300X 192GB

generation:

173.0 tokens/s

quant:

BF16 (safetensors)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3-Coder-30B-A3B in BF16 on a single MI300X 192GB, reporting 173 tok/s decode on a 32K prompt with SGLang defaults. Setup is SGLang 0.5.21 with a 131K context window, prefix caching off, and 256-token replies; vLLM 0.29.1 with AITER attention reached 164 tok/s and llama.cpp (HIP) 108 tok/s with a 20.5 s first token. SGLang was fastest from 1 to 64 users; vLLM only pulled ahead at 128 users. Prefix caching was the only tuning knob that helped, adding 8 to 11 percent. The user notes vLLM without AITER attention collapsed to 27 tok/s total on 16 users with 32K prompts versus 133 on SGLang.

generation:

196.0 tokens/s

quant:

BF16 (safetensors)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3-Coder-30B-A3B-Instruct in BF16 and FP8 on a single MI300X 192GB, reporting 196 tok/s at 1 concurrent request for BF16 and 136 tok/s for FP8. Setup is vLLM 0.29.1 in the rocm/vllm container on AMD Developer Cloud, 1024 prompt tokens and 256 output tokens, vllm bench serve. The 1-request figures are medians of 3 warm runs. FP8 is 31% slower at 1 request but 35% faster at 64 concurrent (3442 vs 2555 tok/s). Accuracy on GSM8K (250 problems) was 96.4% BF16 vs 94.0% FP8. llama.cpp with Q4_K_M reached 187 tok/s at 1 user. Also ran gpt-oss-120b (72 tok/s at 1, 1423 at 64) and Qwen2.5-72B BF16 (26 tok/s at 1, 421 at 32). Total cost about $5.73.

User reports Qwen3.8-27B at 495 t/s on a single MI300X at 1M context, up from 311 t/s. Setup is dstack's toolkit with source-level patches to SGLang's AITER attention backend. The figure is for four concurrent users at 10k in / 1.5k out, with p50 TTFT under 1.5s.