Local model performance on your hardware

generation:

50.0 tokens/s

quant:

BF16 (safetensors)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at BF16 generating about 50 t/s single user on six RX 7900 XTX GPUs. Setup is vLLM with tensor parallel 6, achieved by padding the model architecture numbers with zeroes to make them divisible by six, at a 256k context window. Aggregate throughput is 200 t/s with 8 concurrent prompts. GPU KV cache size is 520,784 tokens, giving a maximum concurrency of 1.99x for 262,144 tokens per request.

generation:

100.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DiffusionGemma 26B-A4B-it at about 100 t/s generation on 4x RX 7900 XTX, or 45-60 t/s total including prompt processing. Setup is vLLM on the dgemma branch with the diffusion sampler entropy_bound, a 152,671-token KV cache, and max concurrency 1.16x at 131k context.

generation:

58.0 tokens/s

prompt processing (prefill):

83.0 tokens/s

quant:

FP16 (safetensors)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 E4B at 57.96 gen tok/s and 82.96 prompt tok/s on an RX 7900 XTX. Setup is vLLM with ROCm on the default path. Source is flexinfer.ai.