Gemma 3 4B
on NVIDIA RTX Pro 6000 Blackwell · vLLM
Summary
User reports Gemma 3 4B at 132.52 t/s on an RTX 6000 Pro Blackwell, a 3.34x speedup over Ollama.
Setup is vLLM (dev build) with BF16 weights and MTP speculative decoding using 3 draft tokens, single request, ~512-token prompt and 64 output tokens, averaged over 3 runs.
The user also tested Qwen3-VL 8B (Abliterated) with 1 MTP layer, noting Gemma 3's 3 MTP layers explain its larger gain; going beyond the MTP layer count as draft tokens does not help.