- reported speed:
- 16384.0 tokens/s prompt processing
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
Benchmark of llama.cpp PR #22675 for Mamba-2 prefill acceleration on Nemotron-Nano-9B-v2. Table shows speedups at various ub values; pp16384 tok/s reported. Also mentions DeepSeek-V4-Flash-UD-IQ2_XXS on M4 Max for a Metal FWHT kernel PR, but the primary benchmarked model is Nemotron-Nano-9B-v2.
- reported speed:
- 44.0 tokens/s generation · 564.0 tokens/s prompt processing
- quant:
- UD-Q8_K_XL
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
Benchmark on Bosgame M5 with RTX PRO 6000 Max-Q eGPU. Multiple quants tested: UD-Q8_K_XL (150.8 GiB, 44.0 t/s decode, 564 t/s prefill, draft acceptance 0.535), UD-Q4_K_XL (144.4 GiB, 48.4 t/s decode, 585 t/s prefill, draft acceptance 0.532), UD-Q2_K_XL (90.2 GiB, 59.5 t/s decode, 1513 t/s prefill, no drafter). Uses DSpark drafter ported from closed PR. Mixed GPU setup with CUDA0 and ROCm0.
- reported speed:
- 58.0 tokens/s generation
- quant:
- BF16
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
Decode speed increased from 18 tok/s to 58 tok/s with DFlash2 + XQA vs Flash2 + FA2 at 192K context. Lossless BF16 weights and KV.
- reported speed:
- 240.0 tokens/s generation
- quant:
- NVFP4
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
Optimized Qwen3.8-Next-Flash to ~240 t/s decode on a single RTX Pro 6000 Max-Q (300W) using a patched sglang. Techniques: further quantization of lm head and layers to FP8, kernel tuning, MTP config tuning. Theoretical bandwidth limit ~280 t/s. Model checkpoint: RadixArk/Qwen3.8-Flash-Next-NVFP4.
Benchmark of DeepSeek V4 Pro GGUF (794GB) on llama.cpp branch with expert offloading. Hardware: Epyc 9374F, 12x96GB DDR5, RTX PRO 6000 Max-Q. Prompt processing speeds range from 192 t/s (8K context) to 66 t/s (1M context). Generation speeds range from 11.73 t/s to 5.83 t/s. RAM usage 69.3% of 1152GB, VRAM usage 78986MiB of 96GB. Power ~500W during PP. Notes on mainline llama.cpp issues: memory waste, broken quantized KV cache, bugs with prompt cache reuse.
- reported speed:
- 7.1 tokens/s generation · 46.2 tokens/s prompt processing
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
Benchmark at various context depths. GPU VRAM usage 90815MiB/97887MiB. RAM usage 907.5GB/1152GB. CPU: Epyc 9374F. Power: GPU ~100W PP, ~150W TG; CPU+MB ~400W. Original model files, no conversion.