NVIDIA H100 80GB

generation:

168.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-4.7-Flash at 168 tok/s on a single NVIDIA H100 80GB, up from a 120 tok/s baseline with EAGLE3 speculative decoding. Setup is SGLang v0.5.6 with the FlashInfer backend, a 277 MB EAGLE3 draft head, 6 draft tokens proposed per step, and a 40% acceptance rate averaging 2.4 accepted tokens per verification step. At batch size 32 the same setup reaches 440 tok/s versus 259 tok/s baseline (1.70x), with per-request latency improving 1.30x. The user also reproduced the EAGLE3 paper's Llama-3.1-8B result at 1.25x versus the published 1.32x, and notes a SGLang token-counting bug that deflated measured throughput by about 35%.

quant:

BF16 (GGUF)

User reports post-training Qwen3.8 27B into Swift-Qwen3.8-27B, achieving 58% fewer thinking tokens and 1.95x speedup while maintaining accuracy. Training used an 8xH100 box with LoRA SFT, On-Policy Distillation, and a token penalizer targeting overthinking patterns. Benchmarks ran at xhigh thinking effort in BF16. GPQA-Diamond: 88.3% vs 88.4% base with 58% fewer tokens; LiveCodeBench v6: 81.6% vs 76.8%; Terminal-Bench 2.1: 65.8% vs 66.7%; MMLU-Pro: 85.0% vs 85.5%; C-Eval: 90.6% vs 90.0%; IFBench: 71.8% vs 73.5%; AIME 2026: 94.0% vs 98.7%; HMMT: 96.0% vs 99.3%; ERQA: 66.3% vs 67.5%. Token savings hold across efforts: xhigh 41%, medium 23%, low 26%.

generation:

193.0 tokens/s

quant:

W4A16-FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash at 193 t/s on Hopper. Setup is vLLM with Canada-Quant quants and patched MTP code. User also reports roughly 400 t/s with 4 concurrent threads.

User reports a model based on Qwen3.5-4B, trained on 8xH100 for 3 days. It supports Safetensors, GGUF and MLX weights, requires as little as 4GB VRAM, and is available in GPTQ, W8A8, FP8, Q4 and Q6 quantizations. It has been tested with vLLM, SGLang and llama.cpp.

generation:

125.3 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Gemma 4 31B dense with MTP and DFlash speculative decoding, alongside Gemma 4 26B-A4B MoE (25.2B total, 3.8B active). At concurrency 1 the dense baseline runs 40.3 tok/s, MTP 125.3 tok/s and DFlash 122.1 tok/s, making MTP 3.11x and DFlash 3.03x faster than baseline. At concurrency 16 the dense baseline reaches 375 tok/s, MTP 953 tok/s and DFlash 725 tok/s. For the MoE at concurrency 1 the baseline runs 177.1 tok/s, MTP 264.2 tok/s and DFlash 306.4 tok/s. At concurrency 16 the MoE baseline reaches 975 tok/s, MTP 1808 tok/s and DFlash 1957 tok/s. Coding, math, STEM and reasoning benefited more.

generation:

45.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-27B at 45 t/s on a rented GPU instance with 2x H100s and 160 GB of VRAM. Setup is vLLM at 128K context, serving multiple agents hitting the API at once, including Claude Code, QwenCode and social media bots. The instance costs about $0.90/hr, with $120 spent last month. The user says the model outperformed a 120B model in tests.