llamaperf

Qwen3.8 125B (6B active) Flash-Next

on 2× CMP 170HX · vLLM

Tone: positive
Oct 19, 2026
Throughput
6300-7100 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
W4A16 AutoRound (AutoRound)

Use cases

coding

Summary

User reports Qwen3.8 Flash-Next on two CMP 170HX cards, with aggregate decode of 226 tok/s for code, 198 for JSON and 157 for prose at one concurrent request. Setup is a patched vLLM with the W4A16 AutoRound checkpoint, pipeline parallelism and native MTP speculative decoding; the n-gram table is memory-mapped from NVMe and everything else fits on the two GPUs, which run over PCIe Gen2 x4. Aggregate throughput rises to 840 tok/s for code at eight concurrent requests, where per-request decode was 116 tok/s with 231 ms to first token. Prefill is 6,300-7,100 tok/s on 4k-64k prompts and about 5,900 tok/s at 128k; prose decode falls from 158 tok/s at 4k context to 142 tok/s at 128k. HumanEval scored 157/164 and GSM8K 1,271/1,319, with a full 262,144-token window.