Qwen3.8 125B (6B active) Flash-Next
on 2× CMP 170HX · vLLM
Use cases
coding
Summary
User reports Qwen3.8 Flash-Next on two CMP 170HX cards, with aggregate decode of 226 tok/s for code, 198 for JSON and 157 for prose at one concurrent request.
Setup is a patched vLLM with the W4A16 AutoRound checkpoint, pipeline parallelism and native MTP speculative decoding; the n-gram table is memory-mapped from NVMe and everything else fits on the two GPUs, which run over PCIe Gen2 x4.
Aggregate throughput rises to 840 tok/s for code at eight concurrent requests, where per-request decode was 116 tok/s with 231 ms to first token. Prefill is 6,300-7,100 tok/s on 4k-64k prompts and about 5,900 tok/s at 128k; prose decode falls from 158 tok/s at 4k context to 142 tok/s at 128k. HumanEval scored 157/164 and GSM8K 1,271/1,319, with a full 262,144-token window.