llamaperf

Qwen3.8 125B (6B active) Flash-Next

on 4× NVIDIA RTX 3090 · vLLM · 262,144 ctx

NVIDIA hardware
Sep 27, 2026
Throughput
153.5-186.7 t/s gen · 5123-5410 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
W4A16 (AutoRound)
KV cache
FP8
System RAM
96 GB

Use cases

agentictool-uselong-contextvisioncoding

Summary

User reports Qwen3.8-Flash-Next (125B MoE, ~6B activated) served on four RTX 3090s at full 262,144-token context, with an 806,792-token FP8 KV pool. Setup is vLLM 0.30.0 (v2.2.0 release, 58 commits over the public tag) with a W4A16 AutoRound checkpoint, TP2 x PP2 plus expert parallel, MTP K=3, and FP8 KV cache with a scale sidecar. Cold prefill runs 5,123-5,410 t/s from 10K to 200K tokens; decode with thinking on is 153.5-186.7 t/s (median of three prompts per depth, one boot). The release is a reliability update at parity with v2, not faster: decode lands at 0.973-1.040 of v2 in every cell. On a rented 4x RTX 3090 without peer-to-peer, a cold 54,713-token prompt took 13.8 s (about 4,000 t/s).