llamaperf

Qwen3.8 125B (6B active) Flash-Next

on 6× NVIDIA RTX 3090 · vLLM · 250,000 ctx

Tone: mixedNVIDIA hardware
Oct 12, 2026
Throughput
57.6 t/s gen · 6574.0 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
GPTQ-INT4 (GPTQ)
KV cache
int8
System RAM
187 GB

Summary

User reports Qwen3.8-Flash-Next at 57.6 t/s decode and 6574 t/s prefill on six RTX 3090s. Setup is vLLM with a GPTQ-INT4 checkpoint and int8 KV cache, 250k context, PLE table on SSD, pipeline parallelism across six cards. Decode falls to 29.4 t/s per user with seven concurrent sessions (206 t/s aggregate); llama.cpp-multigpu holds 30 t/s per user at five sessions. NVFP4 checkpoints are 10-12% slower than GPTQ on these cards. Load to healthy is 4-5 minutes.