llamaperf

Qwen3.8 125B (6B active) Flash-Next

on 6× NVIDIA V100 16GB · 1cat-vllm · 262,144 ctx

Tone: positiveNVIDIA hardware
Sep 20, 2026
Throughput
43.4 t/s gen · 4679.0 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
MTP (Multi-Token Prediction)
on

Summary

User reports Qwen3.8 Next running on a 6x V100 16GB setup with TP2 PP3, reaching up to 4,679 t/s prefill at 16K input and 43.38 t/s generation with MTP enabled. Setup uses the 1cat-vllm engine with speculative decoding limited to speculative=1; KV cache is 8.78 GiB, 531,288 tokens, supporting 2.03x concurrency at 262,144 tokens per request. MTP nearly doubles generation throughput versus base (22.59 t/s average without MTP). A 20-minute gpu-burn stress test peaked at 65°C and stabilized at 64°C with fans at 76%.