llamaperf

Qwen3.8 125B (6B active) Flash-Next

on 2× NVIDIA RTX 3090 · llama.cpp · 261,888 ctx

NVIDIA hardware
Sep 10, 2026
Throughput
34.3 t/s gen · 223.7 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
UD-Q4_K_XL (GGUF)
KV cache
f16
Flash Attention
on
MTP (Multi-Token Prediction)
on

Summary

User reports prefill improvements of 2.2-2.5x on Qwen3.8-Flash-Next across 8k, 37k, and 119k contexts on 2x RTX 3090 with dual Broadwell Xeon and DDR4, with decode performance unchanged. Setup is llama.cpp on a custom branch flashnext-e06, a 150-slot expert cache moved off GPU during prompt processing, 261k context, and an f16 KV cache. Hardware changed from 4 DIMMs to 6x32GB DDR4-2133 ECC. Prefill is 223.7 t/s at 8k (was 99.9), 212.6 t/s at 37k (was 88.1), and 206.5 t/s at 119k (was 81.3). Decode is 34.3 t/s at 8k, 41.2 t/s at 37k, and 33.9 t/s at 119k. Time to first token improved from 82s to 37s at 8k, 424s to 176s at 37k, and 1461s to 575s at 119k. A quality screen showed no regression, and MTP acceptance is 0.79-0.83.