llamaperf

Qwen3.8 125B (6B active) Flash-Next

on 2× NVIDIA RTX 3090 · llama.cpp · 261,888 ctx

Tone: positiveNVIDIA hardware
Sep 7, 2026
Throughput
25-29 t/s gen · 350.0 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
UD-Q6_K_XL (GGUF)
KV cache
F16
System RAM
192 GB
VRAM reported
24 GB

Use cases

coding

Summary

User reports 25-29 t/s decode at short and mid context on 2x RTX 3090 over PCIe 3.0 with dual Xeon E5-2696 v4 and 188 GB usable DDR4-2133 LRDIMM, up from ~17 t/s before. Setup is llama.cpp with the Qwen3.8-Flash-Next UD-Q6_K_XL GGUF, f16 KV cache, full 261,888 context, all 48 expert layers pinned in host RAM and everything else on GPUs, using PR #27861 GPU-resident LRU expert cache with 135 cache slots per layer. Decode holds ~17 t/s at 131k depth, prefill is unchanged at ~350 t/s on a 26k prompt, and the cache hit rate is 80-85% on code. Dropping ubatch from 2048 to 512 freed ~5 GB per GPU, and prefill is slower on long prompts. Thread count, poll, CPU masks, q8 KV, lazy PLE, n-gram drafts on prose, MTP at temp 0.7, and more than 2 cache uploads per step did not help. Cache size is ~100 MB per slot per GPU on Q6, and the user plans to test UD-Q4_K_XL next.