Qwen3.8 125B (6B active) Flash-Next
on NVIDIA RTX 5060 Ti 16GB · llama.cpp
Summary
User reports Qwen3.8-Flash-Next IQ3_S (83.6 GB) decoding at ~15 tok/s (~67 ms/token) in steady state on an RTX 5060 Ti 16 GB with 15.38 GiB RAM and a Gen3-class NVMe.
Setup is a patched llama.cpp (Sabah-MoE-Accelerator) that streams the expert bank from disk into a 9.8 GB VRAM slab with unbuffered I/O; the 28.8 GB per-layer token embedding is left on CPU via -ot, peak VRAM is 15.36 of 16.31 GB, and host RAM stays flat at 4.78 GiB.
Time to first token is 11.2 s for a 53-token prompt (12.68 GB cold fill at 1.11 GB/s), so a 256-token answer is 7.1 tok/s end to end and 10.4 tok/s over the decodes alone; warm-up runs 79-204 ms/token. The author notes speculative decoding and larger batches were 2.8-7.7x slower on this rig, and a second GTX 1080 was measured as no faster than disk.