Qwen3.8 125B (6B active) Flash-Next
NVIDIA RTX 4060 Ti 16GB · llama.cpp · 100,000 ctx
- generation:
- 18.7 tokens/s
- prompt processing (prefill):
- 210.0 tokens/s
- quant:
- Q5_K_M (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash at 18.7 t/s generation on a Ryzen 7700 with 64 GB DDR5 and an RTX 4060 Ti 8 GB. Setup is a custom llama.cpp fork (myllama.cpp) running the 35B MoE model with 6B active parameters, CUDA and Windows specific, with no MTP support. An initial IQ4_XS quant gave 100 t/s prefill and 20 t/s generation, degrading to 15 t/s at 100K context. A switch to Q5_K_M dropped to 15 t/s prefill and 10-13 t/s generation from MMAP page faulting. A four-layer system followed: an unbuffered raw file reader with a RAM buffer for 2 slabs reached 150 t/s prefill, a RAM expert cache with decay factor (55-56 GB, covering 85-95% of requests) gave 14-15 t/s generation, memcpy from cache to buffer during prefill reached 215 t/s prefill, a 2 GB VRAM cache for top experts gave 18.0 t/s generation, and direct IO for embeddings gave 18.7 t/s.