llamaperf

Qwen3.8 125B (6B active) Flash-Next

on NVIDIA RTX 4070 · llama.cpp

Tone: positiveNVIDIA hardware
Sep 15, 2026
Throughput
20.6 t/s gen · 300-350 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
4.27 bpw (GGUF)
MTP (Multi-Token Prediction)
on
System RAM
64 GB
VRAM reported
12 GB

Use cases

codingsummarizationcreative-writing

Summary

User reports Qwen3.8-Flash-Next (125B-A6B MoE) at 20.65 t/s generation on an RTX 4070 12GB with 64GB DDR5-5600 and Gen4 NVMe on Linux. Setup is llama.cpp with AtomicChat's 4.27 bpw GGUF, n-gram SSD offloading, --fit-target 512, and an MTP variant (PR #28243 + compact shared-Q4_K_M head) with -ncmoe 45, yielding 77-96% acceptance. Prompt processing is 300-350 t/s. Baseline was 6 t/s; master branch reached 19.35 t/s. User notes MTP gives limited gains at low VRAM because layers must be freed for the MTP head.