llamaperf

Qwen3.8 122B (10B active) Flash-Next Heretic

on M5 Max 128GB · llama.cpp · 4,096 ctx

Apple hardware
Oct 9, 2026
Throughput
20.4 t/s gen
Quant
Q8_0 (GGUF)
KV cache
Q4_0
System RAM
128 GB

Summary

User reports Qwen3.8-Flash-Next-Heretic Q8_0 at 20.45 tok/s on an M5 Max 128GB. Setup is llama.cpp (closed PR #27739 fork) with Q8_0 and Q4_0 KV cache, -ngl 16, 4096 context, 45.1 GiB wired, MTP draft head enabled. Measured on 12 varied prompts with 24 warm-up queries discarded. The model is 122B total / 10B active MoE, 197 GB across 5 shards. A 262144-token server ran on llama.cpp v0.6.0 with ~32 tok/s prefill and decode falling to ~3.6 tok/s at 10k tokens. Under memory pressure a larger GPU tier wins: -ngl 24 at 10.29 tok/s with 15 GiB held by other work versus -ngl 6 at 5.75 tok/s with 45 GiB held. Contiguous layer placement beats scattered expert placement by 28.6%. MTP acceptance is 51-83%, helping +44% when experts are cached in RAM but costing 16-33% when they stream from SSD. Stock llama.cpp was 11-19% faster than the fork at 4k/64k/256k, and v0.6.0 another ~7% faster. No quality measurement was run for this file.