llamaperf
Oct 6, 2026
Throughput
25.7 t/s gen
Quant
4-bit (MLX)
System RAM
256 GB

Summary

User reports Qwen3.8 Flash-Next at 25.67 t/s on an M3 Ultra 256GB. Setup is Rapid-MLX 0.13.2 with mixed 4-bit affine MLX quantization and a 2048-token prefill chunk, at roughly 102.8-103.8GB active unified allocation. Decode falls to 24.27 / 23.40 / 21.72 t/s at the 2048 / 8192 / 32768 prompt targets, with median TTFT of 2.262 / 9.236 / 44.659 seconds. Each row is the median of three batch-size-one runs with cold prefix cache and 256 requested decode tokens.