llamaperf
Oct 13, 2026
Throughput
28.0 t/s gen
Quant
int4 affine group64 (MLX)
System RAM
24 GB

Summary

User reports Edge0 at 28 t/s on a single Apple Mac mini M4 Pro with 24 GB unified memory. Setup is the Edge0 MLX backend with int4 affine group-64 weights, prediction-as-routing staged decode, whole-layer prefill and SSD-streamed experts, using a shared LRU of 64 expert bundles. The figure is the paper's single-device warm decode result for the released Edge0 8B K=8 low-memory profile, a median over three paired rounds; a separate LRU1024 variant reaches 31.8 t/s with about 0.9 GiB more memory.