Local model performance on your hardware

generation:

28.0 tokens/s

quant:

int4 affine group64 (MLX)

User reports Edge0 at 28 t/s on a single Apple Mac mini M4 Pro with 24 GB unified memory. Setup is the Edge0 MLX backend with int4 affine group-64 weights, prediction-as-routing staged decode, whole-layer prefill and SSD-streamed experts, using a shared LRU of 64 expert bundles. The figure is the paper's single-device warm decode result for the released Edge0 8B K=8 low-memory profile, a median over three paired rounds; a separate LRU1024 variant reaches 31.8 t/s with about 0.9 GiB more memory.

generation:

20.4 tokens/s

quant:

int4 affine group64 (MLX)

User reports Edge0 at 20.4 t/s on a single Apple Mac mini M4 Pro with 24 GB unified memory. Setup is the Edge0 MLX pipeline with int4 affine group-64 weights, a K=4 production profile, and SSD-streamed experts; allocated context is not reported. The figure is the paper's single-device warm decode result, a median over three paired rounds, and the timed output count is absent. The same paper reports prefill at 113 t/s cold and 140 t/s warm.