Local model performance on your hardware

generation:

50.9 tokens/s

prompt processing (prefill):

511.9 tokens/s

quant:

MLX affine 4-bit, group 64 (MLX)

User reports Grug-12B at 50.9 t/s generation and 511.9 t/s prompt processing on an M4 Max with 64 GB unified memory. Setup is oMLX with MLX affine 4-bit, group 64, at a pp8192/tg128 workload, with 8.26 GB peak memory. The same row lists TTFT 16003.6 ms, TPOT 19.80 ms and E2E 18.540 s; the 8192-token prompt is a workload label, not a configured maximum context.

generation:

103.9 tokens/s

prompt processing (prefill):

1807.2 tokens/s

quant:

MLX affine 4-bit g64; gate modules 8-bit (MLX)

User reports Grug-35B-A3B at 103.9 t/s generation and 1807.2 t/s prompt processing on an M4 Max 64GB. Setup is oMLX with MLX affine 4-bit group 64, gate modules at 8-bit, at 1024 prompt and 128 generated tokens. The run is single-request, with 566.6 ms TTFT, 9.70 ms TPOT, 1.805 s end-to-end and 19.35 GB peak memory; warm-up, repetition count and KV precision are not stated.