Local model performance on your hardware

generation:

115-120 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

User reports Qwen3.5-35B-A3B at roughly 115-120 tok/s on a Mac Studio M1 Ultra 128GB plus an RTX 3090 24GB over a Thunderbolt 5 eGPU. Setup is llama.cpp with Q4_K_M, about 17GB of the model on the 3090 and about 5GB on the Mac, with the KV cache quantized to Q8. The user says the Q8 KV cache lets them run about 200K context entirely in VRAM, and that this model replaced a two-model agentic setup.

generation:

69.7 tokens/s

prompt processing (prefill):

1172.9 tokens/s

quant:

Q4_K_M (GGUF)

User reports 69.73 t/s generation and 1172.93 t/s prompt processing. Source evidence (verbatim): Source: https://blog.majid.info/llama-bench/utumno.txt # Mac Studio M1 Ultra 128GB: original Gemma4 E4B Q4_K_M llama-bench log First-hand author log linked from https://blog.majid.info/llama-bench/ (April 13, 2026 article). Measured hardware: Mac Studio M1 Ultra 128GB. Gemma4 E4B Q4_K_M GGUF, llama.cpp ff5ef8278 (8763), backend Metal/BLAS. pp512 1172.93 ±0.36 tokens/s; tg128 69.73 ±0.19 tokens/s. 16 CPU threads; recommendedMaxWorkingSetSize 103079.22 MB is a runtime recommendation, not resident memory. Unified system RAM 128GB, GPU core count unknown. Command names --hf-repo unsloth/gemma-4-E4B-it-GGUF; exact tested quantization repository https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF ; tested revision unknown. Log reports 4.62 GiB model size and 7.52B counted parameters; size is not resident memory. Context depth, KV quantization, measured power, driver version and explicit speculation/concurrency declarations unreported. Raw source: https://blog.majid.info/llama-bench/utumno.txt .

generation:

11.1 tokens/s

prompt processing (prefill):

68.1 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User reports Llama 2 70B Chat at 11.1 t/s generation and 68.1 t/s prompt processing on a Mac Studio M1 Ultra with 128GB unified memory. Setup is llama.cpp build 2164 with Q4_K_M quantization, 512-token context, F16 K and F16 V cache, all 81 layers offloaded to the Metal GPU. The figures are the runtime's separate prompt-eval and eval counters, not wall-clock request throughput; prompt evaluation was 56 tokens and token evaluation was 333 runs.

generation:

65.5 tokens/s

prompt processing (prefill):

392.4 tokens/s

quant:

MXFP4 (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports gpt-oss 120B MXFP4 at 65.5 t/s generation and 392.4 t/s prompt processing on a Mac Studio M1 Ultra with 128 GB unified memory. Setup is llama.cpp build 6470 with MXFP4 and F16 K/V cache, one Metal device, all layers offloaded, batch and microbatch 4096, mmap disabled. The two figures are separate synthetic tests, not one 65536-input/128-output request: prompt processing ran 65536 tokens and generation 128 tokens at depth 0, so generation is not measured at 65K depth.

generation:

370.7 tokens/s

prompt processing (prefill):

12605.1 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports SmolLM-135M-Instruct-4bit at 370.7 t/s generation and 12605.1 t/s prompt processing on an M1 Ultra 128GB. Setup is mlxcel 0.7.0-beta.1 with 4-bit weights and FP16 KV cache, one Metal GPU, 512 input tokens and 128 output tokens, greedy, no speculative decoding. Prefill excludes loading, tokenization and client delivery, so it is not end-to-end TTFT; the decode rate covers 128 output tokens over 127 forward steps.

generation:

81.2 tokens/s

prompt processing (prefill):

518.0 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports Qwen3 8B 4-bit at 81.2 t/s generation and 518 t/s prompt processing on an M1 Ultra 128GB Mac Studio. Setup is mlxcel 0.7.0-beta.1 with FP16 KV cache, one Metal GPU, 512 input tokens and 128 output tokens, greedy temperature 0. The decode figure covers 128 output tokens over 127 forward steps; prefill excludes loading, tokenization and transport, so it is not end-to-end TTFT.

generation:

116.9 tokens/s

prompt processing (prefill):

942.3 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports Qwen3 4B 4-bit at 116.9 t/s generation and 942.3 t/s prompt processing on an M1 Ultra 128GB Mac Studio. Setup is mlxcel 0.7.0-beta.1 with FP16 KV cache, one request of 512 input tokens and 128 output tokens, greedy temperature 0, no speculative decoding. The decode figure covers 128 output tokens over 127 forward steps; prefill excludes loading, tokenization and transport, so it is not end-to-end TTFT.

generation:

69.7 tokens/s

prompt processing (prefill):

1172.9 tokens/s

quant:

Q4_K_M (GGUF)

User reports Gemma 4 E4B at 69.7 t/s generation and 1172.9 t/s prefill on an M1 Ultra 128GB. Setup is llama.cpp at Q4_K_M, one request, speculative decoding off. Fazal Majid's llama-bench run published 2026-04-13. A source-reported measurement, not independently reproduced.

generation:

25.1 tokens/s

prompt processing (prefill):

153.3 tokens/s

quant:

4bit (MLX)

mtp (multi-token prediction):

off

User reports Qwen3.8 27B at 25.11 t/s generation and 153.27 t/s prefill on a Mac Studio M1 Ultra 128GB, with a 512-token prompt and 128 generated tokens. Setup is mlxcel 0.7.0-beta.1 with the mlx-community 4-bit checkpoint, one request, speculative decoding off. The figures are one timed pass from the mlxcel benchmark results. The harness counts the first token in prefill, so the decode figure would read about 24.91 t/s over 127 intervals.

generation:

16.0 tokens/s

quant:

IQ3_XXS

User reports a patched engine improved speed from 5-6 to 15-16 tok/s. Setup uses the Unsloth UD-IQ3_XXS quant. The user notes a wired limit at 120GB.

generation:

31.2 tokens/s

prompt processing (prefill):

530.2 tokens/s

quant:

IQ4_NL (GGUF)

User benchmarks a custom llama.cpp branch with faster Metal inference and n-gram SSD offload at pp512 530.18 t/s and tg128 31.17 t/s with resident n-gram. The model is 68.37 GiB and 125.74 B params. With SSD read mode, pp512 is 197.92 t/s and tg128 is 28.47 t/s.