- reported speed:
- 15.8 tokens/s generation · 81.8 tokens/s prompt processing
- quant:
- 4bit (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User compares MLX and llama.cpp on Qwen3.8-27B.
MLX uses mlx-community/Qwen3.8-27B-4bit at ~16.1GB, with prompt processing at 81.76 tok/s, generation at 15.81 tok/s, and peak memory of 16.39GB. llama.cpp uses unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M at 15.32 GiB and 27.32B params, with full Metal offload and Flash Attention enabled, giving prompt processing at 99.61 ± 0.44 tok/s and generation at 9.69 ± 0.34 tok/s.
llama.cpp is ~22% faster at prompt processing, while MLX is ~63% faster at generation.
- reported speed:
- 15.8 tokens/s generation · 81.8 tokens/s prompt processing
- quant:
- 4bit
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User compares MLX and llama.cpp on Qwen3.8-27B. MLX reaches 15.81 t/s generation and 81.76 t/s prompt processing. llama.cpp reaches 9.69 t/s generation and 99.61 t/s prompt processing.
The llama.cpp run uses the UD-Q4_K_M quant.
- reported speed:
- 16.0 tokens/s generation
- quant:
- IQ3_XXS
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports a patched engine improved speed from 5-6 to 15-16 tok/s.
Setup uses the Unsloth UD-IQ3_XXS quant.
The user notes a wired limit at 120GB.
- reported speed:
- 8.0 tokens/s generation · 30.0 tokens/s prompt processing
- quant:
- IQ3-XXS
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports DeepSeek-V4-Flash-0731 at about 8 t/s decode and about 30 t/s prefill on an M1 Max 64GB.
Setup is a patched llama.cpp with the IQ3-XXS quant at 104 GB and context limited to 64k.
- reported speed:
- 10.0 tokens/s generation · 165.0 tokens/s prompt processing
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports a run on a 64GB M1 Max.
No drafter was used.
User mentions n-gram and stripping embeddings as potential optimizations.
- reported speed:
- 31.2 tokens/s generation · 530.2 tokens/s prompt processing
- quant:
- IQ4_NL (gguf)
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User benchmarks a custom llama.cpp branch with faster Metal inference and n-gram SSD offload at pp512 530.18 t/s and tg128 31.17 t/s with resident n-gram.
The model is 68.37 GiB and 125.74 B params.
With SSD read mode, pp512 is 197.92 t/s and tg128 is 28.47 t/s.
- reported speed:
- 21.0 tokens/s generation · 83.0 tokens/s prompt processing
- quant:
- Q4 (mlx)
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports a roughly 2x speed boost for Qwen3.8 27B on Apple Silicon using the MTPLX framework.
User also reports Qwen3.6 35B A3B at about 55 t/s decode and about 300 t/s prefill, with a peak of 623 t/s.
- reported speed:
- 180.0 tokens/s prompt processing
- quant:
- Q4 (gguf)
- mtp (multi-token prediction):
- on
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
User reports a custom Q4 quant with spliced tensors from Unsloth and AtomicChat quants, with MTP enabled giving +70% decode at 22 btps.
Setup uses custom metal-optimized sparse attention, with prefill reduced to 170 tps at 4K and 150 tps at 256K context.
Q4_0 MTP matches Q8_0 acceptance rates at half RAM.
- reported speed:
- 17.5 tokens/s generation
- quant:
- Q4 (gguf)
Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.
text-generation
User reports Gemma 4 at 15-20 t/s, a range the user calls usable for simple tasks.