Local model performance on your hardware

generation:

1.8 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

User reports Qwen3.8 Flash-Next at 1.76 t/s generation on a Mac mini M1 16GB, streaming experts from the internal SSD. Setup is Slotstream 0.2.17 on MLX Swift with PipeNetwork's MLX 4-bit weights, one request, MTP off, a 512-token context and a 19-token prompt. The model's 105 GB sits on disk while the process peaks near 5 GB of memory. A second run gave the same speed.

generation:

95.6 tokens/s

quant:

Q4 (GGUF)

User benchmarks Qwen2.5-0.5B Q4 at 95.6 t/s decode with 18 ms TTFT on an Apple M1 16GB, averaged across defaults and normalized modes on the chat_turn workload. Setup is llama.cpp (llama-server) with Q4 GGUF weights, Metal backend, 1 warmup plus 4 measured reps per cell. LlamaStash leads raw llama-server (91.9 t/s / 20 ms), LM Studio (88.4 / 68) and Ollama 0.24.0 (79.6 / 102); in defaults mode LlamaStash reaches 99.0 t/s / 15 ms vs raw llama-server's 92.3 / 19, and normalized mode collapses the gap to under 1% (92.2 vs 91.5).

generation:

8.2 tokens/s

quant:

4bit (MLX)

User reports Gemma 4 E4B at around 8.2 tok/s on an M1 MacBook Air with 16GB unified memory. Setup is Unsloth Studio with a 4-bit MLX model and a 131.1k context window. The user says this speed is fine for chatbot use and asks for suggestions of 7B or 8B MLX models for coding and background tasks.