Local model performance on your hardware

prompt processing (prefill):

71.0 tokens/s

quant:

PQ2_0 (GGUF)

User compares low-bit GGUF runtimes on CPU (ARM Neoverse-N1 server and AVX2 x86 VMs) against reference outputs. Ternary-Bonsai 1.7B prompt processing reaches 71 t/s with PQ2_0 in ik_llama.cpp with -rtr, versus 28 t/s in Prism and 2 t/s for Q2_0 g64 in Prism. Setup is ik_llama.cpp with -rtr, which is off by default; without it ik_llama.cpp is often the slowest. llama.cpp merged an AVX2 Q2_0 kernel on Oct 13 (~6x faster) and ik_llama.cpp added Q2_0 the same day (~15x faster than Prism). For the 27B, the smaller PTQ1_0 file is ~4.4x slower than PQ2_0 for prompt processing in Prism on AVX2. The Q2_0 file marked recommended on the Ternary-Bonsai-1.7B card does not load in any of the three runtimes. BitNet b1.58 2B4T on bitnet.cpp main is wrong on CPU, with perplexity ~6x the HF transformers reference on x86 and ???? output on ARM.

generation:

73.0 tokens/s

User reports Bonsai 1.7B at ~73 tokens/sec generation on an Apple M4 Pro CPU-only, pulling 11.7 Watts (about 0.31 Joules per token). Setup is BitRun v0.2.0-beta with GGUF ternary weights, running entirely on CPU memory. On a Core i5-12600K, Bonsai 1.7B generates at 85 tokens/sec pulling around 22 Watts; the 8B Bonsai model reaches 21 tokens/sec on the M4 Pro at the same 11.7 Watts, and BitNet 70M clocks almost 600 tokens/sec.

generation:

9.1 tokens/s

User reports Bonsai 1.7B at about 9.1 t/s on an Intel N97 CPU drawing 12 watts. The model solved simple physics problems in this run.

generation:

25-30 tokens/s

quant:

1-bit

User reports 25-30 t/s decode in a chat UI and 32 t/s raw in a WebGPU browser inference engine, with prompt processing of 1489 tokens in about 25 seconds. The model is natively 1-bit, 27B parameters in 3.8 GB, running at 3072 tokens of context on a 6 GB card. The user also mentions smaller tiers (Qwen3.5 0.8B, 2B, 4B) and a vision tower.

generation:

73.0 tokens/s

quant:

Q1_0

kv:

kvarn4

User reports 43.4 t/s with a q8_0 KV cache and 73.0 t/s with a kvarn4 KV cache at 120K context. The q8_0 cache used about 13.1 GB total and the kvarn4 cache about 9.8 GB total. The kvarn4 cache is 68% faster and uses 3.3 GB less VRAM.