Local model performance on your hardware

generation:

11.6 tokens/s

prompt processing (prefill):

49.1 tokens/s

quant:

Q2_0 (GGUF)

User benchmarks Ternary-Bonsai-2-4B Q2_0 on an Intel Core i5-13420H (4 P-cores, AVX-VNNI) under ik_llama.cpp, CPU only. The offline-repacked Q2_0_R8 build reaches 49.08 t/s prompt processing (pp512) and 11.55 t/s generation (tg128). Native Q2_0 gives 34.46 t/s pp512 and 9.56 t/s tg128; runtime repack with -rtr gives 49.99 t/s pp512 and 11.64 t/s tg128. The PR reports roughly +45% prompt processing and +22% token generation. Perplexity at n_ctx=128 over 3 chunks is 13.0721 native, 13.0763 with -rtr and 13.0763 for Q2_0_R8.

generation:

28.0 tokens/s

prompt processing (prefill):

31.2 tokens/s

quant:

PTQ1_0 (GGUF)

User reports Bonsai 2 27B at 28.0 tok/s on an RTX 4060 Laptop 8GB, with prompt reading at 31.2 tokens/s. Setup is the Kindi 0.3.0 image with a PTQ1_0 GGUF at 24,576 context, leaving 1,367 MiB free. Decode falls from 28.0 tok/s at position 100 to 25.2 tok/s at 24,000; a verify run passed token-exact against llama.cpp at 29.3 tokens/s.

generation:

57.4 tokens/s

quant:

PQ2

User benchmarks Bonsai-2 27B PQ2 at 57.4 tok/s on an RTX 5060 Ti 16GB, where it finished 13 of 14 agent tasks and fixed 10, with no malformed tool calls. The run is part of a 39-model, 50-prompt comparison graded by four frontier models, with 3k-token outputs on a 5800X3D and 64GB RAM. Three of four judges named Bonsai-2 their favourite; judges flagged weak coding (1.5) and honesty (4.5), and it claimed no tests existed without running anything. Underdog-Saluki 27B scored the highest mean (6.42) at 40.9 tok/s, Gemma-4 12B Q6 reached 92.0 tok/s, and LFM3 1.7B was fastest at 192 tok/s but last overall. Rankings are trustworthy to about 0.4 points.

generation:

43-44 tokens/s

quant:

PTQ1_0 (GGUF)

kv:

q4_0

User reports a Qwen3.8-27B derivative (Swift Bonsai 2 27B Katana + MTP) generating around 43-44 t/s on an RTX 3060 12GB, peaking at 45.74 t/s over a three-second window. Setup is the Prism fork of llama.cpp with a PTQ1_0 ternary GGUF and q4_0 KV cache at 128,000 tokens, MTP speculation with a draft limit of 1, all model layers on the GPU and the vision projector on the CPU; the whole process used 9,144 MiB of VRAM. In daily use the user sees 28-40 t/s after a few prompts and 17-25 t/s much later with far more context. The user cites another author's Bonsai 2 PTQ1_0 + MTP tests on the same card at 50.1 t/s with MTP versus 39.8 without at 131K, and 26.86 versus 22.23 at 41.8K tokens filled. Figures are described as observations rather than controlled benchmarks.

generation:

45.0 tokens/s

quant:

ternary

User reports Bonsai-2 27B running at around 45 t/s on an Intel Arc B580 12GB. Setup is PrismML's llama.cpp fork with custom SYCL kernels and fusions, which the user wrote and PrismML merged (PR #356, about 7,700 lines). The models use ternary weights (-1, 0, +1). The user also trained small ternary models in cloud data centres that run at home on the B580: SmolLM2-360M shrunk from 690 MB to 162 MB at 1.4x the original perplexity and about 94% of benchmark scores, and SmolLM2-135M from 257 MB to 73 MB at about 1.5x and about 91%. They say these are not yet Bonsai quality, whose models keep about 99%. The models are not released yet.

generation:

300.0 tokens/s

User reports Bonsai 2 at 300 tok/s on an RTX 5060 Ti 16GB using a fork of llama.cpp. The user states the fork previously ran Bonsai 2 at 140 tok/s on an RTX 4070 Ti Super 16GB, and that after further optimization it reaches 140 tok/s on the 5060 Ti 16GB. The 300 tok/s figure is given as a correction for running without full context. The fork was also modified to run more than one session in parallel, as it previously ran only one.

generation:

500.0 tokens/s

quant:

ternary

User reports the release of Ternary Bonsai 2 (2B, 4B, and 9B) on Hugging Face. The 2B model runs entirely locally in the browser at up to 500 tok/s on an M4 Max using custom WebGPU kernels.

generation:

2.8 tokens/s

quant:

TQ1_0 (GGUF)

User reports running a 27B ternary model across two phones over a USB-C cable at ~2.8 tok/s. The setup uses llama.cpp RPC over a localhost tunnel created via Android USB accessory mode, with no Wi-Fi, root, or adb. The model is Ternary-Bonsai-2 (27B) in a Hadamard-rotated basis, run with a custom llama.cpp patch that applies rotation to activations. The two-phone pair is not yet faster than the fastest single phone, which achieves 5.62 tok/s on a Redmi, compared to 0.44 tok/s with the vendor's runtime. The requantization increases perplexity to 14.26 from a 14.05 reference (+1.5%). A lossless repacker from 2-bit ternary GGUF to GGML's TQ1_0 base-3 container reduces the model size from 6.5 GiB to 5.3 GiB (−18%).

generation:

22.3 tokens/s

prompt processing (prefill):

214.2 tokens/s

quant:

PQ2_0 (GGUF)

kv:

Q4

flash attention:

on

mtp (multi-token prediction):

off

User reports Bonsai 2 at 22.3 t/s generation and 214.2 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is PrismML-Eng/llama.cpp with PQ2_0 weights, Q4_0 K/V cache, full GPU offload, flash attention, one slot and 262144 context on a single Ryzen AI MAX+ 395 APU with 128 GB unified memory. The figure is the idle-peer fixed-prefix run with 524 input and 256 output tokens; the source gives 22.25989819108211 t/s and 214.23028774725407 t/s.

generation:

5.6 tokens/s

quant:

2.06 bpw (GGUF)

User reports Ternary-Bonsai-2-27B at 5.6 tok/s text generation on a Xiaomi 15 phone, and about 2.8 tok/s when clustered across two Android phones (Xiaomi Redmi 100 Pro Max + ROG 7) over USB with a custom RPC split. Setup is a custom C++ engine forked from llama.rn, with weights compressed to 2.06 bpw (6.52 GiB) in GGUF and kept ternary in the Hadamard-rotated basis, transformed at runtime on the ARM CPU. The cluster run splits layers 4.36 GiB / 2.15 GiB and is bottlenecked by the slower phone. Perplexity is 14.26 against a 14.05 reference, and max temperature sits around 30°C.

generation:

48.6 tokens/s

prompt processing (prefill):

1480.0 tokens/s

quant:

ternary (GGUF)

kv:

q4_0

mtp (multi-token prediction):

on

User reports ternary Bonsai 2 27B at 48.6 t/s plain decode on an Intel Arc B580 12GB. Setup is a custom llama.cpp SYCL branch with ternary weights, q4_0 KV cache and 128K context, with no speculation for this figure. With MTP, n-gram speculation and a trimmed draft head the model reaches 85 t/s writing code, 79 t/s in chat and 89 t/s at 115K tokens of history, with prompt reading at ~1,480 t/s. The user's own draft head was used for the video, so public results may be a few percent lower.

generation:

20.5 tokens/s

prompt processing (prefill):

126.9 tokens/s

quant:

PQ2_0 (GGUF)

kv:

F16

flash attention:

on

User reports a sourced third-party community benchmark of Bonsai 2 27B PQ2_0 on one Apple Mac mini M4 Pro with 64 GB unified memory. Prompt processing pp512 measured 126.89 +/- 0.02 tokens/second and token generation tg128 measured 20.53 +/- 0.01 tokens/second. Setup is llama.cpp PrismML fork (tag prism-b10709-9a9394a) with PQ2_0 GGUF and default F16 KV cache, 10 threads, -ngl 99, flash attention enabled, Metal/BLAS backend. The two figures are separate synthetic llama-bench tests with three repetitions each, not a paired chat request. Caveats: Spotlight indexing consumed approximately 1.5-2 CPU cores throughout the runs; an unrelated 664 MB embedding model reappeared between runs; no swap pressure was observed. The 6.70 GiB GGUF size is a file size, not measured resident or peak RAM. Peak RAM, configured maximum context, exact weight revision, binary hash, and exact measurement date are unknown.

generation:

40.0 tokens/s

User reports Bonsai 2 running at 40 t/s and upwards on an RTX 3060 12GB. No engine, quant or context length is stated. The user says it is fast but gets stuck reasoning for hours for mediocre results, and asks for alternatives or tweaks.

generation:

42.3 tokens/s

prompt processing (prefill):

895.0 tokens/s

quant:

PTQ1_0 (GGUF)

kv:

q4_0

User reports Bonsai 2 27B (PTQ1_0 ternary) at 42.3 t/s plain generation and 895 t/s prompt reading on an Intel Arc B580 12GB. Setup is llama.cpp's llama-bench (tg128 and pp512) on the SYCL backend with PTQ1_0 GGUF weights. With speculation, fresh code answers reach 90 t/s, renames 365 t/s, edits 258 t/s. At 48K context new code runs 63 t/s and edits 34 t/s; at 115K, 44 and 27 t/s. The user also compares against the Vulkan backend and PrismML's own fork, and reports KL divergence of 0.00022 against reference logits.

generation:

34.5 tokens/s

quant:

PQ2_0 (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

User reports Bonsai 2 27B (Ternary) at 34.5 t/s generation on an Apple Mac Studio M4 Max 128GB. Setup is llama.cpp (Prism fork build 10683) with PQ2_0 ternary GGUF, Metal, flash attention on, -ngl 999, 131072 configured context, single request, temperature 0, no speculative decoding. The 7.2 GB figure is the model file size, not measured peak RAM. The 34.5 t/s is the arithmetic mean of three prose runs (35.49, 34.03, 33.87 t/s) over 2097 prompt tokens and 320 output tokens; a related code workload averaged 34.3 t/s. Prefill is not reported because TTFT includes request overhead. KV-cache type, peak memory, CPU core count and power draw are unknown.

prompt processing (prefill):

1600-1800 tokens/s

User reports Bonsai 2 27B running on an RTX 5060 Ti 16GB with NInfer, achieving prefill over 1000 t/s (dropping from an initial 1600-1800 t/s) and decode above 50 t/s at up to about 200k context. Setup uses a modified and optimised version of NInfer, adapted with the help of ChatGPT and Claude to run on the 5060 Ti. Prefill and decode figures are given as ranges rather than single values.

generation:

26.2 tokens/s

prompt processing (prefill):

171.8 tokens/s

quant:

PQ2_0 (GGUF)

kv:

q4_0

User asks for help running Bonsai 2 27B with Claude via a forked llama.cpp from Prism-ML, reporting a Jinja template error ("System message must be at the beginning") when using Claude, while the DeepSeek harness works fine. Setup is llama.cpp (llama-server.exe) with a PQ2_0 GGUF quant, Q8_0 mmproj, q4_0 KV cache, 200000 context, -ngl 99, flash attention on, and 24 threads. The log shows a prompt eval of 171.77 t/s over 385 tokens and generation of 26.23 t/s over 200 tokens; the user did not state the GPU.

generation:

80-90 tokens/s

quant:

2-bit ternary (GGUF)

kv:

q4_0

User reports Ternary Bonsai 2 27B running on an Intel Arc B580 12GB at 128K context entirely in VRAM, with about 90 t/s writing new code at temperature 0 (about 80 at temperature 0.6), 250 to 370 t/s on code edits with MTP plus n-gram drafts, and 42 t/s plain. Setup is a custom llama.cpp fork with ternary weights repacked to 2 bit, q4_0 KV cache, and decode attention on XMX; the fork is compared against PrismML's basic SYCL path on the same card, model and flags. At 48K tokens of history it still does 63 t/s, and 44 at 115K. Against PrismML's fork: prompt reading 207 vs 895 t/s, plain generation 21 vs 42, generation at 32K depth 8 vs 36, chat with MTP 38 vs 88. Against Gemma 4 12B on the same card, Gemma is a bit better at code (HumanEval+ 152 vs 141 / 164), they tie on GSM8K, and tool calling is close; with Google's QAT assistant as MTP drafter, Gemma 4 12B goes from ~40 to ~100 t/s on new code (edits 64 -> 168).

generation:

29.3 tokens/s

quant:

PTQ1_0 (GGUF)

User reports Ternary Bonsai 2 27B at 29.27 t/s on an RTX 5060 Laptop 8 GB under Windows, using a Prism llama.cpp build. Setup is llama.cpp with PTQ1_0 (5.53 GiB), temp 0, seed 42, a 2048 reasoning budget and a 3072 token cap on 50 MATH-500 questions. A −2 logit bias on "wait", "maybe" and "perhaps" (nine token ids) dropped accuracy from 44/50 to 43/50 and raised average tokens from 845.2 to 872.3; the biased run measured 29.28 t/s. The user calls it one deterministic data point, not proof.

generation:

52.2 tokens/s

prompt processing (prefill):

1210.6 tokens/s

quant:

PTQ1_0 (GGUF)

kv:

q8_0

User reports Bonsai 2 27B PTQ1_0 at 52.18 t/s generation and 1210.6 t/s prefill on an RTX 3080 10GB. Setup is llama.cpp (fork build prism-b10735-842b188, CUDA 12.8) with PTQ1_0 quant and q8_0 KV cache at 32K context, single card, batch 1, depth 0, -ngl 99 -fa on. User also benchmarks PQ2_0 at 61.38 t/s and Qwen3.8-27B-UD-IQ2_XXS at 44.46 t/s on the same card, and reports a context ladder up to 160K q4_0 at 9627 MiB peak VRAM. A LiveCodeBench v6 comparison gives PTQ1_0 32/50 pass@1 versus 16/50 for IQ2_XXS and 28/50 for PQ2_0.