Bonsai 2 reports

generation:

188.0 tokens/s

quant:

Q4/Q5 (GGUF)

kv:

E8 4-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Ternary Bonsai 2 27B at 188 tok/s decode averaged over six mixed prompts on a single RTX 4090 under native Windows. Setup is NInfer, a from-scratch C++/CUDA engine, with Q4/Q5 weights, E8 4-bit KV cache, MTP plus n-gram speculative decoding, and 262K context per request. On edit-style prompts the model reaches 250 tok/s with MTP and 532 tok/s with MTP + n-gram; through the server on 7K-11K-token file edits with thinking on it goes 215 to 378 tok/s. Three concurrent requests give 360 tok/s aggregate. Against Prism's llama.cpp fork on the same card: 101 vs 77 tok/s decode and 3,061 vs 1,363 tok/s pp512. N-gram gains depend on repetition and are zero on free prose (167 tok/s both ways).

generation:

36.5 tokens/s

prompt processing (prefill):

909.0 tokens/s

quant:

PTQ1_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Ternary-Bonsai-2-27B at PTQ1_0 running on a 32 GB RTX PRO 4500 Blackwell via llama.cpp, with 36.5 tok/s decode and 909 tok/s prefill at 32,768 tokens of context in a single stream. Setup uses the 6.58 GB ternary GGUF with a 131,072-token context window; the Q8_0 mmproj vision tower still works in the ternary GGUF. At 131,072 tokens of context the single stream drops to 14.1 tok/s decode, 630 tok/s prefill and 207 s time to first token. Two and four concurrent streams at 32,768 tokens give 12.4 and 4.6 tok/s decode per stream, and the concurrent cells at 131,072 tokens returned HTTP 500. The user notes decode collapses with concurrency, unlike EXL3 rows which hold a flat decode rate.

generation:

60.0 tokens/s

quant:

PQ2_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Ternary Bonsai 2 27B at ~60 tok/s with PQ2_0 and ~52 tok/s with PTQ1_0 on a 12 GB RTX card. Setup uses a custom LM Studio runtime built on Prism's llama.cpp fork (b10709) to load the custom ternary GGUF types, with vision (mmproj) working. On Apple Silicon, an oMLX adapter runs the MLX 2-bit pack at 18 tok/s decode on an M1 Max 64 GB, about 30% slower than the v1 pack because it does not use oMLX's tuned 2-bit kernel.

generation:

7.6 tokens/s

prompt processing (prefill):

42.0 tokens/s

quant:

ternary (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Bonsai 2 27B at 7.6 t/s generation and 42.0 t/s prompt processing on an 8GB M2 Mac mini, with all layers offloaded to the GPU and zero swap. Setup is PrismML's llama.cpp fork (prism-b10683-d8f26ee) with a ternary 5.95GB GGUF, flash attention on, -ub 256, q4_0 KV cache, -np 1, and iogpu.wired_limit_mb=6656. The default -ub 512 and f16 KV both fail to allocate; -ub 128 with f16 KV also runs at identical speed. Setting the wired limit to 7168 kernel-panicked the machine. In real use with llama-server at 8k context, a 1,896-token prompt took 47s to first token (pp ~40 t/s) and generated at 7.0 t/s; the same prompt via prefix cache took 0.8s. 16k context works but 32k OOMs, and PQ2_0 (7.21GB) does not fit. The user notes greedy decoding falls into repetition loops and recommends temp 1.0 / top-p 0.95 / top-k 20.

generation:

6.0 tokens/s

prompt processing (prefill):

18.0 tokens/s

quant:

PTQ1_0 (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Ternary Bonsai 2 27B at ~6 t/s generation and ~18 t/s prefill on an NVIDIA T1000 8GB, comparing it against Gemma 4 26B-A4B for audiobook speaker attribution. Setup is a PrismML llama.cpp fork with PTQ1_0 ternary weights (5.95 GB), q8_0 KV cache, 8192 context, and all layers on GPU; the model scores 448/480 (93.3%) exact speaker matches on four chapters. The user does not adopt it, citing narrator identity drift, adjacent-turn swaps, dropped lines, and an intonation regression at 5-10x the wall-clock of the incumbent Gemma 4 26B-A4B MoE.

generation:

101.0 tokens/s

quant:

PQ2_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Ternary Bonsai 2 27B at ~101 t/s on an RTX 5090 32GB, generating 106,396 tokens in ~17.5 minutes. Setup is a PrismML llama.cpp fork (release prism-b10683-d8f26ee, CUDA 12.8) with the PQ2_0 GGUF at 7.2 GB, 262,144 context, flash attention on, and reasoning effort xhigh. User compares it against Qwen 3.5 9B Q6_K at ~168 t/s and Gemma 4 12B Q8_0 at ~96 t/s on the same pagoda prompt, and rates Bonsai's output substantially better.