Find which open-weight LLMs fit in your GPU or Mac and compare the speeds people report on setups like yours.
Pick your GPU or Mac, then read the speeds people reported on it, or estimate which models fit and how fast they run.
Free to use. No account needed. Memory estimates and community measurements are labelled separately.
Search for a model to see its reported speeds across GPUs and Macs.
These are individual setups, not a controlled benchmark. Compare GPU count, quantization, context and offloading before comparing speeds. How to read a report →
Exact recorded values. Context may be a configured limit; matching filters does not establish identical prompts, offloading or concurrency.
NVIDIA RTX 4060 Laptop 8GB · Kindi · 24,576 ctx
User reports Bonsai 2 27B at 28.0 tok/s on an RTX 4060 Laptop 8GB, with prompt reading at 31.2 tokens/s. Setup is the Kindi 0.3.0 image with a PTQ1_0 GGUF at 24,576 context, leaving 1,367 MiB free. Decode falls from 28.0 tok/s at position 100 to 25.2 tok/s at 24,000; a verify run passed token-exact against llama.cpp at 29.3 tokens/s.
User benchmarks Bonsai-2 27B PQ2 at 57.4 tok/s on an RTX 5060 Ti 16GB, where it finished 13 of 14 agent tasks and fixed 10, with no malformed tool calls. The run is part of a 39-model, 50-prompt comparison graded by four frontier models, with 3k-token outputs on a 5800X3D and 64GB RAM. Three of four judges named Bonsai-2 their favourite; judges flagged weak coding (1.5) and honesty (4.5), and it claimed no tests existed without running anything. Underdog-Saluki 27B scored the highest mean (6.42) at 40.9 tok/s, Gemma-4 12B Q6 reached 92.0 tok/s, and LFM3 1.7B was fastest at 192 tok/s but last overall. Rankings are trustworthy to about 0.4 points.
Intel Arc B580 12GB · llama.cpp
User reports Bonsai-2 27B running at around 45 t/s on an Intel Arc B580 12GB. Setup is PrismML's llama.cpp fork with custom SYCL kernels and fusions, which the user wrote and PrismML merged (PR #356, about 7,700 lines). The models use ternary weights (-1, 0, +1). The user also trained small ternary models in cloud data centres that run at home on the B580: SmolLM2-360M shrunk from 690 MB to 162 MB at 1.4x the original perplexity and about 94% of benchmark scores, and SmolLM2-135M from 257 MB to 73 MB at about 1.5x and about 91%. They say these are not yet Bonsai quality, whose models keep about 99%. The models are not released yet.
NVIDIA RTX 5060 Ti 16GB · llama.cpp
User reports Bonsai 2 at 300 tok/s on an RTX 5060 Ti 16GB using a fork of llama.cpp. The user states the fork previously ran Bonsai 2 at 140 tok/s on an RTX 4070 Ti Super 16GB, and that after further optimization it reaches 140 tok/s on the 5060 Ti 16GB. The 300 tok/s figure is given as a correction for running without full context. The fork was also modified to run more than one session in parallel, as it previously ran only one.
AMD Strix Halo 128GB · Prism llama.cpp 1a07bfa5 + input-sync patch / HIP · 262,144 ctx
User reports Bonsai 2 at 22.3 t/s generation and 214.2 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is PrismML-Eng/llama.cpp with PQ2_0 weights, Q4_0 K/V cache, full GPU offload, flash attention, one slot and 262144 context on a single Ryzen AI MAX+ 395 APU with 128 GB unified memory. The figure is the idle-peer fixed-prefix run with 524 input and 256 output tokens; the source gives 22.25989819108211 t/s and 214.23028774725407 t/s.
Intel Arc B580 12GB · llama.cpp · 128,000 ctx
User reports ternary Bonsai 2 27B at 48.6 t/s plain decode on an Intel Arc B580 12GB. Setup is a custom llama.cpp SYCL branch with ternary weights, q4_0 KV cache and 128K context, with no speculation for this figure. With MTP, n-gram speculation and a trimmed draft head the model reaches 85 t/s writing code, 79 t/s in chat and 89 t/s at 115K tokens of history, with prompt reading at ~1,480 t/s. The user's own draft head was used for the video, so public results may be a few percent lower.
M4 Pro 64GB · llama.cpp
User reports a sourced third-party community benchmark of Bonsai 2 27B PQ2_0 on one Apple Mac mini M4 Pro with 64 GB unified memory. Prompt processing pp512 measured 126.89 +/- 0.02 tokens/second and token generation tg128 measured 20.53 +/- 0.01 tokens/second. Setup is llama.cpp PrismML fork (tag prism-b10709-9a9394a) with PQ2_0 GGUF and default F16 KV cache, 10 threads, -ngl 99, flash attention enabled, Metal/BLAS backend. The two figures are separate synthetic llama-bench tests with three repetitions each, not a paired chat request. Caveats: Spotlight indexing consumed approximately 1.5-2 CPU cores throughout the runs; an unrelated 664 MB embedding model reappeared between runs; no swap pressure was observed. The 6.70 GiB GGUF size is a file size, not measured resident or peak RAM. Peak RAM, configured maximum context, exact weight revision, binary hash, and exact measurement date are unknown.
User reports Bonsai 2 running at 40 t/s and upwards on an RTX 3060 12GB. No engine, quant or context length is stated. The user says it is fast but gets stuck reasoning for hours for mediocre results, and asks for alternatives or tweaks.
Intel Arc B580 12GB · llama.cpp · 131,072 ctx
User reports Bonsai 2 27B (PTQ1_0 ternary) at 42.3 t/s plain generation and 895 t/s prompt reading on an Intel Arc B580 12GB. Setup is llama.cpp's llama-bench (tg128 and pp512) on the SYCL backend with PTQ1_0 GGUF weights. With speculation, fresh code answers reach 90 t/s, renames 365 t/s, edits 258 t/s. At 48K context new code runs 63 t/s and edits 34 t/s; at 115K, 44 and 27 t/s. The user also compares against the Vulkan backend and PrismML's own fork, and reports KL divergence of 0.00022 against reference logits.
M4 Max 128GB · llama.cpp · 131,072 ctx
User reports Bonsai 2 27B (Ternary) at 34.5 t/s generation on an Apple Mac Studio M4 Max 128GB. Setup is llama.cpp (Prism fork build 10683) with PQ2_0 ternary GGUF, Metal, flash attention on, -ngl 999, 131072 configured context, single request, temperature 0, no speculative decoding. The 7.2 GB figure is the model file size, not measured peak RAM. The 34.5 t/s is the arithmetic mean of three prose runs (35.49, 34.03, 33.87 t/s) over 2097 prompt tokens and 320 output tokens; a related code workload averaged 34.3 t/s. Prefill is not reported because TTFT includes request overhead. KV-cache type, peak memory, CPU core count and power draw are unknown.
NVIDIA RTX 5060 Laptop 8GB · llama.cpp
User reports Ternary Bonsai 2 27B at 29.27 t/s on an RTX 5060 Laptop 8 GB under Windows, using a Prism llama.cpp build. Setup is llama.cpp with PTQ1_0 (5.53 GiB), temp 0, seed 42, a 2048 reasoning budget and a 3072 token cap on 50 MATH-500 questions. A −2 logit bias on "wait", "maybe" and "perhaps" (nine token ids) dropped accuracy from 44/50 to 43/50 and raised average tokens from 845.2 to 872.3; the biased run measured 29.28 t/s. The user calls it one deterministic data point, not proof.
NVIDIA RTX 3080 10GB · llama.cpp · 32,768 ctx
User reports Bonsai 2 27B PTQ1_0 at 52.18 t/s generation and 1210.6 t/s prefill on an RTX 3080 10GB. Setup is llama.cpp (fork build prism-b10735-842b188, CUDA 12.8) with PTQ1_0 quant and q8_0 KV cache at 32K context, single card, batch 1, depth 0, -ngl 99 -fa on. User also benchmarks PQ2_0 at 61.38 t/s and Qwen3.8-27B-UD-IQ2_XXS at 44.46 t/s on the same card, and reports a context ladder up to 160K q4_0 at 9627 MiB peak VRAM. A LiveCodeBench v6 comparison gives PTQ1_0 32/50 pass@1 versus 16/50 for IQ2_XXS and 28/50 for PQ2_0.
NVIDIA RTX 4090 · NInfer · 262,144 ctx
User reports Ternary Bonsai 2 27B at 188 tok/s decode averaged over six mixed prompts on a single RTX 4090 under native Windows. Setup is NInfer, a from-scratch C++/CUDA engine, with Q4/Q5 weights, E8 4-bit KV cache, MTP plus n-gram speculative decoding, and 262K context per request. On edit-style prompts the model reaches 250 tok/s with MTP and 532 tok/s with MTP + n-gram; through the server on 7K-11K-token file edits with thinking on it goes 215 to 378 tok/s. Three concurrent requests give 360 tok/s aggregate. Against Prism's llama.cpp fork on the same card: 101 vs 77 tok/s decode and 3,061 vs 1,363 tok/s pp512. N-gram gains depend on repetition and are zero on free prose (167 tok/s both ways).
NVIDIA RTX Pro 4500 Blackwell 32GB · llama.cpp · 131,072 ctx
User reports Ternary-Bonsai-2-27B at PTQ1_0 running on a 32 GB RTX PRO 4500 Blackwell via llama.cpp, with 36.5 tok/s decode and 909 tok/s prefill at 32,768 tokens of context in a single stream. Setup uses the 6.58 GB ternary GGUF with a 131,072-token context window; the Q8_0 mmproj vision tower still works in the ternary GGUF. At 131,072 tokens of context the single stream drops to 14.1 tok/s decode, 630 tok/s prefill and 207 s time to first token. Two and four concurrent streams at 32,768 tokens give 12.4 and 4.6 tok/s decode per stream, and the concurrent cells at 131,072 tokens returned HTTP 500. The user notes decode collapses with concurrency, unlike EXL3 rows which hold a flat decode rate.
M2 8GB · llama.cpp · 8,192 ctx
User reports Bonsai 2 27B at 7.6 t/s generation and 42.0 t/s prompt processing on an 8GB M2 Mac mini, with all layers offloaded to the GPU and zero swap. Setup is PrismML's llama.cpp fork (prism-b10683-d8f26ee) with a ternary 5.95GB GGUF, flash attention on, -ub 256, q4_0 KV cache, -np 1, and iogpu.wired_limit_mb=6656. The default -ub 512 and f16 KV both fail to allocate; -ub 128 with f16 KV also runs at identical speed. Setting the wired limit to 7168 kernel-panicked the machine. In real use with llama-server at 8k context, a 1,896-token prompt took 47s to first token (pp ~40 t/s) and generated at 7.0 t/s; the same prompt via prefix cache took 0.8s. 16k context works but 32k OOMs, and PQ2_0 (7.21GB) does not fit. The user notes greedy decoding falls into repetition loops and recommends temp 1.0 / top-p 0.95 / top-k 20.
NVIDIA T1000 8GB · llama.cpp · 8,192 ctx
User benchmarks Ternary Bonsai 2 27B at ~6 t/s generation and ~18 t/s prefill on an NVIDIA T1000 8GB, comparing it against Gemma 4 26B-A4B for audiobook speaker attribution. Setup is a PrismML llama.cpp fork with PTQ1_0 ternary weights (5.95 GB), q8_0 KV cache, 8192 context, and all layers on GPU; the model scores 448/480 (93.3%) exact speaker matches on four chapters. The user does not adopt it, citing narrator identity drift, adjacent-turn swaps, dropped lines, and an intonation regression at 5-10x the wall-clock of the incumbent Gemma 4 26B-A4B MoE.
NVIDIA RTX 5090 · PrismML llama.cpp fork · 262,144 ctx
User benchmarks Ternary Bonsai 2 27B at ~101 t/s on an RTX 5090 32GB, generating 106,396 tokens in ~17.5 minutes. Setup is a PrismML llama.cpp fork (release prism-b10683-d8f26ee, CUDA 12.8) with the PQ2_0 GGUF at 7.2 GB, 262,144 context, flash attention on, and reasoning effort xhigh. User compares it against Qwen 3.5 9B Q6_K at ~168 t/s and Gemma 4 12B Q8_0 at ~96 t/s on the same pagoda prompt, and rates Bonsai's output substantially better.
On Qwen3.8 27B at 4-bit, plain single-GPU runs. Full ranking