Local model performance on your hardware

generation:

57.4 tokens/s

quant:

PQ2

User benchmarks Bonsai-2 27B PQ2 at 57.4 tok/s on an RTX 5060 Ti 16GB, where it finished 13 of 14 agent tasks and fixed 10, with no malformed tool calls. The run is part of a 39-model, 50-prompt comparison graded by four frontier models, with 3k-token outputs on a 5800X3D and 64GB RAM. Three of four judges named Bonsai-2 their favourite; judges flagged weak coding (1.5) and honesty (4.5), and it claimed no tests existed without running anything. Underdog-Saluki 27B scored the highest mean (6.42) at 40.9 tok/s, Gemma-4 12B Q6 reached 92.0 tok/s, and LFM3 1.7B was fastest at 192 tok/s but last overall. Rankings are trustworthy to about 0.4 points.

generation:

57.2 tokens/s

quant:

3-bit (GGUF)

User benchmarks Qwen3.8-27B 3-bit at 57.2 tok/s on an RTX 5060 Ti 16GB, scoring 95/100 on a 100-question test of programming, game development, reasoning and hallucination detection. Setup is the Ignis engine with a 3-bit GGUF, Max Thinking enabled, 5,828 output tokens in 110.79 s. User compares it with Qwen3.8-Flash-Next Gyro-S (25.6 tok/s, 95/100, 7m 26s) and Gyro-M (16.7 tok/s, 99/100, 9m) run on the Strata engine; scores are the user's own answer-key evaluation and may need verification.

generation:

11.7 tokens/s

quant:

1.75 bpw

User reports DeepSeek at 1.75 bpw decoding at a median 11.7 t/s on an RTX 5060 Ti 16GB in a 64 GB RAM PC, with experts streamed from an SSD cache. The figure comes from 36 decode requests comparing nominal expert-cache budgets; the --ssd-cache 63 setting reached 11.7 t/s median in both rounds. The article distinguishes configured cache budgets from resident RAM usage and examines speed, memory headroom and reclamation.

generation:

27.0 tokens/s

User reports Qwen3.8 27B at about 27 t/s on dual RTX 5060 Ti 16GB with no speculation (no MTP, DFlash, n-gram or DSpark). Setup is vLLM nightly 0.32.1rc1 under WSL2 on Windows 11, with a Ryzen 5 3600 and 32GB DDR4; the second card sits in the first M.2 slot via an adapter. On a second machine (Ryzen 7 7800X3D, 64GB DDR5) the same run gave about 40 t/s, rising to 44.4 t/s after NCCL tuning flags; the user says NCCL was effectively disabled without them, since disabling it entirely reproduced the 27 t/s.

generation:

40.0 tokens/s

quant:

IQ3_S (GGUF)

User reports about 40 tokens per second running a Swift-1.5 Qwen3.8 27B IQ3_S GGUF on an RTX 5060 Ti 16GB. Setup is llama.cpp with MTP speculative decoding at 81920 context size, on a Ryzen 7 9800X3D with 32GB DDR5-6000. User asks whether this is the best they can do for general use, coding and agentic tasks.

prompt processing (prefill):

400.0 tokens/s

quant:

EXL3 2.9bpw (EXL3)

User reports DeepSeek V4.1 Flash 1017 running at up to 40 tok/s, averaging around 30 tok/s, on 2x CMP 170HX plus an RTX 5060 Ti 16GB (144GB total VRAM) with 128GB DDR4-3000 and an Intel Core i9-10980XE. Setup uses LibertAIDAI's EXL3 2.9 bpw quantization with DSpARK enabled and expert caching: 93% of routed experts run on the GPUs, the rest on the CPU, with frequently used experts migrated to VRAM in the background. Prefill is around 400 tokens/sec, which the user calls slow. The 40 tok/s is a peak; the average is about 30 tok/s.

generation:

81.5 tokens/s

quant:

IQ3_S (GGUF)

User benchmarks nine local Qwen 3.8 configurations against Claude Opus 4.6, with Flash-Next IQ3_S split across both GPUs reaching 81.5 t/s decode at 96K context and 75.5 t/s on a short code prompt. Setup is Strata with IQ3_S GGUF weights (83.6 GB) on an RTX 5060 Ti 16GB over PCIe x16 plus an RTX PRO 4000 Blackwell 24GB over PCIe x1, with experts streamed over PCIe; the 27B IQ3_S runs at 93.2 t/s on a short prompt and 48.5 t/s at 32K. The narrow x1 link is a major bottleneck: Flash-Next read a 4K prompt at about 145 t/s on the PRO 4000 alone versus 1,433 t/s on the 5060 Ti. Best local official-sampling HumanEval was 96.3% against 100% for Opus, while local models won MATH-500 (96.5% vs 95.0%) and instruction following (100% vs 94.9%).

generation:

8-10 tokens/s

prompt processing (prefill):

100.0 tokens/s

quant:

NVFP4

User reports Nemotron 3.5 Super VL running at ~8-10 t/s generation and 100 t/s processing on dual RTX 5060 Ti 16GB cards with 48GB DDR5-6000 system RAM. Setup uses unsloth's NVFP4 quants; the dual 48GB system RAM suggests CPU offload. The user tested obscure game knowledge from Anno 1602 without web search and found Nemotron 3.5 Super VL came closest, though it still hallucinated some details. They compare it against GPT-OSS 120B-A5B, Qwen3.5-122B-A12B, DeepSeek V4 Flash and Gemma 4 31B, all of which also had errors.

generation:

15.0 tokens/s

quant:

IQ3_S (GGUF)

User reports Qwen3.8-Flash-Next IQ3_S (83.6 GB) decoding at ~15 tok/s (~67 ms/token) in steady state on an RTX 5060 Ti 16 GB with 15.38 GiB RAM and a Gen3-class NVMe. Setup is a patched llama.cpp (Sabah-MoE-Accelerator) that streams the expert bank from disk into a 9.8 GB VRAM slab with unbuffered I/O; the 28.8 GB per-layer token embedding is left on CPU via -ot, peak VRAM is 15.36 of 16.31 GB, and host RAM stays flat at 4.78 GiB. Time to first token is 11.2 s for a 53-token prompt (12.68 GB cold fill at 1.11 GB/s), so a 256-token answer is 7.1 tok/s end to end and 10.4 tok/s over the decodes alone; warm-up runs 79-204 ms/token. The author notes speculative decoding and larger batches were 2.8-7.7x slower on this rig, and a second GTX 1080 was measured as no faster than disk.

generation:

60.0 tokens/s

prompt processing (prefill):

1100.0 tokens/s

quant:

QAT

User reports Gemma 4 31B IT QAT at over 60 t/s generation and 1100 t/s prompt processing on dual RTX 5060 Ti 16GB cards. Setup is llama.cpp 0.7.0 with QAT weights, tensor split and MTP enabled across the two cards. Earlier runs on the same setup gave 40 t/s gen and 980 t/s pp, and 17 t/s gen with 970 t/s pp at 32K for Q6_K. Qwen3.8 27B runs at around 70 t/s with MTP and 100 t/s when generating code. The user uses the models for roleplay in SillyTavern.

generation:

300.0 tokens/s

User reports Bonsai 2 at 300 tok/s on an RTX 5060 Ti 16GB using a fork of llama.cpp. The user states the fork previously ran Bonsai 2 at 140 tok/s on an RTX 4070 Ti Super 16GB, and that after further optimization it reaches 140 tok/s on the 5060 Ti 16GB. The 300 tok/s figure is given as a correction for running without full context. The fork was also modified to run more than one session in parallel, as it previously ran only one.

generation:

60.0 tokens/s

quant:

IQ3_S (GGUF)

kv:

Q8

User reports Qwen3.8-Flash-Next IQ3_S at 60 t/s generation on an RTX 5060 Ti 16GB, running on Strata with system RAM overflow. Setup uses Q8 KV cache at 260k context, with the model overflowing to system RAM. The user also tested Qwen3.8 27B IQ3_S at 25-30 t/s with 120k context and Qwen3.8 27B IQ4_XS at 25-30 t/s with 80k context (100k without MTP). User recommends Flash-Next for coding and 27B for non-coding and SVG generation. Notes that Q8 KV cache produced better quality than Q4 in lm-eval benchmarks (73.5% vs 70% on 200 gsm8k samples).

generation:

75.0 tokens/s

prompt processing (prefill):

1200.0 tokens/s

quant:

Q2 (GGUF)

kv:

Q4

User reports Qwen3.8 Flash-Next at 75 tok/s generation and 1200 tok/s prefill at 128k context on an RTX 5060 Ti 16GB. Setup is Basalt (a Strata fork for Blackwell) with a Q2 quant and Q4 KV cache, on a system with 32GB RAM and about 5GB left for the OS. User says the model is used as a dedicated coding agent and had to add a build flag to allocate more RAM.

generation:

50.0 tokens/s

prompt processing (prefill):

900.0 tokens/s

quant:

IQ3_XXS (GGUF)

kv:

Q8_0

User reports Qwen3.8 27B at just over 50 t/s decode and just under 900 t/s prompt processing on a single 16GB RTX 5060 Ti. Setup is llama.cpp with an IQ3_XXS GGUF pruned to 9.3 GB via vocabulary pruning, Q8_0 KV cache, MTP speculative decoding, and everything in VRAM. The user gets just over 100k context with Q8 KV cache and a little over 170k with Q4 KV cache. They previously ran Gemma 4 26B A4B QAT at 90 t/s generation and just under 3000 t/s prompt processing, and Qwen 3.6 35B A3B, but found their intelligence insufficient for agentic use cases.

generation:

77.0 tokens/s

quant:

Q4 (GGUF)

User benchmarks llama.cpp, LM Studio, and Ollama on an RTX 5060 Ti 16GB and an Apple M3 Max 36GB, serving Qwen2.5-Coder-7B Q4 GGUF. On the RTX 5060 Ti, llama.cpp reaches 77.0 tok/s, LM Studio 76.8 tok/s (0.3% behind), and Ollama 69.1 tok/s (10.3% slower). On the M3 Max, llama.cpp reaches 53.5 tok/s, Ollama 46.2 tok/s (14% slower), and LM Studio 38.2 tok/s (29% slower). The metric is wall-clock tokens per second over a 12-prompt suite with 256 tokens generated per prompt and greedy decoding. The author attributes the gap to wrapper overhead rather than the shared llama.cpp engine.

generation:

38-60 tokens/s

quant:

GSQ-RCO 3 xxs (GGUF)

kv:

Q8

User reports Swift-1.5-Qwen3.8-27B-GSQ-RCO 3 xxs running on an RTX 5060 Ti 16GB with 4-step MTP and a 100k context at Q8 KV cache, leaving around 300MB of VRAM free. Performance is about 38-60 t/s, stabilizing around 45-50 t/s on long tasks; without MTP the model runs at 22-30 t/s. User says the quant is faster and better than the previous Swift model, with code quality close to Q4 performance.

prompt processing (prefill):

620.0 tokens/s

User reports Qwen3.8-Flash-Next 125B on an RTX 5060 Ti 16GB with about 620 tok/s prompt processing at 16K; decode is given only per workload, 26.4 tok/s code, 21.3 agent, 21.8 reasoning and 17.7 at 21K context, with no overall figure. Setup is the author's QwFN-hybrid engine on one RTX 5060 Ti 16GB, the 76GB model's experts offloaded across VRAM, 32GB DDR5 system RAM and an NVMe SSD. llama.cpp on the same machine prefilled at about 165 tok/s at 16K. The author replays workloads token by token against llama.cpp, and the current release passes 4/5 parity gates.

generation:

28.5 tokens/s

prompt processing (prefill):

878-892 tokens/s

quant:

IQ3_S (GGUF)

kv:

q8_0

User reports Qwen3.8 27B at 28.5 t/s decode on a 14,644-token prompt on an RTX 5060 Ti 16GB, with prefill at 878-892 t/s. Setup is an adaptive KV-streaming llama.cpp fork on one RTX 5060 Ti with GSQ-RCO IQ3_S weights, q8_0 KV cache, a 1536 MiB KV window and 131K context allocated, the whole model on the GPU, no speculative decoding; the host is a Ryzen 9600X with 32 GB single-channel DDR5. Decode falls as the prompt grows: 25.2 t/s at 35K, 20.6 at 55K, 18.7 at 69K and 12.6 at 94K with the 1536 MiB window, and about 9 t/s near 131K. A smaller 512 MiB window drops faster (8.7 t/s at 94K). Prefill falls from 878-892 to 630-635 t/s. MTP is not usable with the fork; it gained about 5 t/s on stock llama.cpp.

prompt processing (prefill):

1600-1800 tokens/s

User reports Bonsai 2 27B running on an RTX 5060 Ti 16GB with NInfer, achieving prefill over 1000 t/s (dropping from an initial 1600-1800 t/s) and decode above 50 t/s at up to about 200k context. Setup uses a modified and optimised version of NInfer, adapted with the help of ChatGPT and Claude to run on the 5060 Ti. Prefill and decode figures are given as ranges rather than single values.

generation:

55.0 tokens/s

prompt processing (prefill):

1500.0 tokens/s

quant:

IQ1_M (GGUF)

User reports Qwen3.8 Flash-Next Coder at 55 tok/s generation and 1500 tok/s prompt processing on an RTX 5060 Ti 16GB with 32GB system RAM. Setup is Strata with IQ1_M quant at 65k context. A one-shot landing page demo ran at 46 tok/s over about 2 minutes; an interactive 3D globe took 8 minutes plus a minute of JS fixes. User says the IQ1_M output quality surprised them, comparable to above Q3 for other local models.