Local model performance on your hardware

generation:

20.0 tokens/s

quant:

Q5 (GGUF)

User reports Gemma 4 31B Q5 at 20 t/s on two RTX 3060 12GB cards in llama.cpp, up from 14 t/s with layer splitting, a 42% gain. One card runs on PCIe x16 and the other on a PCIe 3.0 x1 riser; the user notes the fit and context are tight and that testing was limited while awaiting a PSU upgrade. On a rented 2x RTX 5060 Ti 16GB machine, L3.1 Nemotron Sunfall 71B IQ3_XS at 2k context with flash attention gave 20.7 t/s generation and 464.4 t/s prompt processing under tensor split, versus 11.7 t/s and 591.8 t/s under layer split. PCIe traffic during decode measured roughly 90-110 MB/s per direction, with load spikes above 4,000 MB/s.

generation:

7.4 tokens/s

quant:

FP4

User reports the full DeepSeek-V4.1-Flash-1017 model at up to 7.4 tok/s warm (5.8 cold) on a 160-token code prompt, on an RTX 3060 12GB with 64 GB DDR4 and two NVMe drives. Setup is the user's own PROMETHEVS engine, which streams the 510 GB original FP4 weights through VRAM, RAM and both drives, computing tokens on a 2-bit copy of the experts and recomputing uncertain tokens with the original weights; DeepSeek's draft head proposes blocks of five tokens. Server runs give 5-8 tok/s on code and 4.5-5.4 on reasoning and agent prompts, against 2.14 tok/s for plain exact decoding of the full model on the same PC. Agreement with the full weights is 97.4% for the 2-bit copy alone and 98.3% with corrections over 960 audited tokens. Only tested on this one Windows machine so far, needs about 1.1 TB of NVMe, and has not been compared with DeepSeek's reference implementation.

generation:

7.4 tokens/s

quant:

2-bit

User reports DeepSeek-V4.1-Flash-1017 at 7.4 tok/s with a warm cache and 5.8 tok/s cold on an RTX 3060 12GB with 64 GB DDR4 and NVMe storage. Setup is a custom engine that computes tokens on a 2-bit copy of the experts, recomputes flagged tokens with the original FP4 weights, and uses a draft head proposing blocks of five tokens. The 510 GB model streams from disk; plain exact decoding on the same machine gives 2.14 tok/s. Reasoning and agent prompts run at 4.4 to 5.2 tok/s. On 960 audited tokens the 2-bit copy matches the exact path 97.4% of the time, 98.3% with corrections; a 1,101-token agent prompt scores 95.9%. The engine is not yet released.

generation:

17.9 tokens/s

prompt processing (prefill):

230.1 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

User reports Qwen3.8-27B Uncensored at 17.89 t/s generation and 230.07 t/s prompt eval on a two-card rig of RTX 3060 plus RTX 2070 (about 20GB VRAM total). Setup is Ollama in Docker with Q4_K_M GGUF, q8_0 KV cache, flash attention on, 68115-token context and all 65/65 layers on the GPUs; the host is an i5-13600 with 48GB DDR4. The prompt eval figure is sustained over a 29,396-token run lasting 27m23s, with 19,489 MiB of VRAM in use.

generation:

2.3 tokens/s

quant:

Q5_K_M (GGUF)

kv:

Q8

User asks about a budget upgrade to run Gemma 4 31B Q5_K_M at 16K context and at least 15 t/s, currently getting 2.3 t/s generation (2.7 t/s with MTP) on an RTX 3060 12GB with 47% GPU / 53% CPU offload. Setup is Ollama/llama.cpp with Q5_K_M weights and Q8 KV cache on an i5-14400F with 32GB DDR4; the 243-token prompt was processed with the model partly offloaded to system RAM. User says IQ3_M drops quality without adding speed, and is weighing an RTX 5060 Ti 16GB plus the 3060 on a PCIe 3.0 x1 riser, dual Tesla P100s on an X79/X99 Xeon system, or a cheaper AMD 16GB card.

generation:

50.0 tokens/s

prompt processing (prefill):

1000.0 tokens/s

quant:

IQ1_M (GGUF)

User reports Qwen3.8 Flash-Next Coder at 50 t/s generation and 1k t/s prefill on 2x RTX 3060 12GB. Setup is Strata engine with IQ1_M quant, 48 GB system RAM, Ryzen 7 5700X. User built a voxel pagoda as a first test and asks what others are building.

generation:

79.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

User reports Qwen3.6-35B-A3B at a median 79 tok/s generation on a single RTX 3060 12GB with 32GB DDR5. Setup is llama.cpp with Q4_K_M and q8_0 KV cache, using --cpu-moe with a 7500MiB MoE expert cache, mlock, and MTP speculative decoding at 32768 context. Prompt processing is around 800 tok/s under 1k tokens, ~1100 tok/s for 1k-4k, and ~1190 tok/s for 4k-16k. MTP draft acceptance is around 85% and VRAM sits at ~11.4GB. The 79 tok/s median comes from 433 responses longer than 500 tokens, with most between 76 and 83 tok/s. A fixed 3.6k prompt / 700 token output test showed 50-58 tok/s with --n-cpu-moe 25, 71-73 tok/s with --cpu-moe plus cache, 75-81 tok/s with mlock, and 77-84 tok/s after updating llama.cpp. Forcing 4 experts reached 91-95 tok/s but caused looping on long outputs.

generation:

60.0 tokens/s

quant:

IQ3_S (GGUF)

User reports Qwen3.8 Flash-Next at 60 t/s average code generation on empty context, with peaks of 60-70 t/s, on two RTX 3060s. Setup is Strata with IQ3_S weights and MTP speculation enabled; the user tuned --pcie-frac to 0.2, --pool-workers, --trim-stage-weights, --pipeline-windows 6, and forced GPU 1 (the faster PCIe lane) as the main GPU via gpu_order as_given. The user says the previous baseline was about 35 t/s decode peak, and that performance degrades with full context but still holds a solid +15-20% over the old baseline.

generation:

12.9 tokens/s

prompt processing (prefill):

25.0 tokens/s

quant:

IQ4_XS (GGUF)

User reports Qwen3.8-Flash-Next at 12.9 t/s generation and 25 t/s prompt processing on two NVIDIA RTX 3060 12GB cards. Setup is llama.cpp with the IQ4_XS GGUF quant, run as separate llama-bench PP512 and TG128 tests rather than one request. The 87.24 GiB model file exceeds the cards' combined 24 GB VRAM, and GPU offload, KV cache type and context length are unreported.

generation:

30.8 tokens/s

prompt processing (prefill):

755.0 tokens/s

quant:

Q4_K_S (GGUF)

User reports Qwen3.8-27B at 30.8 t/s generation and 755 t/s prompt processing on two NVIDIA RTX 3060 12GB cards. Setup is llama.cpp with Q4_K_S, run as separate llama-bench tests: PP512 with 512 newly processed prompt tokens and TG128 with 128 generated tokens. Repetitions, KV cache type, context length, thread counts and GPU placement are unreported, and the 14.29 GiB figure is model-file size, not measured resident memory.

generation:

97.5 tokens/s

prompt processing (prefill):

1870.0 tokens/s

quant:

IQ4_XS (GGUF)

User reports Ornith 1.5 35B-A3B at 97.5 t/s generation and 1870 t/s prompt processing on two NVIDIA RTX 3060 12GB cards. Setup is llama.cpp with an IQ4_XS GGUF quant; the author's own Ornith quant, similar to Unsloth UD-IQ4_XS, with a 16.94 GiB model file. The figures are separate llama-bench tests, PP512 and TG128, not one 512-input/128-output request; repetitions, KV cache type, context length and GPU split are unreported.

generation:

15.0 tokens/s

quant:

IQ3_XXS (GGUF)

User reports Qwen3.8-Flash-Next at 15 t/s on 2x RTX 3060 12GB with 64GB RAM, compared to Strata's 35 t/s on thinking and 50 t/s on code. Setup is vanilla llama.cpp with IQ3_XXS and no MTP drafter, as the only available MTP Q4_K_M drafter is 3.5x larger and slows decode. User asks for a working llama recipe, noting tensor split fails with a cudaMalloc error on device 0.

generation:

40.0 tokens/s

User reports Bonsai 2 running at 40 t/s and upwards on an RTX 3060 12GB. No engine, quant or context length is stated. The user says it is fast but gets stuck reasoning for hours for mediocre results, and asks for alternatives or tweaks.

generation:

43.2 tokens/s

User reports glm-4.7-flash at 43.16 tok/s on a Framework host with Core Ultra CPU and RTX 3060 12GB. Setup is Ollama with a 192-token generation cap; the model scored 5/5 on the repo's compact coding, tool-use and agent-orchestration quality suite. It is the fastest perfect-quality model in the capture, ahead of qwen3.5:35b-a3b at 39.90 tok/s and qwen3-coder-next at 32.55 tok/s. A separate llama-server comparison on a T5500 host lists omnicoder:9b-q4_k_m at 1.48 tok/s, qwen3.5:35b-a3b-q2_k_l at 3.28 tok/s, lfm2:24b at 4.36 tok/s and granite4:7b-a1b-h at 6.48 tok/s.

generation:

42.0 tokens/s

prompt processing (prefill):

684.9 tokens/s

quant:

IQ4_XS (GGUF)

kv:

Q8_0

User reports Glimmer 30B at 41.97 t/s generation and 684.86 t/s prompt processing on an RTX 3060 12GB plus RTX 3070 8GB. Setup is llama.cpp with IQ4_XS weights, Q8_0 KV cache, 65,536-token context, Flash Attention on, and a DFlash draft model (Q4_0) on CUDA1 with a 45/55 layer split. The benchmark prompt was 994 tokens, not a filled 65K context; generation ranged 41.71–42.45 t/s over 128 output tokens, with 4.5 s average request time and 9.7 s model load.

generation:

27.9 tokens/s

User reports Qwen3.8 Flash at 27.9 tok/s on an RTX 3060. Setup is Strata on a single RTX 3060. User finds the model impressive for the GPU but hits a recurring engine error ("layer 1 never rang") after several minutes of use, requiring a reboot, and asks how to troubleshoot it.

generation:

56.0 tokens/s

prompt processing (prefill):

497.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

q4_0

flash attention:

on

User reports Qwen3.8-27B at 56.0 tok/s decode and 497 tok/s prefill on 2× RTX 3060 12GB. Setup is llama.cpp CUDA with UD-Q4_K_XL GGUF, q4_0 KV cache, 131072 context, tensor split across the two cards, and MTP speculative decoding. The 106,294-token context gate took 213.8 s. A second recipe runs Qwen3.6-35B-A3B on the same machine at 72.6 tok/s decode and 512 tok/s prefill with q8_0 KV cache and layer split, keeping some expert tensors in system RAM (8.22 GiB host RAM drop).

generation:

47.3 tokens/s

quant:

Q4_K_XL (GGUF)

User reports Qwen3.8-27B at 47.3 tok/s decode on 2x RTX 3060 12GB. Setup is llama.cpp (server-cuda12-b11146) with UD-Q4_K_XL GGUF, tensor split, auto-fit and the GGUF's built-in MTP head, at 131,072 tokens of context. TTFT 537 ms. The same PR also validates 20 single-card RTX 3060 12GB recipes and adds a second two-card recipe, Qwen3.6-35B-A3B at 74.2 tok/s decode and 147 ms TTFT at 131,072 tokens. Long-context recall was tested at 104,941 tokens. Several candidates failed (missing quant tag pulling default Q4_K_M, empty model path, OOM) and 16 were skipped for 512-2,048-token contexts.

generation:

30.0 tokens/s

quant:

iq2_xxs (GGUF)

User reports Strata at iq2_xxs quant running at about 30 t/s on 2x RTX 3060 12GB, comparable to vanilla llama.cpp with Qwen3.8 27B. Peak speeds reach up to 50 t/s when working with code. Setup is llama.cpp with iq2_xxs quant, 48GB system RAM, Ryzen 5 3600, and Pi Harness. User notes the larger model offers better quality even at this quant, but it loops badly and fails to complete meaningful tests, becoming indecisive and going in circles without producing output.

generation:

40.0 tokens/s

quant:

IQ3 (GGUF)

kv:

KvarnN 3/3

User reports Qwen3.8-27B at 40 t/s on an RTX 3060 12GB. Setup is Beellama with an IQ3 head and KvarnN KV quantization at 3/3, 80k context, and Dflash2 speculative decoding. The model is a custom hybrid combining Escha Labs W2 body with a Lowgpu IQ3 head, 8.65GB total size. User also tested on a 4070 Ti and 50-series cards.