NVIDIA RTX 3060 12GB

generation:

20.0 tokens/s

quant:

Q5 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Gemma 4 31B Q5 at 20 t/s on two RTX 3060 12GB cards in llama.cpp, up from 14 t/s with layer splitting, a 42% gain. One card runs on PCIe x16 and the other on a PCIe 3.0 x1 riser; the user notes the fit and context are tight and that testing was limited while awaiting a PSU upgrade. On a rented 2x RTX 5060 Ti 16GB machine, L3.1 Nemotron Sunfall 71B IQ3_XS at 2k context with flash attention gave 20.7 t/s generation and 464.4 t/s prompt processing under tensor split, versus 11.7 t/s and 591.8 t/s under layer split. PCIe traffic during decode measured roughly 90-110 MB/s per direction, with load spikes above 4,000 MB/s.

generation:

43-44 tokens/s

quant:

PTQ1_0 (GGUF)

kv:

q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a Qwen3.8-27B derivative (Swift Bonsai 2 27B Katana + MTP) generating around 43-44 t/s on an RTX 3060 12GB, peaking at 45.74 t/s over a three-second window. Setup is the Prism fork of llama.cpp with a PTQ1_0 ternary GGUF and q4_0 KV cache at 128,000 tokens, MTP speculation with a draft limit of 1, all model layers on the GPU and the vision projector on the CPU; the whole process used 9,144 MiB of VRAM. In daily use the user sees 28-40 t/s after a few prompts and 17-25 t/s much later with far more context. The user cites another author's Bonsai 2 PTQ1_0 + MTP tests on the same card at 50.1 t/s with MTP versus 39.8 without at 131K, and 26.86 versus 22.23 at 41.8K tokens filled. Figures are described as observations rather than controlled benchmarks.

generation:

7.4 tokens/s

quant:

FP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports the full DeepSeek-V4.1-Flash-1017 model at up to 7.4 tok/s warm (5.8 cold) on a 160-token code prompt, on an RTX 3060 12GB with 64 GB DDR4 and two NVMe drives. Setup is the user's own PROMETHEVS engine, which streams the 510 GB original FP4 weights through VRAM, RAM and both drives, computing tokens on a 2-bit copy of the experts and recomputing uncertain tokens with the original weights; DeepSeek's draft head proposes blocks of five tokens. Server runs give 5-8 tok/s on code and 4.5-5.4 on reasoning and agent prompts, against 2.14 tok/s for plain exact decoding of the full model on the same PC. Agreement with the full weights is 97.4% for the 2-bit copy alone and 98.3% with corrections over 960 audited tokens. Only tested on this one Windows machine so far, needs about 1.1 TB of NVMe, and has not been compared with DeepSeek's reference implementation.

generation:

7.4 tokens/s

quant:

2-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash-1017 at 7.4 tok/s with a warm cache and 5.8 tok/s cold on an RTX 3060 12GB with 64 GB DDR4 and NVMe storage. Setup is a custom engine that computes tokens on a 2-bit copy of the experts, recomputes flagged tokens with the original FP4 weights, and uses a draft head proposing blocks of five tokens. The 510 GB model streams from disk; plain exact decoding on the same machine gives 2.14 tok/s. Reasoning and agent prompts run at 4.4 to 5.2 tok/s. On 960 audited tokens the 2-bit copy matches the exact path 97.4% of the time, 98.3% with corrections; a 1,101-token agent prompt scores 95.9%. The engine is not yet released.

generation:

30-40 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0/q4_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports running Qwen3.8-27B-pi at 30-40 tok/s generation on an RTX 3080 10GB plus RTX 3060 12GB, with 32GB DDR5 4800 system RAM. Setup is llama.cpp (llama-server) with Q4_K_M weights, q8_0 K cache and q4_0 V cache, 75776 context, layer split with tensor-split 107,93, and MTP draft speculation (draft model Q4_0 on CUDA1, spec-draft-n-max 3). Average MTP acceptance is about 0.75; pushing n-max above 3 causes OOM. User tried DFlash2 but found it slower than MTP, and is generally happy with quality and speed, having started at 10 tokens per second with Qwen 3.8.

generation:

17.9 tokens/s

prompt processing (prefill):

230.1 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B Uncensored at 17.89 t/s generation and 230.07 t/s prompt eval on a two-card rig of RTX 3060 plus RTX 2070 (about 20GB VRAM total). Setup is Ollama in Docker with Q4_K_M GGUF, q8_0 KV cache, flash attention on, 68115-token context and all 65/65 layers on the GPUs; the host is an i5-13600 with 48GB DDR4. The prompt eval figure is sustained over a 29,396-token run lasting 27m23s, with 19,489 MiB of VRAM in use.

generation:

2.3 tokens/s

quant:

Q5_K_M (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User asks about a budget upgrade to run Gemma 4 31B Q5_K_M at 16K context and at least 15 t/s, currently getting 2.3 t/s generation (2.7 t/s with MTP) on an RTX 3060 12GB with 47% GPU / 53% CPU offload. Setup is Ollama/llama.cpp with Q5_K_M weights and Q8 KV cache on an i5-14400F with 32GB DDR4; the 243-token prompt was processed with the model partly offloaded to system RAM. User says IQ3_M drops quality without adding speed, and is weighing an RTX 5060 Ti 16GB plus the 3060 on a PCIe 3.0 x1 riser, dual Tesla P100s on an X79/X99 Xeon system, or a cheaper AMD 16GB card.

generation:

50.0 tokens/s

prompt processing (prefill):

1000.0 tokens/s

quant:

IQ1_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next Coder at 50 t/s generation and 1k t/s prefill on 2x RTX 3060 12GB. Setup is Strata engine with IQ1_M quant, 48 GB system RAM, Ryzen 7 5700X. User built a voxel pagoda as a first test and asks what others are building.

generation:

79.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at a median 79 tok/s generation on a single RTX 3060 12GB with 32GB DDR5. Setup is llama.cpp with Q4_K_M and q8_0 KV cache, using --cpu-moe with a 7500MiB MoE expert cache, mlock, and MTP speculative decoding at 32768 context. Prompt processing is around 800 tok/s under 1k tokens, ~1100 tok/s for 1k-4k, and ~1190 tok/s for 4k-16k. MTP draft acceptance is around 85% and VRAM sits at ~11.4GB. The 79 tok/s median comes from 433 responses longer than 500 tokens, with most between 76 and 83 tok/s. A fixed 3.6k prompt / 700 token output test showed 50-58 tok/s with --n-cpu-moe 25, 71-73 tok/s with --cpu-moe plus cache, 75-81 tok/s with mlock, and 77-84 tok/s after updating llama.cpp. Forcing 4 experts reached 91-95 tok/s but caused looping on long outputs.

generation:

60.0 tokens/s

quant:

IQ3_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 60 t/s average code generation on empty context, with peaks of 60-70 t/s, on two RTX 3060s. Setup is Strata with IQ3_S weights and MTP speculation enabled; the user tuned --pcie-frac to 0.2, --pool-workers, --trim-stage-weights, --pipeline-windows 6, and forced GPU 1 (the faster PCIe lane) as the main GPU via gpu_order as_given. The user says the previous baseline was about 35 t/s decode peak, and that performance degrades with full context but still holds a solid +15-20% over the old baseline.

generation:

44-52 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Swift-1.5-Qwen3.8-27B decoding at roughly 50 tok/s (44 to 52) on an RTX 3060. Setup is the NInfer RTX 3060 port with an IQ3_XXS quant, 131K context and a 46,400-token KV cache. The user says it passed a one-liner prompt on the first try.

generation:

12.9 tokens/s

prompt processing (prefill):

25.0 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 12.9 t/s generation and 25 t/s prompt processing on two NVIDIA RTX 3060 12GB cards. Setup is llama.cpp with the IQ4_XS GGUF quant, run as separate llama-bench PP512 and TG128 tests rather than one request. The 87.24 GiB model file exceeds the cards' combined 24 GB VRAM, and GPU offload, KV cache type and context length are unreported.

generation:

30.8 tokens/s

prompt processing (prefill):

755.0 tokens/s

quant:

Q4_K_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 30.8 t/s generation and 755 t/s prompt processing on two NVIDIA RTX 3060 12GB cards. Setup is llama.cpp with Q4_K_S, run as separate llama-bench tests: PP512 with 512 newly processed prompt tokens and TG128 with 128 generated tokens. Repetitions, KV cache type, context length, thread counts and GPU placement are unreported, and the 14.29 GiB figure is model-file size, not measured resident memory.

generation:

97.5 tokens/s

prompt processing (prefill):

1870.0 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Ornith 1.5 35B-A3B at 97.5 t/s generation and 1870 t/s prompt processing on two NVIDIA RTX 3060 12GB cards. Setup is llama.cpp with an IQ4_XS GGUF quant; the author's own Ornith quant, similar to Unsloth UD-IQ4_XS, with a 16.94 GiB model file. The figures are separate llama-bench tests, PP512 and TG128, not one 512-input/128-output request; repetitions, KV cache type, context length and GPU split are unreported.

generation:

15.0 tokens/s

quant:

IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 15 t/s on 2x RTX 3060 12GB with 64GB RAM, compared to Strata's 35 t/s on thinking and 50 t/s on code. Setup is vanilla llama.cpp with IQ3_XXS and no MTP drafter, as the only available MTP Q4_K_M drafter is 3.5x larger and slows decode. User asks for a working llama recipe, noting tensor split fails with a cudaMalloc error on device 0.

generation:

20.1-21.1 tokens/s

quant:

IQ2_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next 125B-A6B at 20.14-21.13 tok/s on an RTX 3060 12GB with 16GB DDR4 RAM, using a custom llama.cpp build with --moe-direct-io prefetching. Setup is llama.cpp with IQ2_XS quant and a 68GB model file, run under a 6G cgroup memory limit; the model is streamed from NVMe SSD with no expert pruning and bit-exact output. Stock llama.cpp on the same machine gives 1.41-2.12 tok/s with 1,140-1,565 major page faults per token; warm cache runs reach 24+ tok/s, cold starts begin at 3.5-4.5 tok/s, and prompt processing remains slow.

generation:

40.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Bonsai 2 running at 40 t/s and upwards on an RTX 3060 12GB. No engine, quant or context length is stated. The user says it is fast but gets stuck reasoning for hours for mediocre results, and asks for alternatives or tweaks.

generation:

38-40 tokens/s

quant:

IQ3 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 38-40 t/s on three RTX 3060 12GB cards in a mining rig frame with Strata, against 13.2 t/s with llama.cpp. Setup is an IQ3 quant through Strata, with 64GB DDR4 and an 8th gen i7; at IQ3 the model does not fit in 36 GB of VRAM, so part of it runs from system RAM. The user asks whether others run dated mining hardware with similar success.

generation:

1-2 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 1-2 tokens per second on coding tasks with Qwen3.8 27B on an RTX 3060 12GB. Setup is Ollama on Windows 11 with WSL, with 72GB system RAM and a Core i9 14900k. The user also mentions Qwen3-Coder 30B, Gemma 4 31B, and Qwen2.5-Coder 7B. The user says every coding task is painfully slow, with 15-minute thinking times, no output, and stream timeout errors. They ask for help setting up a usable local coding workflow.

generation:

43.2 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports glm-4.7-flash at 43.16 tok/s on a Framework host with Core Ultra CPU and RTX 3060 12GB. Setup is Ollama with a 192-token generation cap; the model scored 5/5 on the repo's compact coding, tool-use and agent-orchestration quality suite. It is the fastest perfect-quality model in the capture, ahead of qwen3.5:35b-a3b at 39.90 tok/s and qwen3-coder-next at 32.55 tok/s. A separate llama-server comparison on a T5500 host lists omnicoder:9b-q4_k_m at 1.48 tok/s, qwen3.5:35b-a3b-q2_k_l at 3.28 tok/s, lfm2:24b at 4.36 tok/s and granite4:7b-a1b-h at 6.48 tok/s.