Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Glimmer 30B at 41.97 t/s generation and 684.86 t/s prompt processing on an RTX 3060 12GB plus RTX 3070 8GB.
Setup is llama.cpp with IQ4_XS weights, Q8_0 KV cache, 65,536-token context, Flash Attention on, and a DFlash draft model (Q4_0) on CUDA1 with a 45/55 layer split.
The benchmark prompt was 994 tokens, not a filled 65K context; generation ranged 41.71–42.45 t/s over 128 output tokens, with 4.5 s average request time and 9.7 s model load.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash at 27.9 tok/s on an RTX 3060.
Setup is Strata on a single RTX 3060.
User finds the model impressive for the GPU but hits a recurring engine error ("layer 1 never rang") after several minutes of use, requiring a reboot, and asks how to troubleshoot it.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 56.0 tok/s decode and 497 tok/s prefill on 2× RTX 3060 12GB.
Setup is llama.cpp CUDA with UD-Q4_K_XL GGUF, q4_0 KV cache, 131072 context, tensor split across the two cards, and MTP speculative decoding. The 106,294-token context gate took 213.8 s.
A second recipe runs Qwen3.6-35B-A3B on the same machine at 72.6 tok/s decode and 512 tok/s prefill with q8_0 KV cache and layer split, keeping some expert tensors in system RAM (8.22 GiB host RAM drop).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 47.3 tok/s decode on 2x RTX 3060 12GB.
Setup is llama.cpp (server-cuda12-b11146) with UD-Q4_K_XL GGUF, tensor split, auto-fit and the GGUF's built-in MTP head, at 131,072 tokens of context. TTFT 537 ms.
The same PR also validates 20 single-card RTX 3060 12GB recipes and adds a second two-card recipe, Qwen3.6-35B-A3B at 74.2 tok/s decode and 147 ms TTFT at 131,072 tokens. Long-context recall was tested at 104,941 tokens. Several candidates failed (missing quant tag pulling default Q4_K_M, empty model path, OOM) and 16 were skipped for 512-2,048-token contexts.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Strata at iq2_xxs quant running at about 30 t/s on 2x RTX 3060 12GB, comparable to vanilla llama.cpp with Qwen3.8 27B. Peak speeds reach up to 50 t/s when working with code.
Setup is llama.cpp with iq2_xxs quant, 48GB system RAM, Ryzen 5 3600, and Pi Harness.
User notes the larger model offers better quality even at this quant, but it loops badly and fails to complete meaningful tests, becoming indecisive and going in circles without producing output.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-27B at 40 t/s on an RTX 3060 12GB.
Setup is Beellama with an IQ3 head and KvarnN KV quantization at 3/3, 80k context, and Dflash2 speculative decoding.
The model is a custom hybrid combining Escha Labs W2 body with a Lowgpu IQ3 head, 8.65GB total size. User also tested on a 4070 Ti and 50-series cards.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 10 t/s decode and 150 t/s prefill on an RTX 3060 12GB with 128 GB RAM.
Setup is llama.cpp with IQ4_XS GGUF and q8_0 KV cache at 200k context, with 48 MoE layers offloaded to CPU.
The post is a general inference engineering guide; the figures come from the user's sample launch command for this model.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
rag
User reports Qwen 3.5 3.5B at 34 t/s generation and 1000+ t/s prompt processing on two RTX 3060 12GB cards.
Setup uses a 4x oculink to mini PCIe 3x connection. User is building a new system with dual 16x lanes and NCCL.
User notes the cards are used for RAG and multiple purposes, and mentions the 12GB 3060 cards are becoming scarce and expensive.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
agenticcoding
User reports Qwen3.8-27B at ~9.7 tok/s generation and ~225 tok/s prompt processing on a single RTX 3060 12GB.
Setup is llama.cpp with Q4_K_S weights and Q4_0 KV cache at 96K context, using MTP speculative decoding with 46 FFN layers offloaded to system RAM.
MTP acceptance was ~88% with 2.76 mean accepted tokens; peak generation reached ~10.6 tok/s and VRAM usage was ~11.04 GB.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-27B at 29.65-29.69 generated tokens/s on one RTX 3060 12 GB, with a 33.21 tokens/s high-acceptance peak.
Setup is llama.cpp with IQ3_XXS GGUF, Q4_0 KV cache, 65,536-token context, MTP depth 6, and Deferred-Commit Factor Replay.
A separate prefill sweep improved an 8,595-token prompt from 82.13 to 320.24 prompt tokens/s. Q4 at 32K measured 16.06 tok/s on a Rust task and 14.30 tok/s on a C++ task; Q3 at 96K and 128K measured 21.3-22.0 and 19.7-20.9 tok/s respectively.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-use
User reports DeepSeek V4 Flash REAP-150B at 2.8-3.4 tok/s on an RTX 3060 12GB with a Ryzen 9 7900 and 64GB DDR5-6000 under WSL2.
Setup is FreeToken with FP4 experts and an experimental disk tier called Overspill; the expert math runs on the CPU via the AVX-512 path and experts stream from disk through RAM. The model is about 85GB.
User compares against llama.cpp at 0.4-0.5 tok/s and Colibri at 1.1-1.2 tok/s, and reports TTFT of 102s for a 6.4k prompt, 43s for the first short prompt after cold start and 10s for the next short prompt. User notes the runs are not bit-identical because llama.cpp stores the non-expert weights in Q8_0 while FreeToken and Colibri use FP8, and asks others to test on different hardware.
User reports running Qwen3.8-27B on a 2x3060 machine with llama.cpp, achieving generation speeds ranging from 43.37 to 49.99 t/s and prompt processing speeds from 45.01 to 198.10 t/s.
Setup uses Q4_K_M quantization, 50000 context length, tensor split, flash attention, and speculative decoding with ngram-mod and draft-mtp.
User notes the main limitation is short context, which is too short for long agentic sessions.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at 20-40 tokens per second on 2x RTX 3060 12GB.
Setup is ExLlamaV3 with an exl3-4.0bpw quant at roughly 170k context.
The user asks for tips to improve performance.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen2.5 7B Instruct at 49.0 tok/s on an RTX 3060 12GB, and 6.4 tok/s CPU-only on the same machine.
Setup is Ollama 0.11.4 with Q4_K_M weights, single-stream, warm speed as the median of three runs with 200-token answers. About 1.2 GB of the card was already in use by the desktop and a Stable Diffusion web UI.
The same model measured 48.9 tok/s on the GPU in a separate CPU-vs-GPU comparison, with a cold first answer of 5.6 s on GPU against 31.3 s CPU-only, a 7.6x speed-up. The user notes the CPU is a 2012 Xeon E5-2665 without AVX2, and that a modern CPU with AVX2 or AVX-512 will beat these CPU numbers.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 25 t/s on three RTX 3060 12GB cards.
Setup is llama.cpp in layered mode with a Q6 quant.
The figure is a current baseline; the user asks whether moving to four or eight RTX 3060s under vLLM would improve or hurt token generation.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingtool-useagenticlong-context
User reports Qwen3.8 Flash Next Q4_K_XL at 66.14 t/s prompt processing and 22.09 t/s generation on an RTX 3060 12GB, Intel i7-14700 and 96GB DDR5.
Setup is ik_llama / llama-server on Windows 11 at 262,144 tokens of context with MTP3 and NCMOE48. A fresh OpenCode agent session produced a 10,890-token initial prompt, and MTP draft acceptance for that request was 89.5%.
After the initial ~11K-token prefill, a second short prompt completed in 1.6s with short-generation throughput around 22–26 t/s. At approximately 71K active tokens, generation was around 10 t/s, with an OpenCode session average around 12 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at ~29 t/s on an RTX 3060 12GB at around full context, with 34–40 t/s at lower context.
Setup is llama.cpp on CachyOS/Arch Linux with MTP/speculative decoding, comparing the ISTA-DASLab GSQ-RCO-IQ3-XXS quant (~10.4GB, ~2.5 BPW) against the ByteShape IQ3-XXS quant (~500MB smaller, 2.88 BPW).
The ByteShape quant performed worse in practice: the GSQ version generated a 3D voxel diorama in one shot under 55k tokens, while ByteShape took roughly three shots and still hadn't completed after over 98K tokens.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B GSQ-RCO-IQ3-XXS at 29 t/s on an RTX 3060 12GB at full context, and 40 t/s at near-empty context.
Setup is llama.cpp with MTP speculative decoding and high thinking enabled, using a 122K context window.
The run used roughly 75K tokens and the context was compacted twice; the user rates this as the most impressive local model in a nine-model web-dev comparison.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at 45 t/s generation and 600 t/s prompt processing on a dual RTX 3060 12GB setup.
Setup uses a Q4 model and Q4 KV cache at roughly 100k context, with the model running fully on the GPUs.
The user is asking for upgrade advice and considering a 4x RTX 3060 build or 24GB Intel cards, and wonders whether PCIe lanes would bottleneck a four-card setup.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.8-Flash-Next at ~12 t/s on 2× RTX 3060 12GB at 131,072 context.
Setup is the AtomicChat AD-3.84bpw-IQ4_XS-M64 GGUF (~85GB) with CPU MoE offload, most expert weights partially offloaded to system RAM.
The goal was a quality-first local agent / batch worker rather than maximum chat latency; a 26.5k-token agent prompt was used.