Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports Qwen3.8-27B GGUF quants fail to converge on long thinking tasks on 2x RTX 5060 Ti 16GB, while NVFP4 mixed-precision quants converge in ~17K thinking tokens.
Setup is llama.cpp and vLLM with UD-Q6_K_XL GGUF (23.6 GB) and f16 KV cache, tensor-parallel across 2 GPUs. Throughput was healthy at 23-41 t/s on llama.cpp and 22-23 t/s on vLLM.
The failure appears after 30K-38K thinking tokens with no closing token, tail-looping, or engine crash. The same model in NVFP4 (FP8 attention/GDN, FP4 MLP) converges in ~17K tokens on SGLang at 47-50 t/s. User hypothesizes recurrent state error accumulation under uniform quantization.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Gemma 4 26B-A4B-it at 95.22 tok/s decode and 3,476 tok/s prompt processing on an RTX 5060 Ti 16 GB.
Setup is a custom llama.cpp build (commit 0b484ab2b plus 12 commits) with MXFP4-MOE quantization (experts MXFP4, dense Q8_0, 4.66 BPW) and q4_0 KV cache, 1 slot, flash attention enabled, 65,536 token context.
The MXFP4-MOE quant brings the model to 13.70 GiB, fitting the 16 GB card where Q4_K_M at 15.85 GiB does not. On an RTX 5090 the same quant gives 10,733 tok/s prompt and 196.6 tok/s decode versus 8,744 and 219.9 for Q4_K_M, with perplexity 3,864 vs 3,615.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen 3.8 27B at about 40 t/s generation and 500 t/s prompt processing on dual GPUs (RTX 5060 Ti 16GB + RTX 3060 Ti 8GB).
Setup uses a Q3 quant with 140k context across 24GB of VRAM, for single-user agentic coding via Opencode.
User finds the usable work window between context loads too small for complex projects and is asking whether to add a second 5060 Ti and more RAM for a Strata setup, or switch to a 128GB Strix Halo.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports running Qwen3.8 27B with tensor parallelism across 4x RTX 5060 Ti 16GB in a Lenovo P620 (Threadripper PRO 3975WX, 128GB DDR4).
Setup is vLLM with FP8 weights and MTP3 speculative decoding, headless Ubuntu, NCCL 2.30.7. Decode at C1 is 100-110 tokens/s and prefill is 2.5-3k tokens/s per stream.
User compares NVFP4+MTP3 (137-141 t/s C1, 422 t/s C4 sum, ~3.5k prefill), SGLang NVFP4+DFlash (118-137 t/s C1, 357-386 t/s C4 sum, ~3.4k prefill), and llama.cpp Q6+MTP3 (95 t/s C1, 83 t/s C4 sum, ~1.2k prefill). NVFP4 was 1.4x faster than FP8 but degraded in MTP3 tests (F1 0.44 vs 0.73).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8-Flash-Next at 30.5 t/s generation and 250.4 t/s prompt processing on 2x RTX 5060 Ti with 32GB system RAM.
Setup is llama.cpp with an IQ1_M GGUF (27.58 GB), 98304 context, q8_0 KV cache, tensor split 1,1, and 8 MoE layers offloaded to CPU.
User asks whether their settings are correct and what config others run for this model on this hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.8-Flash-Next at 94-99 t/s sustained decode on 4x RTX 5060 Ti 16GB, with short turns reaching ~110 t/s.
Setup is a custom Strata fork (1 main + 3 expert tiers) with UD-Q4_K_XL quant, 524k context via ported rope-scaling, and a draft vocab fix. Prefill measured at 2159 t/s at 16k and 2420 t/s at 220k; 312k warm replay in 12ms.
User abandoned the setup because the Q4_K_XL quant loops in agentic use, repeating thoughts and re-firing tool calls, unlike their int4 W4A16 lane. They compare it favorably to their vLLM lane (~40 t/s at 455k with batch 4) but say the quality bar is not met for daily driving.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User compares two local Qwen models against Claude Opus 4.6 on three Python coding tasks, scoring hidden tests and code review. The local Qwen3.8-Flash-Next Coder model tied Opus 4.6 at 92.7 out of 100, while Qwen 3.8 27B Q6 scored 87.0.
Setup is two RTX 5060 Ti 16GB cards (32 GB VRAM total) on an Intel Core i5-14400 with 48 GB DDR4 RAM, Windows 11. The Coder model is a pruned Strata variant keeping half the experts per layer, as an IQ1_M GGUF, running between 50 and 90 tokens/s, around 60-65 on average. The 27B Q6 Unsloth quant runs at a steady 50 tokens/s.
User notes one run per model, differences of 2-3 points are meaningless, and the Coder model was graded harder with more weird-input cases. The review was done by Claude Fable 5.1 with public grading sheets.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen 3.8 27B at 29.18 tok/s decode on an RTX 5060 Ti 16 GB at 4K context.
Setup is llama.cpp build ad1de39e0 with UD-IQ3_XXS quant and q4_0 K/V cache, full GPU offload, flash attention, one parallel slot.
Baseline decode without speculation is 29.18 tok/s at 4K, 23.62 at 32K and 19.77 at 64K; MTP n=2 gives 56.91/42.53/34.95 and MTP n=3 gives 63.89/45.48/39.06 tok/s at the same contexts. N-gram speculation added only 1.7% (29.66 tok/s). A 96K prompt measured 617.73 prompt tok/s and 36.74 decode tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B at a steady 50-55 t/s with MTP on a single RTX 5060 Ti 16GB.
Setup is llama.cpp with UD-IQ3_XXS GGUF weights and Q4_0 KV cache for K and V, 94208-token context, batch and ubatch 512, flash attention on, draft-mtp speculation with spec-draft-n-max 3, parallel 1.
Without MTP the average is 35 t/s. The machine has an Intel i5 10th gen and 32GB DDR4.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at 63.7 tok/s decode on an RTX 5060 Ti 16GB, using MTP-5 speculative decoding at 8K context.
Setup is llama-tierkv (a llama.cpp fork) with IQ4_XS weights and a Q4_0 KV cache kept in three tiers (VRAM, RAM, SSD); the weights are on the GPU. Without speculation the same configuration decodes at 25.7 tok/s, and prefill is 680-850 tok/s.
At 256K context without speculation, decode is 22.70 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports about 35 t/s or more with Qwen3.8 27B GSQ-RCO-Uncensored on an RTX 5060 Ti 16GB.
Setup is llama.cpp with IQ3_XXS quant, q4_0 KV cache, 131072 context, and MTP draft plus ngram speculative decoding.
The user is on Fedora 44 with an AMD Ryzen 9600x and 16 GB system RAM, and had struggled to get reasonable speed before finding this model.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-35B-A3B at 47-51 tok/s generation on an RTX 5060 Ti 16GB over OCuLink in a Proxmox VM.
Setup is llama.cpp with UD-IQ3_XXS quant and q4_0 KV cache, fully on the GPU with 348 MiB headroom, at 160K context.
Prompt eval of 75K tokens took 64.8 s (1,168 tok/s). The 9B variant reached 40-50 tok/s with UD-Q4_K_XL and about 47-51 tok/s with IQ3_XXS.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-uselong-contextvisioncoding
User reports Qwen3.8-27B at 256K context on an RTX 5060 Ti 16GB, with decode staying around 30-40 tok/s.
Setup is a KVMem-enabled llama.cpp server (v0.14.0) with ISTA IQ3_S quant, MTP, Q8 KV cache, vision on GPU, and a 32K GPU window plus 16K reserved for new tokens; the rest of the KV cache stays in 32GB system RAM.
In a 33-request tool task ending at 262058/262144 tokens, prefill was 437 tok/s first pass and 243 tok/s overall, decode 30 tok/s over the tool rounds and 29 tok/s on the last 512 tokens, with 15.5 GB peak VRAM and 13.1 GB RAM. An IQ4_XS variant with Q5 KV and CPU vision reached 466/255 tok/s prefill and 33/35 tok/s decode. A shorter image and code task ran about 37-41 tok/s decode.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Nemotron 3.5 Lightning 30B-A3B at 79.98 t/s generation and 4737.46 t/s prompt processing on 2x RTX 5060 Ti 16GB.
Setup is llama.cpp with NVFP4 GGUF weights and q8_0 KV cache, 1048576 token context, flash attention on, no MTP layers.
The run processed a 54025-token prompt and generated 2156 tokens; the user notes it fits in 32 GB VRAM with no expert-layer offloading to RAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports failing to reproduce a claimed 50+ t/s on an RTX 5060 Ti 16GB, getting 20-35 t/s instead.
Setup is a freshly built llama.cpp fork (beellama.cpp) for sm120 with MTP speculative decoding, kvarn3 KV cache, 98304 context, and parallel 1.
The user questions whether the original high-throughput post is fake or if they are missing something.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
visionlong-context
User reports Qwen3.8 27B running on an RTX 5060 Ti 16GB with MTP speculative decoding, reaching up to 50 t/s in the first half of a 100k context and dropping to about 25-30 t/s at the end.
Setup is a modified llama.cpp fork with adaptive KV cache streaming, IQ3_XXS GGUF weights, q8_0 K cache and q4_0 V cache, a 2100 MiB KV stream buffer, and a BF16 vision head.
The user notes the streaming fork only works on NVIDIA hardware and that the speed depends on context size versus KV stream buffer size.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports MiMo 2.6 Distill Qwen 9B at 41.63 t/s generation and 1718.29 t/s prompt processing on an RTX 5060 Ti 16GB.
Setup is llama.cpp (Llama UI) with GGUF Q8_0 weights, q4_0 KV cache, 122880-token context, and Flash Attention enabled.
The run produced 11979 output tokens over 4 min 47 s from a 1929-token prompt; the first generated version failed and a second review pass by the model produced the final working demo.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
visionagenticlong-context
User reports Qwen3.8 27B Swift-Genesis at 76 t/s in their benchmark on 2x RTX 5060 Ti 16GB, with a range of 40-113 t/s and 45-65 t/s under normal agentic work.
Setup is llama.cpp with GGUF weights at 262k context, MTP4 speculative decoding, and vision offloaded to system RAM.
The user notes the model fits under 17GB with vision, allowing full 262k context on 32GB VRAM, and asks what the tradeoff of this model is compared to other Swift NVFP4 models.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next 177B NVFP4 at 9.06 tok/s decode on a single RTX 5060 Ti 16GB with 32GB DDR5, using a custom SSD-streaming engine that keeps 99 GiB of the 119 GiB model on NVMe.
Setup streams routed experts and a 50.7 GiB n-gram table from a Gen5 NVMe SSD on 8 read threads, with about 270 MiB read per token and roughly 75% of expert lookups hitting VRAM or RAM. Prefill is 49.2 tok/s on a 5.5k-token prompt.
Best decode turn reached 10.4 tok/s; llama.cpp on the same machine averaged 4.9 tok/s. The same engine runs Qwen3.6-35B-A3B NVFP4 at 47.3 tok/s decode and 591 tok/s prefill. v1 is limited to RTX 50-series on Windows 11 and WSL2 with greedy decoding only.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Gemma 4-26b-a4b at 80~90 t/s at 2048 context and 40+ t/s at 96K context on an RTX 5060 Ti 16GB.
Setup is IQ4-XS quantization on an Intel i7-4790 with DDR3 1600MHz RAM.
User compares it favorably to Qwen 3.5-35b-a3b at 45 t/s and notes the older DDR3 platform limits memory bandwidth.