Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B (Swift-1.5 fine-tune, IQ3_XXS GGUF) at 70 t/s on an RTX 4080 16GB with 81920 context, using MTP plus n-gram speculation.
Setup is llama.cpp (llama-server CUDA build in WSL2/Debian) with q8_0 KV cache, flash attention on, all layers on GPU, and 14.9 GB of the 16 GB VRAM used.
With n-gram only the user got 45 t/s at 204800 context. They note MTP was worth the tradeoff and could grow context further by dropping MTP or allowing spillover into system memory.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B Heretic (UD-IQ4_XS) at 55 tok/s on code and 50 tok/s on prose on an RTX 4080 16GB with MTP 2 draft tokens.
Setup is llama.cpp b11457 with a 15K-token prompt, 32K context, MTP 2 drafts, mean of 3 fixed seeds. Without MTP the same config gives 29 tok/s on both code and prose.
User notes 2 draft tokens beat 3 (68 vs 56 tok/s on code, 55 vs 36 on prose at 40K), 48K vs 40K produced identical tokens but was 22% slower, and VRAM spill from the desktop dropped code decode to 68, 51 or 42 tok/s depending on 1.2, 1.4 or 1.9 GB of shared VRAM. MTP head precision barely matters (q6_K vs IQ3_S: 83% vs 83% acceptance on code, 58% vs 54% on prose). The 12GB and 24GB context numbers are computed from llama.cpp buffers, not measured.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.8 27B at 166.7 t/s generation and 2719.9 t/s prefill on an RTX 4080 16GB.
Setup is NInfer with GSQ safetensors and rk4v4-e8 KV cache, single card, DFlash2 speculative decoding at 8K context.
User also reports MTP3 decode at 151.2 t/s and DFlash2 K=7 decode at 262.3 t/s at 32K, with prefill up to 2720 t/s and 100k context support.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 55 t/s on an RTX 4080 with 128 GB DDR5 RAM, using Strata with Q3 weights and Q8 KV cache at roughly 120-150k context.
Setup is Strata with the 3060 left unused; adding the second GPU drops generation to 45 t/s and prompt processing to 250 t/s from 590 t/s.
With Unsloth Studio the same model reached 27 t/s on the 4080 alone, 23 t/s with both GPUs, and 16 t/s with both GPUs on an older PCIe 4.0 x1 link. The user asks whether the 3060 is counter-productive alongside the 4080.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 Flash-Next at 16.5 t/s generation and 350 t/s prompt processing on an RTX 4080 16GB with 64GB system RAM, at 131k context.
Setup is llama.cpp built from a pr-mtp-fix branch with the Unsloth MTP head as the draft model, AtomicChat AD-4.27bpw Q4_K_M target weights, and a Q8 KV cache. The --spec-draft-cpu-moe flag keeps draft experts in RAM so the target's hot experts fit in the 16GB of VRAM.
An IQ4_XS quant was also tested and reached about 10 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6 35B A3B at 45 t/s on an RTX 4080 16GB, and Gemma 4 12B at 34 t/s.
Setup is llama.cpp with CUDA 13.3, MXFP4_MOE GGUF, 65535 context, q8_0 KV cache, and draft-MTP speculative decoding with n-max 4.
User finds MTP speculative decoding consistently reduces throughput rather than improving it, contrary to reports of 20-30% gains.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B at 56.48 t/s decode and 980.2 t/s prefill on an RTX 4080 16GB at 131,072 context with 102,400 active input tokens.
Setup is ExLlamaV3 1.5.0 with TabbyAPI, the turboderp SC_3.00bpw_H4_V4 EXL3 quant, a 6,5 KV cache, and MTP k=2 with a Q6 draft cache. VRAM peaked around 15.2GB of 16.4GB.
Without MTP decode was 33.68 t/s, so MTP gave about a 68% decode increase at roughly a 5% prefill cost. Q4 draft cache was about 9% slower than Q6, and dynamic drafting was slower than fixed k=2.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 8 t/s without MTP, 14 t/s with MTP, and about 19 t/s with full optimizations.
Setup is beellama 0.4.3, a llama.cpp fork.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports running a 182B model on an RTX 4080 with 64GB RAM by offloading ngrams to SSD.
User claims it is faster and more intelligent than Qwen3.8 27B and provides the launch command.