llamaperf

NVIDIA RTX 2080 Ti 22GB (modded) reports

11 reports. These are individual setups, not a controlled benchmark. NVIDIA RTX 2080 Ti 22GB (modded) speeds and models →

Speeds are as each source reported them. GPU count, offloading and concurrent requests change them, and prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing two reports.

Tone: positiveNVIDIA hardware
generation:
60.1 tokens/s
prompt processing (prefill):
28.0 tokens/s
quant:
IQ3_S (GGUF)

User reports Qwen3.8-Flash-Next at 60.1 tok/s decode and 28 tok/s prefill on two modded RTX 2080 Ti 22GB cards, with a peak of 70.9 tok/s at 61K of 128K context. Setup is Strata with IQ3_S weights, custom tensor parallelism over an NVLink bridge on M.2 OCuLink PCIe Gen3 x4, on a Minisforum AR900i with a Core i9-13900HX and 64GB DDR5. The 60.1 tok/s figure comes from 41,288 tokens in 695 s at 41K context; the internal RTX 5070 was idle during these runs.

Nov 20, 2026
Tone: positiveNVIDIA hardware
generation:
27.0 tokens/s
prompt processing (prefill):
616.0 tokens/s
kv:
INT8
codingvision

User reports Qwen3.6/3.8-27B decoding at 27.0 tok/s without MTP on a single modded 22 GB RTX 2080 Ti under NInfer v0.3.0 on Windows 11, with prefill of 616 tok/s on a 512-token prompt. Setup uses a 64k context with an INT8 KV cache; the weights take 15.4 GiB and the engine runs entirely in VRAM with no system RAM offloading. MTP speculative decoding is on by default. With MTP, decode reaches 47.5 tok/s on a real code prompt (26.3 without) and 28.8 tok/s on a real prose prompt (26.0 without); prefill on a 64k-token prompt is 458 tok/s. The engine supports only Qwen3.6/3.8-27B, and dropping RTX 5090, FP8, FP4 and 35B-A3B code paths cut the binaries from 1.56 GB to 0.28 GB.

Oct 30, 2026
NVIDIA hardware
generation:
220.8 tokens/s
quant:
NVFP4 (NVFP4)
kv:
FP8

User reports 220.84 tok/s decode in their fork's README at a 4K prompt for Qwen3.8 27B NVFP4 on two RTX 2080 Ti 22GB cards, and 209.35 tok/s at 32K. Setup is 2x RTX 2080 Ti 22GB joined by NVLink with tensor parallel 2, NVFP4 weights, FP8 KV cache, a 256K window and DFlash2 speculative decoding (K=7), on a single request. The figures use synthetic text inputs with high speculative-hit rates, and the README says real-task throughput may not reach them. 4x Tesla T10 with Qwen3.8 27B FP8 reaches 191.89 and 189.38 tok/s on the same test.

Oct 8, 2026
Tone: positiveNVIDIA hardware
generation:
120.0 tokens/s
quant:
AWQ 4-bit (safetensors)
kv:
turboquant_k3v4_nc

User reports Qwen3.6-35B-A3B at AWQ 4-bit decoding at about 120 tokens/second under a single request on 2x modded 22GB RTX 2080 Ti with NVLink. Setup is vLLM with tensor parallelism across the two cards. The post's main benchmark ran a W8A8-INT8 build with a TurboQuant KV cache (about 395K tokens) and up to 16 concurrent requests, but those results appear only in screenshots; the author notes AWQ 4-bit is the better choice on Turing.

Oct 8, 2026
Tone: positiveNVIDIA hardware
generation:
69.0 tokens/s
quant:
UD-IQ4_XS (GGUF)
kv:
q8_0
coding

User reports Qwen3.6-35B-A3B at 4-bit (UD-IQ4_XS) scoring 89.63% pass@1 on HumanEval at 69 tok/s on a single RTX 2080 Ti 22GB. Setup is llama.cpp with UD-IQ4_XS dynamic 4-bit (4.25 bpw), q8_0 KV cache, 16384 context, flash-attn on, whole model resident in VRAM with no offload. A routing patch (MoE expansion, 20 experts instead of 8 on layers 25-39) reached 90.85% pass@1 at 56 tok/s, a 19% decode speed cost; the user calls the +2 problems within statistical noise and notes the tests are original HumanEval, not EvalPlus.

Oct 7, 2026
Tone: positiveNVIDIA hardware
generation:
59-68 tokens/s
quant:
GPTQ-Int4 (GPTQ)
kv:
turboquant_k3v4_nc
codingagentictool-usevisionlong-context

User reports Qwen3.8-27B at ~59-68 tok/s single-stream decode on 2x modded RTX 2080 Ti 22GB (SM75, TP=2, NVLink). Setup is a vLLM fork with GPTQ-Int4 self-quant, turboquant_k3v4_nc KV cache, 524,288 max-model-len (YaRN 2x), MTP K=2 speculative decoding, and vision enabled. Aggregate throughput is 364.7 tok/s at 16 lanes; 24 lanes regresses. The S4 scoped re-emission drafter measured 811.5 tok/s on copy-shaped spans but crashes at current HEAD, and v3 GPU-merge async is negative.

Oct 7, 2026
Tone: positiveNVIDIA hardware
generation:
95.5 tokens/s
prompt processing (prefill):
465.8 tokens/s
quant:
Q4_K_P (GGUF)
kv:
Q8_0
agenticlong-context

User reports Qwen3.6-35B-A3B at 95.5 t/s decode and 465.8 t/s prefill on a mixed three-GPU Turing setup of 2x CMP 50HX 10GB plus 1x RTX 2080 Ti 22GB. Setup is llama.cpp with a ported DP2A patch (PR #25834) and -fmad=false, Q4_K_P GGUF weights, Q8_0 KV cache, Flash Attention, 262144 context, MTP speculative decoding at --spec-draft-n-max 3, tensor split 1,1,2.5 with the RTX as tail stage, ubatch 448. Baseline DP4A gave 47.8 t/s decode and 372.7 t/s prefill; DP2A alone 54.0 t/s; adding -fmad=false 62.2 t/s. A repo-28k workload measured 405.4 t/s prefill and 82.3 t/s decode. An end-to-end rerun showed 91.4 t/s raw eval, 4% below the 95.5 reference. Context 368640 also works but quality beyond the trained window was not measured.

Oct 7, 2026
Tone: positiveNVIDIA hardware
generation:
36.6 tokens/s
quant:
IQ3_S (GGUF)
kv:
q8_0
long-contextagentictool-usevision

User reports Qwen3.8-27B at 36.6 tok/s decode on a single modded RTX 2080 Ti 22GB, with 262,144 tokens of context. Setup is KVMem (retrieval-based long context) with IQ3_S weights, q8_0 KV cache, MTP speculative decoding (58.6%/61.6% acceptance) and vision, using 16,552 of 22,528 MiB VRAM. The user compares against the upstream author's RTX 5060 Ti 16GB at 31.7 tok/s, and notes the lossless KV-streaming alternative runs 30-42 tok/s in the resident window but drops to ~9.6 tok/s past 135K tokens; a 260,096-token needle-in-a-haystack test hit exactly.

Oct 7, 2026
Tone: positiveNVIDIA hardware
generation:
101.3 tokens/s
prompt processing (prefill):
1841.7 tokens/s
quant:
AWQ (AWQ)
agenticlong-context

User reports Qwen3.6-27B-AWQ at 101.3 tok/s decode and 1841.7 tok/s prefill on dual modified RTX 2080 Ti 22GB cards with NVLink. Setup is vLLM 0.21.0 with AWQ Marlin, TP=2, MTP K=3, FlashInfer/FA2 attention, and FlashQLA SM70/SM75 legacy GDN prefill. The speed columns use the PP4096/TG128 repeat. The same rig reached a 735,084 token KV cache with turboquant_4bit_nc at max_model_len=262144, and passed a PP262000/TG1 gate at 785.26 tok/s prefill. A sequential 60-request Ragent6 run averaged 700.9 tok/s prefill and 35.2 tok/s generation. Gemma4 31B GPTQ reached 99.64 tok/s decode and 1655.65 tok/s prefill on the same runtime.

Oct 7, 2026
Tone: positiveNVIDIA hardware
generation:
44.1 tokens/s
quant:
IQ3_S (GGUF)
kv:
Q8_0
flash attention:
on
long-contextvision

User reports Qwen3.8-27B-GSQ-RCO at 44.07 t/s decode on a hardware-modded RTX 2080 Ti 22GB, with 38~44 t/s quoted as the overall range. Setup is a llama.cpp fork (TurboQuant 4-bit + Adaptive KV Streaming) with IQ3_S weights, 256K context (262,144 tokens), Q8_0 key cache and turbo4 value cache, a 2048 MiB GPU staging pool, and MTP speculative decoding with 2 draft tokens. Needle tests at 8K/16K/32K gave 41.23, 39.00 and 34.84 t/s decode with 81-83% draft acceptance and 100% recall at 82% depth; a sustained 256K run gave 40.55 t/s at 89.17% acceptance. Peak VRAM was 18,619 MiB and GPU power peaked at 266.4 W.

Sep 28, 2026
Tone: positiveNVIDIA hardware
generation:
25.0 tokens/s
quant:
W8A16
kv:
Q8

User reports Qwen3.8 27B at ~25 tok/s with standard autoregressive decoding (MTP0) on a modded RTX 2080 Ti 22GB. Setup is a Turing port of the NInfer engine with the official groupwise-int (W8A16) artifact and Q8 KV cache. With MTP3 speculative decoding (draft window = 3) the title gives 45 tok/s and the body ~456 tok/s, at ~65% acceptance rate. VRAM usage is ~17.5 GiB with the MTP draft weights loaded, leaving ~4.5-5.0 GiB free for KV cache.

Sep 7, 2026