NVIDIA RTX 3090 Ti

generation:

34.0 tokens/s

quant:

W4A16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at about 34 t/s per stream across four concurrent requests on an RTX 3090 Ti, roughly 135-160 t/s total. Setup is vLLM with 4-bit W4A16 weights fully in VRAM, 69.6k context, four requests batched at once; four 17.9k-token prompts read together gave first token after about 19 s (~3,700 t/s total). A CPU-only Qwen3.6-35B-A3B Q8 slot adds a fifth stream at ~9 t/s writing and 103 t/s reading on a 1.7k prompt (67 t/s at 12.8k). User compares this against Strata running Qwen3.8-Flash-Next 125B IQ3_S at 106-144 t/s writing and ~2,800 t/s reading, and Maya-M running GLM-5.3-Flash 321B at 18-24 t/s writing and ~500-1,150 t/s reading, both one request at a time. The main model, DeepSeek-V4.1-Flash at 2.77 bpw on 2x DGX Spark, writes ~40-55 t/s and reads ~1,600-2,000 t/s.

generation:

148.9 tokens/s

prompt processing (prefill):

1886.8 tokens/s

quant:

IQ3_S (GGUF)

kv:

int8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 148.86 tok/s narrative decode and 174.67 tok/s code decode on a dual-GPU rig (RTX 3090 Ti + RTX 3090). Setup is Strata with IQ3_S quant, int8 KV cache, 262,144-token context, MTP speculation, --spec 4, --kv-resident 32768, --mmap-experts, vision on, automatic layer split. The 148.86/174.67 figures are the ISO18 profile (PREFILL_HELP), which also gives 1,886.76 tok/s at 10K prefill and 3,641.54 tok/s at 90K. ISO19 (PF_FUSED) is the prefill winner at 1,995.53 tok/s at 10K and 4,149.45 tok/s at 90K, but about 5% slower on narrative decode and 3% slower on code decode; its output-sanity check was still pending. The post traces a Strata 0.1.41 regression (decode about 8% slower, 10K prefill about 15% faster, 90K prefill about 7% slower) to automatic GPU placement, and verifies a fix (STRATA_SPLIT_UNIFORM=1 plus gpu_order: as_given) that restores the 0.1.40.3 baseline range. The fix is opt-in and validated on this specific rig.

generation:

105.0 tokens/s

prompt processing (prefill):

2000.0 tokens/s

quant:

IQ3-XXS (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next Swift-1.5 at 105 t/s decode and around 2000 t/s prefill on an RTX 3090 Ti plus a V100 16GB. Setup is Strata 0.1.39 with IQ3-XXS weights, int8 KV cache, 393216 max context, yarn rope scaling at 1.5, and speculative decoding with --spec 4. Layers are split 2/3 on the 3090 Ti and 1/3 on the V100. User says the same model previously reached only around 50 t/s decode with llama.cpp layer splitting.

generation:

67.8 tokens/s

quant:

IQ4_XS (GGUF)

kv:

turbo5

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 67.8 tok/s after a 250,000-token prompt on an RTX 3090 Ti. Setup is the llamAmpere v0.4 fork of llama.cpp with an IQ4_XS GGUF, turbo5/turbo4 KV cache, 262,144-token context, and the model's MTP drafter at depth 4. The run generated 5,120 tokens and peaked at 22,346 MiB of VRAM. The release tree decodes 104.28 tok/s on the v0.3.1 ship corpus and 103.09 tok/s at 100K KV depth.

generation:

15.0 tokens/s

prompt processing (prefill):

150.0 tokens/s

quant:

IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash at about 150 tok/s prompt processing and 15 tok/s generation in short contexts on an RTX 3090 Ti 24GB with 96GB DDR5-6400 and an AMD 9950X CPU. Setup is llama.cpp with an IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8 GGUF, 98304 context, -ngl 99 and --n-cpu-moe 40, so 40 MoE layers ran on the CPU. User says it works well in the pi harness with the llama.cpp chat template.

generation:

100.0 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports tensor split-mode raising throughput from 70+ t/s to 100+ t/s, with a peak of 130 t/s. Power draw is 750W+.