- generation:
- 531.0 tokens/s
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3-0.6B at 531 tok/s decode on an RTX 3090 using a custom CUDA megakernel, 3.9x faster than HuggingFace.
Setup is a batch=1 bf16 cooperative megakernel with static shapes and short context; the project is a learning exercise, not a production engine.
Decode falls to 422 tok/s at position 200 and 158 tok/s at longer contexts, where TensorRT-LLM's 355 tok/s is faster. The kernel is latency-bound by synchronization at 5% memory bandwidth utilization.
- generation:
- 95.0 tokens/s
- quant:
- IQ4_XS-M (GGUF)
- kv:
- TQ5/TQ4
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 95+ t/s generation for Qwen3.8 27B on a single RTX 3090, sustained through 100K generated tokens at 262K context.
Setup is the llamAmpere fork of llama.cpp (v0.4) with an IQ4_XS-M GGUF quant and TQ5/TQ4 KV cache, full GPU offload (-ngl 99), flash attention on, temp=1.
The figure is a minimum across the run; the user notes ~10% speed gain over the previous version and that the KV quant adds under a third of the KLD of the 8-bit to 4.6-bit weight quant.
- generation:
- 111.7 tokens/s
- prompt processing (prefill):
- 3865.4 tokens/s
- quant:
- Q4_K_M (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks LLaMA 3 8B at 111.74 t/s generation and 3865.39 t/s prompt processing on an RTX 3090 24GB.
Setup is llama.cpp with Q4_K_M GGUF weights, 1024-token generation and 1024-token prompt eval, single card.
The same card reaches 115.42 t/s at 512 tokens and drops to 87.49 t/s at 8192 tokens; the F16 build of the same model runs at 46.51 t/s generation and 4239.64 t/s prompt processing.
- generation:
- 19.3 tokens/s
- prompt processing (prefill):
- 690-790 tokens/s
- quant:
- EXL3 3.0 bpw (EXL3)
- kv:
- fp8
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-contextagenticcodingmath
User reports DeepSeek-V4.1-Flash at 19.3 tok/s decode (C1) on a single RTX 3090 24GB with 503 GiB DDR4 and NVMe.
Setup is vLLM 0.13 with EXL3 3.0 bpw quantization, fp8 KV cache, 262,144-token context, and CPU expert offload. All 15,360 routed experts are pinned in host RAM (190.5 GiB), with an AVX2 CPU tier computing cold decode misses and an elastic VRAM expert cache holding the hottest experts.
Decode scales to 24.0 tok/s at 2 streams and 26.6 tok/s at 4 streams. Prefill ranges from 690 to 790 tok/s across 8k to 261k tokens. An experimental Intel Arc B70 expert tier reached 41.3 tok/s at 4 streams (+55%), but quality A/B is pending. Prefill fidelity failed the inherited guard (top-1 0.9856 vs 0.988 required, mean KL 0.0068 vs 0.00103 allowed).
- generation:
- 82.7 tokens/s
- quant:
- QAT (W4A16)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Gemma 4 12B at 82.7 tok/s decode on vLLM and 85.0 tok/s on Ollama on one RTX 3090 24 GB capped at 300 W.
Setup is vLLM 0.30.0 with Google's W4A16 QAT checkpoint and Ollama 0.34.4 with gemma4:12b-it-qat, temperature 0, 256 tokens out, one request at a time.
At sixteen concurrent requests vLLM reached 506.8 tok/s against Ollama's 142.3 tok/s; first token was 213.0 ms on vLLM against 441.9 ms on Ollama. Board energy at sixteen requests was 590 joules per 1,000 tokens on vLLM and 1,384 on Ollama. On 108 questions the two agreed on 63 strictly and 98 leniently.
- generation:
- 100.0 tokens/s
- quant:
- IQ4_XS (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next-OrcaRouter IQ4_XS at a bit over 100 t/s decode on short context on 2x RTX 3090 24GB with Strata.
Setup is Strata on 2x RTX 3090 with IQ4_XS weights, which keeps the useful experts on the GPUs and the rest in 64GB DDR5 system RAM, vision enabled, experimental context extension configured for 512K.
A prior vLLM setup did about 60-70 t/s decode. Context extension runs: 200K prompt at ~2,004 t/s prompt processing and ~73 t/s decode; 400K at ~1,633 t/s and ~46 t/s; 900K at ~1,110 t/s and ~44 t/s.
- prompt processing (prefill):
- 3843.0 tokens/s
- quant:
- W8A8 (safetensors)
- kv:
- bf16
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.8-27B W8A8 on 4x RTX 3090 capped at 220 W with 3,843 t/s prefill at 512 prompt tokens (single request); decode is given only per task, 146.6 t/s narrative and 265.5 t/s code, with no overall figure.
Setup is SGLang v0.5.19 on 4x RTX 3090 with TP=4, W8A8 INT8 weights, bf16 KV cache, DFlash2 speculative decoding with 8 draft tokens, HiCache ratio 1.5 and a 262144 context window.
The c=1 prefill sweep ran 3,843 t/s at 512 tokens up to a 5,058 t/s peak at 4,096, and about 4,000-4,300 t/s at 10K and 3,029 t/s at 90K. All four GPUs were capped to 220 W of the 350 W default (62%), so every figure is power-limited. A concurrency sweep reached 1,811 t/s aggregate code decode at c=16 (113.2 t/s per request).
- prompt processing (prefill):
- 834.8 tokens/s
- quant:
- PQ2_0 (GGUF)
- kv:
- q8_0
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8 27B Bonsai 2 at PQ2_0 (2.13 bpw, 7.21 GB) on one RTX 3090 with the llama-cpp-prism ternary fork, prefilling at 834.8 t/s at 10K.
Setup holds the full 262144 context on the single card at 17.1 GB with q8_0 KV; the quant drops the MTP head, so the run is un-drafted. Decode is given per workload only, 42.76 t/s narrative and 41.01 code, with 141 ms TTFT.
The same harness PR adds an exl3/TabbyAPI engine on the 2x 3090 reference rig, whose CPU-computed MoE slugs decode at 50.5 and 39.3 T/s at 204800 context with Q4 KV.
- prompt processing (prefill):
- 904.0 tokens/s
- quant:
- 3.00bpw (EXL3)
- kv:
- fp8
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-27B EXL3 3.00bpw on a bare RTX 3090 24 GB through SGLang with the sglang-exl3 plugin, with cold prefill at 904 tok/s.
Setup uses fp8 KV cache, MTP speculative decoding on, thinking off, one request, at the full 262,144-token window; the 2.4 GB token embedding sits in pinned host memory.
Decode is given per workload: 98.8 t/s on prose, 143.3 on code and 89.6 on prose with a 32k prompt. A Qwen3.6-35B-A3B EXL3 recipe on the same card reached 251.9 prose and 352.1 code. A prior AWQ-INT4 27B recipe with no draft is cited at 45.1 t/s prose.
- generation:
- 17.5 tokens/s
- quant:
- IQ4_XS (GGUF)
- kv:
- Q8_0
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Step-3.5-Flash at about 17/18 t/s on llama.cpp with an RTX 3090 and two RTX 5060 Ti 16GB cards, 64GB RAM and a Ryzen 9 5950X.
Setup is llama.cpp with an IQ4_XS GGUF around 96GB, 16384 context and Q8_0 KV cache, with the model split across GPU and system RAM.
The same model on ik_llama.cpp reaches only 10 t/s and is a bit unstable, so the user asks whether the multi-GPU mix is the problem or whether the setup can be refined.
- generation:
- 129.1 tokens/s
- prompt processing (prefill):
- 5124.4 tokens/s
- quant:
- Q4_0 (GGUF)
- kv:
- f16
- flash attention:
- on
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks the new tensor parallel (split mode tensor) implementation in llama.cpp against graph parallel in ik_llama.cpp on a 2x RTX 3090 system with 48 GiB of VRAM.
Setup is llama.cpp with Q4_0 quantized Llama 3 8B, f16 KV cache, 65536 context, 100 GPU layers, and flash attention enabled. The run crashed with CUDA out of memory at 34816 tokens of context.
User reports 129.08 t/s generation and 5124.38 t/s prompt processing at zero context, degrading to 26.89 t/s generation and 3092.68 t/s prompt processing at 34816 tokens. User calls the PR a gimmick not ready for prime time, noting memory is not released and performance lags well behind ik_llama.cpp graph parallel.
- generation:
- 33.7 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
- kv:
- q8_0
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User reports Qwen3.8-Flash-Next at 33.7 t/s decode on six RTX 3090s over PCIe without NVLink, up from 10.2 t/s on stock llama.cpp.
Setup is a llama.cpp-multigpu fork with UD-Q4_K_XL GGUF and q8_0 KV cache, 5 slots of 262k context, layer split across the cards.
Prefill rose from 263 to 2111 t/s at 250k context on the 3090s, and 744 to 7403 t/s on six rented RTX 4090s. Five-session decode went from 2.3 to 27.3 t/s per session on the 3090s. The user notes the 10.8x figure is against a 2.3 t/s baseline and should not be quoted. N-gram speculation gave 2-2.5x on code rewrites and 1.5x on code explanation but was off for the benchmarks. MTP was tried and was slower, so it is not included.
- generation:
- 43.9 tokens/s
- prompt processing (prefill):
- 1817.0 tokens/s
- quant:
- Q6_K_XL (GGUF)
- kv:
- int8
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.8-Flash-Next at 43.9 tok/s generation and 1817 tok/s prompt processing on an RTX 3090 + RTX A4000 at 32k context.
Setup is Strata with UD-Q6_K_XL GGUF and int8 KV cache, 262144 native context, speculative decoding with MTP, experts offloaded to CPU.
At 200k context Strata Q6 does 48.8 tok/s versus llama.cpp's 10.6 tok/s, and TTFT drops from 595 s to 118 s. Q6 is 12-31% faster than Q8 and 19 GB smaller.
- generation:
- 97.0 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.8-Flash-Next at 97 t/s decode on 2x RTX 3090 with Strata, with about half the experts in system RAM.
Setup is Strata v0.1.40.1 with unsloth UD-Q4_K_XL GGUF, a layer split, speculative decoding, and 121 GB DDR4 on a Ryzen 9 3950X.
The 97 t/s is at a 95% expert hit rate; shrinking the cache to 90% and 81% hit rates gave 84 t/s and 68 t/s. The user estimates +34% decode from eliminating misses on their box and about +75% headroom at a single-24GB-card share, and reports +16% decode from --pcie-frac 0.2 --spec-min-p 0.7 over about 280 runs. The prefetch feature is not implemented yet.
- generation:
- 44.2 tokens/s
- prompt processing (prefill):
- 1710.2 tokens/s
- quant:
- Q8_K_XL (GGUF)
- kv:
- q8_0
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.8-27B on dual RTX 3090s, comparing llama.cpp against vLLM int8. llama.cpp with Q8_K_XL GGUF and q8_0 KV cache reaches 44.22 tok/s decode on narrative prompts, roughly half the 84.56 tok/s of the vLLM int8 w8a16 container with fp8 KV cache and MTP.
Setup uses two RTX 3090s with a 0.55/0.45 split and ngl=99 for llama.cpp; vLLM uses custom all-reduce with P2P granted. Prefill at 10K tokens is 1710.16 tok/s for llama.cpp versus 1891.57 tok/s for vLLM.
TTFT is about 5x slower on llama.cpp (337 ms vs 69 ms narrative). The 90K prefill run for llama.cpp was stopped early. User considers adding more system RAM for LMCache instead.
- generation:
- 60.0 tokens/s
- quant:
- exl3-5.00bpw (EXL3)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingtool-use
User reports Qwen3.8 27B at ~60 t/s on a single RTX 3090.
Setup is ExLlamaV3 with an exl3-5.00bpw quant at ~150k context; prefill was not benchmarked. The machine has an Intel 12400 CPU and 32GB DDR4 RAM.
User asks whether to buy 128GB DDR5 and a Ryzen 9700X to run the IQ3 quant under Strata, and how Qwen3.8 27B compares to Qwen Next on output accuracy for coding and Hermes assistant use.
- generation:
- 19.8 tokens/s
- prompt processing (prefill):
- 437.8 tokens/s
- quant:
- UD-Q4_K_XL (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports GLM-5.3-Flash at 19.8 tok/s decode and 437.8 tok/s prefill on one RTX 3090 24 GB with 251 GB RAM.
Setup is the Strata engine with a UD-Q4_K_XL GGUF pack, --chunk 4096, 16K-token prompt, tiered expert cache streaming from RAM and NVMe.
Decode splits into 12.4 ms expert copies, 0.2 ms expert kernels and 37.2 ms dense/sync per 49.8 ms token. A 64K prompt prefills at 420.7 tok/s; a 1K prompt at 132 tok/s prefill and 19.8 tok/s decode; disk-only tier decodes at 1.15 tok/s. --chunk 8192 OOMs on 24 GB.