NVIDIA RTX 4090 reports

generation:

130.0 tokens/s

quant:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B Uncensored-HauHauCS at about 130 t/s decode on an RTX 4090. Setup is NInfer with INT8 quantization, 262K context, and a model size of about 17 GB. The user notes this is a port in progress, with a CPU build possibly running overnight; the 130 t/s is an expected figure rather than a measured result.

generation:

170.6 tokens/s

quant:

AWQ INT4 (AWQ)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Mistral 7B at 170.63 t/s on an RTX 4090 with TensorRT-LLM, compared to 100.43 t/s with llama.cpp. Setup is TensorRT-LLM 0.7.1 with AWQ INT4 quantization, batch size 1, input length 2048, output length 512, on a single RTX 4090 with 32GB system RAM. TensorRT-LLM was 69.89% faster than llama.cpp on the same hardware. The post also reports results for RTX 3090 (144.19 t/s), RTX 4070 Laptop (51.57 t/s), and RTX 4090 eGPU (104.95 t/s).

generation:

80.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User asks how a MacBook Pro M5 Pro 64GB would compare to their work setup of an RTX 4090 with 128 GB DDR5 for running local LLMs. They report running Qwen3.8-27B at Q4_K_M under llama.cpp on the RTX 4090 at roughly 80 t/s with a ~127k context window, though they note they are on holiday and cannot verify the figures. The question is whether to buy the 64GB or 32GB MacBook variant for local LLM use at home.

generation:

149.0 tokens/s

quant:

INT8

kv:

E8 4-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at up to 149 tok/s code decode on one RTX 4090. Setup is NInfer with INT8 prefill and E8 4-bit KV cache, full native 262K context, MTP3 speculative decoding, sm_89-retuned attention prefill, vision, and llama.cpp-compatible /metrics and /slots. User says it is their go-to even with Strata, which is limited to IQ2_XS.

generation:

125.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen 3.8 27B with vision and MTP at 125 tok/s on an RTX 4090. Setup uses the integrated GPU for display output, freeing about 2.5GB of VRAM, which raised the context window from 65k to 132k. The user notes swapping the display cable back for gaming is a minor annoyance.

generation:

80-90 tokens/s

quant:

WNA16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports completing a full 262,144-token sequence with Qwen3.8-Flash-Next on 2× RTX 4090 24 GB, with a boundary run of 261,991 input + 128 output tokens in 123.3 s (≈2,150 input tok/s). Setup is vLLM (v0.5.0 image) with TP2/EP2, 256K context, hot 84 profile, CPU_OFFLOAD_GB=30, MTP_DEPTH=3, and fast-256k switches; peak VRAM was ~23.8 GiB per card. Decode measured 80–90 tok/s (mean ≈86) with the SM 8.9 gate patch, prefill 1,772–1,814 tok/s at ~14K and 2,294–2,316 tok/s at ~61K; the user notes the x8/x8 PCIe bifurcation likely caps stream-staging and that hot 88 OOM'd on an earlier build.

generation:

120.1 tokens/s

prompt processing (prefill):

4700-4800 tokens/s

quant:

IQ3_S (GGUF)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 120.10 tok/s decode at 32K context on one RTX 4090, with 112.55 tok/s at 128K and roughly 4,700-4,800 tok/s on uncached 32K prefill. Setup is Strata with GSQ IQ3_S GGUF weights, int8 KV cache and a 32K resident window, MTP speculation with four draft tokens, 11 pool workers and a 0.35 PCIe fraction; all experts and the IQ4_NL ngram table stay in RAM, so the run is partially CPU-offloaded. The host is a Ryzen 7900 with 192 GB RAM at a 280 W GPU limit. The 120.10 tok/s figure is the median of launch medians for the selected 24-expert-cache profile, a +0.88% change over the 96-swap baseline; the 128K figure is +4.26%. Earlier matched decode cells give 110.8 tok/s median for the text profile and 97.65 tok/s for the GPU vision profile, and a matched A/B isolates GPU vision residency at 114.85 versus 104.25 tok/s (-9.2%). The 397B partial-offload probe reached 12.42 tok/s with quality unqualified.

generation:

106.1 tokens/s

prompt processing (prefill):

147.0 tokens/s

quant:

IQ2_XS (GGUF)

kv:

8-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 106.1 tok/s whole-request on an RTX 4090 24GB. Setup is Strata with IQ2_XS, 128K context, 8-bit KV cache, MTP on, KV streaming on, images on, speed projection off. The IQ3_XXS tier ran at 98.1 tok/s whole-request with ~120 tok/s peak decode and 131 tok/s prefill; the user notes it was 3.7x longer and qualitatively deeper for only 8% less speed. Both tiers match the README's 24 GB card estimate of 100-140 t/s.

generation:

166.4 tokens/s

quant:

IQ2_XS (GGUF)

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 166.36 output tokens/s on a rented RTX 4090 24 GB. Setup is Strata with IQ2_XS, INT8 KV cache, 65,536-token context and 32,768 GPU-resident tokens, MTP draft 4 at 0.5 minimum probability. The rate is a weighted generation figure (sum of completion tokens over predicted time), not whole-request throughput; MTP acceptance was 84.9% and peak sampled GPU memory 23,918 MiB. The generated Hill Climb Racing-style game was rejected for poor vehicle physics and game feel.

generation:

127-140 tokens/s

kv:

rk4v4-e8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares Qwen3.8-27B dense on NInfer against Qwen3.8-Flash-Next IQ2_XS on Strata 0.1.39, both on one RTX 4090 24 GB with 262K context and MTP. The 27B decodes at 127-140 tok/s with rk4v4-e8 KV and 23,134 MiB peak VRAM; the MoE decodes at 173-185 tok/s with q4_0 KV in VRAM and experts in host RAM, peaking at 23,936 MiB. Recall at 55K/117K/200K is 3/3 for both, with the 27B at 3,037/2,391/1,857 prompt tok/s and the MoE at 3,494/3,904/3,833. On three graded tickets the 27B scored 112/150 versus 86/150 for the MoE. The user attributes the difference to cache reuse: 88.5% for the 27B, including one turn serving 172,343 of 172,578 tokens from cache with first token in 524 ms, against 32% for the MoE, whose typical turn re-read 107K tokens in 28 s with conversation cache off on 32 GB RAM. Limits are n=1 per ticket, one grader model, and the MoE ran first.

generation:

67.0 tokens/s

prompt processing (prefill):

2600-2820 tokens/s

quant:

escha 2-bit (safetensors)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B-Escha-W2 at 67.0 tok/s single-stream decode on an RTX 4090. Setup is SGLang with a 2-bit escha quant (2.469 bits/weight, 10.15 GB of weights) at 64k context, CUDA graphs on, prefix caching off. With the model's own MTP head as a speculative draft the same card reaches 129.3 tok/s (1.92x). Peak server throughput is 649 tok/s at 16 streams; prefill is ~2,600-2,820 tok/s and TTFT at a 2k prompt is 0.73 s. The card also ran 87.1 tok/s on an RTX 5090 and 40.7 tok/s on an RTX 3090 with ESCHA_ROUTE=blackwell.

generation:

135.4 tokens/s

quant:

EXL3 3bpw (EXL3)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 135.4 tok/s decode on a bare RTX 4090 24 GB. Setup is SGLang with EXL3 3bpw weights, 207,356-token KV pool, 22.9 GB peak, 239 ms TTFT over 3 samples. A separate depth sweep on the same card gives 184 tok/s code and 126 tok/s prose at ~50 prompt tokens, falling to 115 and 91 tok/s at 133k-190k tokens; cold prefill was 2,112 tok/s at 19k and 1,636 at 134k. The recipe is the unchanged RTX 3090 SGLang EXL3 config, and it now displaces a TabbyAPI SC4bpw recipe that measured 86 tok/s. No coherence ladder, soak test or 3 bpw vs 4 bpw quality comparison was done.

generation:

158.9 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen2.5-Coder-7B-Instruct Q4_K_M on an RTX 4090, comparing llama.cpp at 158.90 tok/s decode against apr 0.64.0 at 103.26 tok/s. Setup is llama.cpp build 39173bcac and apr 0.64.0 CUDA with the same GGUF, profile medium, 15s warmup, 3 x 30s runs, 10s cooldown, concurrency 1, streaming. Decode is 0.650x of llama.cpp, with ITL p50 6.29 ms vs 9.68 ms and throughput 1.24 vs 0.77 req/s. The user notes apr's figure is 2.6x the ~40 tok/s 7B reference and that an earlier 11.8 tok/s report was an end-to-end rate including ~9.7s model load.

generation:

165.9 tokens/s

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Distill at 165.9 t/s decode on an RTX 4090 24GB. Setup is llama.cpp with llama-swap, q8_0 KV cache, flash attention on, 256K context, and speculative draft-MTP enabled. The post is a registry of many models on the same card; the highest measured decode listed is 363 t/s for Nemotron-Lightning-3.5, and context cliffs are noted for several models.

generation:

115.0 tokens/s

quant:

IQ4_XS (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 115 t/s decode on a single RTX 4090 24 GB. Setup is llama.cpp b11115 with llama-swap v257, IQ4_XS GGUF quant, q8_0 KV cache, 196608 context, and draft-MTP speculative decoding. The user notes the DeepSWE result is a single task, not an average, and that the model passed 40/43 hidden tests and kept 109/109 existing tests.

generation:

80.0 tokens/s

quant:

Q4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a llama.cpp config tuner that measured a 27B Q4 model on an RTX 4090, improving average generation from 60 to 80 tok/s and peak from 80 to 110 tok/s. Setup is llama.cpp with Q4 weights and a 100K context length; the tuner sweeps speculative decoding, KV cache quant, GPU offload layers, and batch/ubatch, running 3 passes and taking the median. At 100K context the tuned config reached 60 tok/s versus 40 hand-tuned. The user says speculative decoding was the biggest single win and asks whether it helps as much on a 4090 as on a 5090.

generation:

107.0 tokens/s

prompt processing (prefill):

5008.0 tokens/s

quant:

Q4/Q5

kv:

rk4v4-e8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 107 tok/s decode with MTP on mixed prompts and 5,008 tok/s prefill at 2K tokens on one RTX 4090 under native Windows. Setup is NInfer with Q4/Q5 weights, rk4v4-e8 KV cache, 262K context per request, MTP with 3 draft tokens and n-gram chaining. Decode reaches 289 tok/s with MTP + n-gram on edit-style prompts and 211 tok/s with DFlash2 on code; prose with thinking on gives 99 tok/s with DFlash2 versus 85 tok/s with MTP + n-gram. Against llama.cpp on the same card, prefill is 5,008 vs 2,729 tok/s at 2K, a 64K prompt takes 17.2 s vs 29.8 s and a 128K prompt 43.0 s vs 73.6 s. Ternary Bonsai 2 27B reaches 188 tok/s decode on mixed prompts, 532 tok/s with MTP + n-gram on edits and 360 tok/s aggregate with three concurrent requests. The 4090 also drives a 4K monitor at 60 Hz, costing about 15% of decode.

generation:

129.0 tokens/s

prompt processing (prefill):

5600.0 tokens/s

quant:

Q5_K_M (GGUF)

kv:

turbo3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 26B-A4B at ~129 tok/s single-request generation on a single RTX 4090, with ~5,600 tok/s prefill. Setup is a TurboQuant fork of llama.cpp with Q5_K_M weights, turbo3 3-bit KV cache, Flash Attention, and a multimodal projector, using 22.3 GB of 24 GB VRAM at 262,144 tokens of context. Concurrent generation is reported at ~83-85 tok/s. A 15-test agentic coding benchmark through Claude Code passed 15/15 with an average quality of 4.2/5.

generation:

188.0 tokens/s

quant:

Q4/Q5 (GGUF)

kv:

E8 4-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Ternary Bonsai 2 27B at 188 tok/s decode averaged over six mixed prompts on a single RTX 4090 under native Windows. Setup is NInfer, a from-scratch C++/CUDA engine, with Q4/Q5 weights, E8 4-bit KV cache, MTP plus n-gram speculative decoding, and 262K context per request. On edit-style prompts the model reaches 250 tok/s with MTP and 532 tok/s with MTP + n-gram; through the server on 7K-11K-token file edits with thinking on it goes 215 to 378 tok/s. Three concurrent requests give 360 tok/s aggregate. Against Prism's llama.cpp fork on the same card: 101 vs 77 tok/s decode and 3,061 vs 1,363 tok/s pp512. N-gram gains depend on repetition and are zero on free prose (167 tok/s both ways).

generation:

148.6 tokens/s

prompt processing (prefill):

1849.0 tokens/s

quant:

E8 4-bit

kv:

INT8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 148.6 tok/s on a single RTX 4090 with MTP3 speculative decoding at 81.0% draft acceptance. Setup is NInfer with the official groupwise artifact, INT8 KV cache, CUDA Graphs on, prefill-chunk 1024, single request greedy decoding. Without speculation decode is 50.5 tok/s; at 128K context depth it is 39.6 tok/s. Prefill is 1,849 tok/s at 64K and 1,561 tok/s at 128K. The E8 4-bit KV default profile costs about 5.7% decode versus INT8, giving roughly 126.6 tok/s on the code probe.