Local model performance on your hardware

generation:

7.5 tokens/s

prompt processing (prefill):

4000.0 tokens/s

quant:

FP4/FP8

User reports DeepSeek V4.1 at 7.5 t/s decode and over 4000 t/s prefill on an RTX 5090 with 32GB system RAM. Setup streams full FP4/FP8 weights from a dual PCIe5 512GB RAID0 array, dropping prefill buffers and caching a hot set of experts, with missed experts fetched via DirectStorage. A slider trades speed for quality: roughly 1.5bpw-equivalent KL divergence gives 24 t/s. The user notes a RAM cache and CPU expert processing would help with more RAM but 32GB is too little.

generation:

236.0 tokens/s

prompt processing (prefill):

6272.0 tokens/s

quant:

IQ3_S (GGUF)

kv:

int8

User reports Qwen3.8-Flash-Next at 236 tok/s decode and 6,272 tok/s prefill on an RTX 5090, upgraded from an RX 9070 XT. Setup is Strata Engine 0.1.52 with IQ3_S GGUF weights, int8 KV cache, 128K context, MTP speculation with spec=4, and adaptive VRAM/RAM expert caching. The user presents this as a head-to-head against the previous 16GB ROCm card.

generation:

234.4 tokens/s

prompt processing (prefill):

8000-9300 tokens/s

quant:

NVFP4 (GGUF)

User compares TensorSharp, llama.cpp, vLLM, SGLang, velox-inferer and NInfer running Qwen3.8 27B on one RTX 5090 32 GB. TensorSharp averages 234.4 t/s decode with DFlash2 speculation across code, prose and copying, against 80.0 t/s plain. Setup is TensorSharp's direct-CUDA backend with an NVFP4 GGUF target and Q4_K_M drafter, 40,960-token context, greedy decoding and thinking disabled. DFlash2 uses seven draft tokens per step. NInfer leads the speculative decode average at 280.5 t/s and four-request aggregate throughput at 488 t/s; llama.cpp has the lowest short-prompt TTFT at 27 ms plain. The author reports up to roughly 4% decode variation between runs and notes the engines used different quantized checkpoints.

generation:

416.0 tokens/s

quant:

3-bit trellis

User reports Qwen3.8-27B at 416 tok/s on a single RTX 5090 held to a 16 GiB budget, with 684 tok/s aggregate at two lanes. Setup is ignis with a 3-bit trellis body (13.4 GB) and the DFlash2 drafter, 8K context, greedy coding prompts. The 5090 figures stand in for 16 GB RTX 50 cards the author does not own; estimates for real cards are 243-307 tok/s on an RTX 5080, 204-270 tok/s on an RTX 5070 Ti and 103-153 tok/s on an RTX 5060 Ti 16 GB. The author also reports 262,144 tokens of context on two lanes and KL divergence from BF16 at 0.77-0.82x that of the 32 GB NVFP4 build.

generation:

193.0 tokens/s

quant:

EXL3 6 bpw (EXL3)

kv:

INT4

User benchmarks Qwen3.8-27B on an RTX 5090 in VoidInfer at 193 tok/s decode with a 3K prompt, against NInfer at 219 tok/s. VoidInfer runs EXL3 6 bpw with OSCAR INT4 KV cache; NInfer runs NVFP4 with FP8 KV. Both use a 208,896-token window, one request, 1,024 tokens out, prefill excluded. Decode falls to 127 tok/s at 200K context versus 153 tok/s for NInfer, a 34% drop against 30%. VoidInfer scores higher on two coding tests and on AIME 2026 at both 82K and 205K budgets, while NInfer is 10-17% faster on decode and about 2-2.5x faster on prefill.

generation:

80.0 tokens/s

quant:

NVFP4 (GGUF)

User compares six engines running Qwen3.8 27B on a single RTX 5090 32GB, reporting a plain-mode three-workload decode average of 80.0 t/s for TensorSharp rising to 234.4 t/s with DFlash2 speculation. Setup is TensorSharp's direct-CUDA backend with an NVFP4 GGUF target and Q4_K_M drafter, 40,960-token context, thinking disabled and vision enabled; vLLM and SGLang used NVFP4/FP8 safetensors, NInfer and velox a mixed NVFP4/FP8 .ninfer target. DFlash2 raises summarized single-request decode roughly 2.5-3.6x across engines; NInfer leads speculative decode at 280.5 t/s average and four-request throughput at 488 t/s, while llama.cpp has the lowest short-prompt TTFT at 27 ms. The author notes up to about 4% decode variation between runs and that quantized targets and cache formats differ between engines.

generation:

84.2 tokens/s

quant:

NVFP4 (GGUF)

User benchmarks Qwen3.8 27B across six engines on one RTX 5090 32 GB, with llama.cpp reaching the highest plain single-request decode at 84.2 t/s. Setup is llama.cpp with an NVFP4 GGUF target and Q4_K_M DFlash2 drafter, 40,960-token context, greedy decoding with thinking off; the plain figure has no speculation. Other engines used different checkpoints and drafters, so results describe whole configurations. With DFlash2, NInfer led at 280.5 t/s single-request and 488 t/s aggregate across four requests; llama.cpp measured 212.9 t/s and 259 t/s aggregate. The author treats differences below roughly 5% as inconclusive.

generation:

35.0 tokens/s

quant:

IQ2_XXS (GGUF)

User asks whether 2x AMD R9700 with 128GB RAM running GLM-5.3-Flash or DeepSeek-V4.1-Flash under Strata, Maya or Xolotl would beat a single RTX 5090 with 128GB RAM. The only figure given is the 5090 setup at roughly 35 tok/s with Strata on deepseek-v4.1-flash-1017-gsq-rco IQ2_XXS. No benchmark of the dual R9700 configuration is reported.

generation:

100.0 tokens/s

prompt processing (prefill):

1500-2500 tokens/s

quant:

Q6_K (GGUF)

kv:

q4_0

User reports Qwen3.8-27B RVN Q6_K at 100 tok/s at 1K context on a single RTX 5090 32 GB, up from 69 tok/s in LM Studio with the same GGUF and q4_0 KV cache. Setup is a native llama.cpp master build (CUDA 13.1, sm_120) with MTP speculation at 3 drafts, q4_0 K and V cache, 196608 configured context, and about 29.5 GB VRAM at 192K with vision. At ~60K context decode is 79-83 tok/s and at ~120K it is 61-66 tok/s; prefill is ~1500-2500 tok/s on both engines. With Chrome visible at 120K decode drops to 51 tok/s, 59 minimized and 64 closed. A 450 W power limit leaves decode within noise (78.6 vs 82.7 tok/s) but cuts prefill 11%. In a Codex CLI agentic run at 155-160K context the model wrote at a median 81 tok/s with peaks of 149.

generation:

164.6 tokens/s

prompt processing (prefill):

6000-6500 tokens/s

quant:

IQ3_S (GGUF)

User reports Qwen3.8 Flash-Next at 164.6 tok/s decode on an RTX 5090 32GB, with prefill of 6,000~6,500 tok/s. Setup is Strata with an IQ3_S GGUF quant, on a Ryzen 9 9950X3D system with 64GB DDR5. The user previously ran IQ3_XSS and switched to IQ3_S for more speed and quality. The prompt asked for a Minecraft-style Eiffel Tower, which the model built in detail.

generation:

208.2 tokens/s

quant:

NVFP4 (NVFP4)

User reports Qwen3.8 27B at an average 208.2 tok/s on a single RTX 5090 32GB. Setup is SGLang in Docker with the gittensor NVFP4 build and the DSpark speculative drafter, served over an OpenAI-compatible API. Five runs of 256 tokens with thinking on ranged from 148.3 tok/s (cold start) to 232.4 tok/s, averaging 208.2; the user says it sits in the 220s once warm and peaks above 230.

generation:

10.0 tokens/s

User reports DeepSeek V4.1 Flash running at about 10 t/s at 8K context on an RTX 5090. Setup is Strata with a Lexar 4TB NVMe and 64 GB of DDR5 system RAM. User says the speed makes the API cheaper than the electricity cost, but is glad it runs at all on the PC.

generation:

147.0 tokens/s

quant:

EXL3 6 bpw (EXL3)

kv:

INT4

User reports Qwen3.8-27B at 147 tok/s decode on one RTX 5090 under Windows 11, using their own VoidInfer engine. Setup is EXL3 6 bpw weights with OSCAR INT4 KV cache and VeriCache verification against FP16 KV history kept in host RAM, giving roughly 400k shared context on the single card. Against NInfer (NVFP4 + FP8 KV) on the same card, VoidInfer is about 25% slower decode (147 vs 190 tok/s) and much slower prefill (3.5K vs 9.3K tok/s at 3K prompt), but claims KL to BF16 of 0.014 vs 0.080 and 99.3% vs 96.0% agreement with BF16.

generation:

166.3 tokens/s

prompt processing (prefill):

3033.4 tokens/s

quant:

NVFP4 (NVFP4)

kv:

int8

User compares Qwen3.8-Flash-Next NVFP4/GPTQ against Unsloth UD-Q6_K_XL on an RTX 5090 32 GB, with NVFP4 at 166.3 t/s generation and 3,033.4 t/s prompt processing at 4,096 tokens. Setup is Strata with int8 KV cache and a 131,072-token context ceiling; the speed figures are medians of three 256-token generations, and Q6's speed campaign had its control vector enabled. On a fresh 84-question MMLU-Pro subset NVFP4 scored 71/84 against Q6's 67/84 in 10m 57s versus 22m 18s, a difference the user calls not statistically established (McNemar p 0.219). Q6 reached 65.6 t/s at 4K and 2,094.4 t/s prefill, and an earlier mixed test had Q6 ahead on IFEval 9/10 to 7/10.

generation:

144.0 tokens/s

User reports Qwen3.8 27B at 144 tok/s on an RTX 5090 32GB, with 96GB of system RAM in the machine. Setup is Ollama with a 260k context window and KV cache enabled; the user does not state the quant. The user asks whether this is the ceiling, since agentic coding sessions run 8 hours at a time with the GPU at 100% utilization and temperatures around 60 C.

generation:

343.0 tokens/s

quant:

IQ2_XXS (GGUF)

kv:

q8_0

User reports Nemotron 3.5 Bolt 74B-A9B at 343 t/s generation on an RTX 5090 32GB, median of 15 runs (303 to 354). Setup is llama.cpp b11750 with NVIDIA's IQ2_XXS GGUF, q8_0 KV cache, 32K context, single stream, and the LiLiCorr drafter at 55% acceptance; 25.0 GiB of VRAM with the drafter and projector. Without the drafter it runs 166 t/s at 22.7 GiB, and 21.0 GiB without drafter or projector. llama-bench gives pp512 5,166 t/s and tg128 172 t/s. On a 33-task DevOps tool-use eval it passed 26 of 33 (79%, score 0.88) with zero safety failures, 54% unrecorded tool calls and 28.3 GiB VRAM at four slots.

generation:

216.0 tokens/s

prompt processing (prefill):

8917.0 tokens/s

User reports an engine built with the crit toolkit running Qwen3.8-27B at 216 tok/s decode and 8,917 tok/s prefill on an RTX 5090. Setup is a from-scratch engine generated by Claude Opus 5.5 from the crit repository, with 262K context and +0.3% perplexity versus the reference. The build took 3.5 hours with almost no intervention, and the user says it performs in the same league as dedicated engines like NInfer.

generation:

121.4 tokens/s

prompt processing (prefill):

2757.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

int8

User reports Qwen3.8-Flash-Next 125B-A6B running a single-prompt Blender room-building challenge on an RTX 5090 32GB, with engine-verified decode median 121.4 t/s and mean prefill 2,757 t/s. Setup is the strata-nvfp4 fork of Strata with self-converted NVFP4 experts plus Q8_0 dense weights, int8 KV cache, 262,144 max context, MTP draft head with spec 5, and 63 GiB of experts pinned in 192GB DDR5 system RAM. The run took 26m 17s over 103 steps and 132,929 output tokens with 179 tool calls and no compactions, peaking at 194k context; TTFT was 0.85s median and 32.9s max, and the vision encoder ran on CPU.

generation:

37.0 tokens/s

User reports Qwen2.5 32B at 37 tokens per second on an RTX 5090. No engine, quant, context length or KV cache settings are stated. The user is an amateur and has not collected full benchmarks yet, and may post more results later.

generation:

30.1 tokens/s

prompt processing (prefill):

1515.8 tokens/s

quant:

Q4_K_M (GGUF)

User benchmarks Qwen3.5-27B Q4_K_M on an RTX 5090 at 30.09 t/s generation and 1515.83 t/s prompt processing with llama-bench (pp1024/tg400, 5 reps). Setup is llama.cpp b10741 with CUDA 13.3 on Windows 11, -ngl 99 and flash attention on, 15.65 GiB GGUF. A Qwen3 MoE 30B-A3B control on the same rig reaches 134.18 t/s tg, ruling out a global Windows/driver overhead. The user notes native Linux with MTP reaches about 76% of roofline and ~1.7x speedup, while the Windows/Ollama path runs 1.5-1.6x slower at every draft depth. PCIe Gen5 x1 vs x4 gave 27.95 vs 30.09 t/s, so decode is not host-link-bound.