Local model performance on your hardware

generation:

80.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports running Qwen3.8 27B at an average generation speed near 80 t/s on a workstation with 2x RTX 4090 and 128 GB DDR5. Setup uses a K_M quant with a 128k context size; the harness is pi, dsh or qwen code, with dsh judged worse than the other two. The user compares it unfavourably with a free hosted DeepSeek Flash, which solved their data collection, figure and table preparation and LaTeX assembly tasks much faster and with better results, and asks whether local models are still worth it.

generation:

95-105 tokens/s

prompt processing (prefill):

2076.0 tokens/s

quant:

IQ3_S (GGUF)

kv:

int8

User reports Qwen3.8-Flash-Next at 2076 tok/s prompt processing on a 145k-token prompt and 95-105 tok/s decode depending on task, with peaks of 145 tok/s, on a single RTX 4090 24 GB. Setup is Strata 0.1.43 with an IQ3_S pack, int8 KV cache, 204800 context, resident experts and MTP speculation, on a Ryzen 9 3900X with 48 GiB DDR4-3200 under Windows 10. A grid search over flags raised decode from 97 to 142 tok/s; the best arm was --kv-resident 20480 plus --adapt-async 1 and STRATA_ADAPT_LAG=2, giving 1526 and 2309 tok/s prompt processing at 10k and 20k. The user notes Windows does not pin memory with CUDA, so every host-to-GPU expert copy is CPU-driven.

generation:

20.0 tokens/s

quant:

3.05bpw_h5_ng5 (EXL3)

kv:

Q4

User reports Qwen3.8-Flash-Next 125B at 20.0 tok/s decode on an RTX 4090 at the full 262k context window, with experts streamed from NVMe SSD. Setup is ExLlamaV3 with a 3.05bpw_h5_ng5 quant and Q4 KV cache, 16 GB VRAM and only 8 GB system RAM allocated. At 100k context the same card gives 20.8 tok/s decode and ~1,536 tok/s prefill; short 8k context gives 28-33 tok/s prose and 19-22 tok/s code with ~1,080 tok/s prefill. An RTX 3080 Laptop with 16 GB VRAM and 32 GB RAM reaches ~18-21 tok/s decode and ~650-700 tok/s prefill at 100k context, peaking at 15.5 GB VRAM.

generation:

47.0 tokens/s

prompt processing (prefill):

2600-3000 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

User reports 47 t/s generation. Source evidence (verbatim): Source: https://computingforgeeks.com/run-qwen3-8-27b-locally/ # Run Qwen3.8-27B Locally: VRAM and Speed Tested Hardware: RTX 4090 24 GB, Ubuntu 24.04, llama.cpp build b11200 (CUDA 12.8), Ollama 0.34.4. Model Qwen3.8-27B unsloth UD-Q4_K_XL, flash attention on, -ngl 99. Without MTP (plain run), 8K context, f16 KV: 47.0 tok/s generation, 17.0 GiB VRAM. 64K f16: 47.0 tok/s. 131K q8_0: 46.6 tok/s. 262K q4_0: 46.5 tok/s, 22.2 GiB. llama-bench generation without MTP: Q4_K_XL 47.9, Q5_K_XL 41.5, Q6_K 39.6 tok/s. llama-bench prompt processing: 2,600 to 3,000 tok/s. A 196,971-token prompt: prefill 121.5 s (1,621 tok/s), generation 26.6 tok/s. With MTP speculative decoding (draft-mtp, mtp-Qwen3.8-27B-Q4_0.gguf) at 8K context: Q4_K_XL 109.4 tok/s (runs 102.8, 109.4, 113.2), Q5_K_XL 95.8 tok/s, acceptance 69%.

generation:

180.0 tokens/s

prompt processing (prefill):

9000-10000 tokens/s

quant:

AWQ 4-bit (AWQ)

User reports Qwen3.6-35B-A3B at about 180 tok/s single-stream decode on two RTX 4090s at roughly 50k context, dropping to about 160 tok/s at roughly 110k. Setup is sglang with a QuantTrio AWQ 4-bit quant, TP=2 over PCIe with no NVLink. Prefill is 9-10k tok/s, and 8 concurrent requests reach 985 tok/s total. The user runs it as a GitLab code-review agent and says repetition is the main problem with local models, with the model looping on itself ending about 7% of runs (56 of 847).

generation:

16.4 tokens/s

User reports Maya-L at a mean 16.4 tokens/s decode on an RTX 4090 24 GB, with a Ryzen 9 9950X3D and 192 GB DDR5. Setup is the Maya engine (v1.0.24) with a 156.3 GB GGUF model folder, 131072 context, one GPU, and STRATA_GLM_PCIE_SHARE set to 0.10. Decode ranged from 15.5 to 18.1 tokens/s over three 256-token answers; prefill was 463 tokens/s at 2048 tokens and 1371 tokens/s at 8192 tokens.

generation:

95.0 tokens/s

quant:

Q3S

User reports a Swift-1.5 Qwen3.8 model at about 95 t/s on dual RTX 4090s after updating Strata from 1.45 to the 2.0 beta. Setup is Strata with Q3S weights; no context length or KV cache quant stated. The same setup previously reached around 155 t/s on Strata 1.45, so the user asks whether others see the same slowdown.

generation:

72.0 tokens/s

prompt processing (prefill):

2000.0 tokens/s

quant:

UD-IQ4_XS (GGUF)

User reports Qwen3.8-Flash-Next at 72 t/s decode on an RTX 4090 with Strata, using Unsloth's UD-IQ4_XS quant at 262K context. Setup is Strata with route tail skip disabled (STRATA_ROUTE_TAIL_SKIP=0) and RAM-budget mode; prompt speed was about 2,000 t/s at 36K and about 2,900 t/s at 136K. The default skip of 7 gave 93 t/s at both 36K and 136K but doubled KL against Q8_0 (0.053 to 0.115), so the user chose quality. Strata's calibrator moved --spec-min-p from 0.5 to 0.7 for about +4% and confirmed --pcie-frac 0.20; values at 0.55 or higher cost 20-40%.

generation:

27.0 tokens/s

prompt processing (prefill):

180-320 tokens/s

quant:

GSQ Uncensored

User reports DeepSeek-V4.1-Flash (552B) at 27 tok/s decode on a single RTX 4090, with prefill of 180-320 tok/s. Setup is the Xolotl engine (based on Strata) with a 152 GiB uncensored GSQ quant, 512K context, weights spread across 24GB VRAM, 90GB DDR4 and NVMe on a Ryzen 7 5700X. User also reports 21/21 HumanEval, 10/10 tool calls and 3/3 agentic bug-fix loops through the real server with thinking on, and says the engine also runs GLM-5.3-Flash, Qwen3.8-Flash-Next and Qwen3.8-27B.

generation:

30.0 tokens/s

prompt processing (prefill):

864.0 tokens/s

quant:

UD-IQ3_XXS (GGUF)

kv:

q8_0

flash attention:

on

User reports 30 t/s generation and 864 t/s prompt processing. Source evidence (verbatim): Source: https://gist.github.com/ryan4yin/48617bbddacc7067f10799770b7cc33f # Qwen3.8-Flash-Next on an RTX 4090 with llama.cpp (mainline), 96GB DDR5 Hardware: RTX 4090 24GB, Intel Core Ultra 7 270K Plus (24C/24T), 96GB DDR5-5600 (2x48GB dual channel). Engine: mainline llama.cpp, image ghcr.io/ggml-org/llama.cpp:full-cuda13. Model: unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ3_XXS (about 82GB) plus mmproj-F16. Flags: --ctx-size 120000 --fit on --fit-target 256 -t 20 -tb 24 -b 6144 -ub 6144 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 -np 1. Experts are offloaded to system RAM. No speculative decoding or MTP (llama.cpp does not support the MTP head yet). Measured with this llama.cpp config: decode about 30 t/s near 6K context, 22.2 t/s at 90K context, 19.5 t/s at 110K input; prefill 864 t/s at batch 6144 (762 t/s at batch 2048).

generation:

46.0 tokens/s

quant:

QAT

User reports Gemma 4 31B QAT at 46 tok/s on an RTX 4090. Setup is Glyd 0.31 with Google's QAT 4-bit checkpoint (4.5 bits per weight) at 32K context, using 21.8 of 24.6 GB. With Google's draft model, chat prompts reach 78-80 tok/s and predictable text up to 176 tok/s. A 30K-token prompt took 20 s. On an RTX 3090 the same model runs at 26 tok/s; a 12B QAT on an RTX 4080 SUPER runs at 82.5 tok/s plain and 109 tok/s with the draft model. Quality vs Google's QAT weights: mean KLD 0.018, median 0.003, top-1 96%, GSM8K 72/75, MMLU-Pro 70/75.

generation:

170.0 tokens/s

quant:

QAT

User reports Gemma 4 26B-A4B QAT at about 170 tokens a second on one RTX 4090, with a full two-call pass taking 5 seconds. Setup is Ollama with QAT weights, one card, 96 GB system RAM; vLLM was faster at 3.5 seconds per pass but held 94% of the card against Ollama's 70%. Sending both calls of a pass together cut it to 3.8 seconds on Ollama and 2.8 seconds on vLLM. MedGemma 27B Q4_K_M took 18 seconds per pass. The user asks about speculative decoding, other engines, other models, and CPU offload.

generation:

43.0 tokens/s

quant:

~3 bits per weight

User reports Nemotron 3 Super (120B, 12B active) at 43 tok/s on a single RTX 4090. Setup uses the glyd engine with a ~3 bits per weight quant (48.7 GB), hot experts on GPU and the rest computed on CPU from RAM; on 32 GB machines the remainder streams from SSD. Same 4090 with llama.cpp and Unsloth Q2_K_XL does 17 tok/s. Also reports 37 tok/s on RTX 3090, 35 tok/s on a 16 GB card, and 13-18 tok/s on a 32 GB RAM PC. Cites GSM8K 97% and MMLU-Pro 77%.

generation:

53.7 tokens/s

prompt processing (prefill):

885.0 tokens/s

quant:

4.0bpw (EXL3)

kv:

FP16

User reports Jev_Qwen3.8-27B at 53.7 t/s decode and 885 t/s prefill on an RTX 4090. Setup is TabbyAPI with EXL3 4.0 bpw and FP16 KV cache, single GPU, max batch size 1, thinking disabled. Decode stayed near 54 t/s at short context and 48.3 t/s at 32K. A warmed TabbyAPI smoke test generated at 53.52 t/s, while a direct ExLlamaV3 probe measured 40.475 t/s. Observed GPU memory was approximately 16.4-16.7 GiB.

generation:

35.9 tokens/s

quant:

EXL3 4.00 bpw (EXL3)

kv:

FP16

User reports AREX-2, a 27B dense multimodal agent model built on Qwen3.8-27B, at 35.94 tok/s decode on an RTX 4090 24 GB. Setup is ExLlamaV3 with an EXL3 4.00 bpw quant and FP16 cache, 8192-token runtime context, batch size 2, ~17 GiB artifact. The figure is a small local smoke benchmark, not a standardized result; a TabbyAPI smoke test on the same card saw about 49.4 tok/s on a very short request. Long-context quality and memory scaling at 32K/64K/262K were not validated.

generation:

33.6 tokens/s

prompt processing (prefill):

1944.0 tokens/s

quant:

4.05bpw (EXL3)

User benchmarks a MoE CPU-offload patch for ExLlamaV3 that streams expert weights directly from a page-locked shared arena, removing the host staging memcpy. On an RTX 4090 with a Ryzen 9 7900X, the Qwen3.8 Flash-Next 125B-A6B model at 4.05bpw reaches 1944 tok/s prefill at 4096 context and 33.6 tok/s decode at context 0 on Windows, roughly doubling upstream prefill. Setup is ExLlamaV3 with EXL3 4.05bpw weights, 32768 context, chunk size 4096, and CPU-offloaded MoE experts. The patch is PCIe-bound at 26-30 ms per ~1 GiB layer. Prefill figures are medians of 5 runs with min-max ranges; WSL2 reaches 2044 tok/s at 4096 context. Decode varies more with machine state than with the patch.

generation:

18.8 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

q8_0

User reports Qwen3.5-122B-A10B at UD-IQ4_XS decoding at 18.75 tok/s on 2x RTX 4090 with expert offload to a Threadripper TRX50 with 128 GB DDR5 (2 of 4 memory channels populated). Setup is llama.cpp on Windows with q8_0 KV cache; the 122B took 11.91 s per agent tool call against 3.40 s for a 27B, so the author shelved it. The rest of the post covers the 27B as a coding subagent: at 64k per agent, Q8_K_XL ran 3 agents (112 s median wait), Q6_K_XL 5 (168 s) and Q4_K_M 9 (330 s), with aggregate prefill near 1,500 tok/s whatever the slot count. Qwen3.6-35B-A3B decoded at 78 tok/s and Qwen3.8-Flash-Next topped out at 23.7 tok/s with experts in system RAM.

generation:

130.0 tokens/s

quant:

INT8

User reports Qwen3.8 27B Uncensored-HauHauCS at about 130 t/s decode on an RTX 4090. Setup is NInfer with INT8 quantization, 262K context, and a model size of about 17 GB. The user notes this is a port in progress, with a CPU build possibly running overnight; the 130 t/s is an expected figure rather than a measured result.

generation:

170.6 tokens/s

quant:

AWQ INT4 (AWQ)

User benchmarks Mistral 7B at 170.63 t/s on an RTX 4090 with TensorRT-LLM, compared to 100.43 t/s with llama.cpp. Setup is TensorRT-LLM 0.7.1 with AWQ INT4 quantization, batch size 1, input length 2048, output length 512, on a single RTX 4090 with 32GB system RAM. TensorRT-LLM was 69.89% faster than llama.cpp on the same hardware. The post also reports results for RTX 3090 (144.19 t/s), RTX 4070 Laptop (51.57 t/s), and RTX 4090 eGPU (104.95 t/s).

generation:

80.0 tokens/s

quant:

Q4_K_M (GGUF)

User asks how a MacBook Pro M5 Pro 64GB would compare to their work setup of an RTX 4090 with 128 GB DDR5 for running local LLMs. They report running Qwen3.8-27B at Q4_K_M under llama.cpp on the RTX 4090 at roughly 80 t/s with a ~127k context window, though they note they are on holiday and cannot verify the figures. The question is whether to buy the 64GB or 32GB MacBook variant for local LLM use at home.