Local model performance on your hardware

generation:

18.8 tokens/s

quant:

UD-IQ4_XS (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5-122B-A10B at UD-IQ4_XS decoding at 18.75 tok/s on 2x RTX 4090 with expert offload to a Threadripper TRX50 with 128 GB DDR5 (2 of 4 memory channels populated). Setup is llama.cpp on Windows with q8_0 KV cache; the 122B took 11.91 s per agent tool call against 3.40 s for a 27B, so the author shelved it. The rest of the post covers the 27B as a coding subagent: at 64k per agent, Q8_K_XL ran 3 agents (112 s median wait), Q6_K_XL 5 (168 s) and Q4_K_M 9 (330 s), with aggregate prefill near 1,500 tok/s whatever the slot count. Qwen3.6-35B-A3B decoded at 78 tok/s and Qwen3.8-Flash-Next topped out at 23.7 tok/s with experts in system RAM.

generation:

80.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User asks how a MacBook Pro M5 Pro 64GB would compare to their work setup of an RTX 4090 with 128 GB DDR5 for running local LLMs. They report running Qwen3.8-27B at Q4_K_M under llama.cpp on the RTX 4090 at roughly 80 t/s with a ~127k context window, though they note they are on holiday and cannot verify the figures. The question is whether to buy the 64GB or 32GB MacBook variant for local LLM use at home.

generation:

158.9 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen2.5-Coder-7B-Instruct Q4_K_M on an RTX 4090, comparing llama.cpp at 158.90 tok/s decode against apr 0.64.0 at 103.26 tok/s. Setup is llama.cpp build 39173bcac and apr 0.64.0 CUDA with the same GGUF, profile medium, 15s warmup, 3 x 30s runs, 10s cooldown, concurrency 1, streaming. Decode is 0.650x of llama.cpp, with ITL p50 6.29 ms vs 9.68 ms and throughput 1.24 vs 0.77 req/s. The user notes apr's figure is 2.6x the ~40 tok/s 7B reference and that an earlier 11.8 tok/s report was an end-to-end rate including ~9.7s model load.

generation:

165.9 tokens/s

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Distill at 165.9 t/s decode on an RTX 4090 24GB. Setup is llama.cpp with llama-swap, q8_0 KV cache, flash attention on, 256K context, and speculative draft-MTP enabled. The post is a registry of many models on the same card; the highest measured decode listed is 363 t/s for Nemotron-Lightning-3.5, and context cliffs are noted for several models.

generation:

115.0 tokens/s

quant:

IQ4_XS (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 115 t/s decode on a single RTX 4090 24 GB. Setup is llama.cpp b11115 with llama-swap v257, IQ4_XS GGUF quant, q8_0 KV cache, 196608 context, and draft-MTP speculative decoding. The user notes the DeepSWE result is a single task, not an average, and that the model passed 40/43 hidden tests and kept 109/109 existing tests.

generation:

80.0 tokens/s

quant:

Q4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a llama.cpp config tuner that measured a 27B Q4 model on an RTX 4090, improving average generation from 60 to 80 tok/s and peak from 80 to 110 tok/s. Setup is llama.cpp with Q4 weights and a 100K context length; the tuner sweeps speculative decoding, KV cache quant, GPU offload layers, and batch/ubatch, running 3 passes and taking the median. At 100K context the tuned config reached 60 tok/s versus 40 hand-tuned. The user says speculative decoding was the biggest single win and asks whether it helps as much on a 4090 as on a 5090.

generation:

129.0 tokens/s

prompt processing (prefill):

5600.0 tokens/s

quant:

Q5_K_M (GGUF)

kv:

turbo3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 26B-A4B at ~129 tok/s single-request generation on a single RTX 4090, with ~5,600 tok/s prefill. Setup is a TurboQuant fork of llama.cpp with Q5_K_M weights, turbo3 3-bit KV cache, Flash Attention, and a multimodal projector, using 22.3 GB of 24 GB VRAM at 262,144 tokens of context. Concurrent generation is reported at ~83-85 tok/s. A 15-test agentic coding benchmark through Claude Code passed 15/15 with an average quality of 4.2/5.

generation:

23.0 tokens/s

prompt processing (prefill):

2248.6 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 23.03 t/s generation and 2248.6 t/s prompt processing on a single RTX 4090 with 32 GB RAM. Setup is llama.cpp via llama-swap with the unsloth UD Q4_K_XL GGUF, a Q8 KV cache, 131072 context, and a BF16 mmproj for multimodal input; cache reuse is disabled because of the multimodal model. The user ran a three-day coding session in VS Code Copilot and hit a failure where the model emitted never-ending slashes; they ask what could cause it.

generation:

3.0 tokens/s

prompt processing (prefill):

30.0 tokens/s

quant:

IQ4_XS

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash 0731 at about 30 t/s prompt processing and about 3 t/s generation on CPU with an RTX 4090 and a Tesla P40. Setup is the IQ4_XS quant at roughly 127 GB with MTP enabled, and layer-by-layer assignment because tensor splitting is not supported. The user mentions an Unsloth 4bit K_XL quant at roughly 144 GB but could not run MTP with it due to memory constraints.

generation:

40.0 tokens/s

prompt processing (prefill):

1650.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports 194,048 context tokens at ~1650 prompt t/s and ~40 t/s generation without MTP, and 131,584 context tokens at ~1650 prompt t/s and ~65 t/s generation with MTP, up to 80 t/s for code, on an RTX 4090 FE with 48 GB DDR4 in WSL2. Setup uses a q8_0 KV cache and q4_0 draft KV cache, with --cache-ram 32768, --ctx-checkpoints 32, --no-mmap and --mlock.

generation:

134.0 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B on an RTX 4090, with throughput ranging from 44 t/s to 134 t/s and no RAM spilling. Setup includes an mmproj file for vision.

generation:

90.0 tokens/s

quant:

IQ4_XS (GGUF)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 90 t/s with MTP enabled and 45-50 t/s without MTP at full context. Setup uses a Q8 KV cache. User sees no difference versus FP16 and prefers speed over context.

generation:

10.9 tokens/s

prompt processing (prefill):

132.5 tokens/s

quant:

UD-Q2_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash at 10.9 t/s generation and 132.5 t/s prompt on an RTX 4090 with 128 GB of DDR5 5600 MT/s. Setup is llama.cpp llama-server with unsloth's UD-Q2_K_XL quant, 64000 context, flash attention off and 8 threads pinned to the P-cores of an Intel Core Ultra 7 270k, on a 24GB GPU plus system RAM. Pinning the P-cores took generation from 6.8 tok/s to 11 tok/s, and the IQ4_NL quant ran at 50.7 t/s prompt and 8.1 t/s generation with only about 10k context. The user finds it smarter than Qwen 3.6 27B Q4_K_XL but still prefers Qwen 3.6 27B for agentic work because it runs so much faster.

generation:

80.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

Q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports MTP draft acceptance of about 73%. Setup is a TBQ4_0 KV cache with an MTP draft of 3, using a fork of llama.cpp.