Local model performance on your hardware

generation:

57.6 tokens/s

prompt processing (prefill):

6574.0 tokens/s

quant:

GPTQ-INT4 (GPTQ)

kv:

int8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 57.6 t/s decode and 6574 t/s prefill on six RTX 3090s. Setup is vLLM with a GPTQ-INT4 checkpoint and int8 KV cache, 250k context, PLE table on SSD, pipeline parallelism across six cards. Decode falls to 29.4 t/s per user with seven concurrent sessions (206 t/s aggregate); llama.cpp-multigpu holds 30 t/s per user at five sessions. NVFP4 checkpoints are 10-12% slower than GPTQ on these cards. Load to healthy is 4-5 minutes.

generation:

100.0 tokens/s

quant:

W8A16 (AWQ)

kv:

FP16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B Swift 1.5 at almost 100 tok/s decode on 2x RTX 3090. Setup is vLLM with an 8-bit AWQ + GPTQ W8A16 quant, 128k context and FP16 KV cache, MTP=5, around 21.6 GB used per card. The quant was made on 4 RTX 3090; the user says it is less of an overthinker than the original Qwen3.8 27B.

generation:

19.3 tokens/s

prompt processing (prefill):

690-790 tokens/s

quant:

EXL3 3.0 bpw (EXL3)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash at 19.3 tok/s decode (C1) on a single RTX 3090 24GB with 503 GiB DDR4 and NVMe. Setup is vLLM 0.13 with EXL3 3.0 bpw quantization, fp8 KV cache, 262,144-token context, and CPU expert offload. All 15,360 routed experts are pinned in host RAM (190.5 GiB), with an AVX2 CPU tier computing cold decode misses and an elastic VRAM expert cache holding the hottest experts. Decode scales to 24.0 tok/s at 2 streams and 26.6 tok/s at 4 streams. Prefill ranges from 690 to 790 tok/s across 8k to 261k tokens. An experimental Intel Arc B70 expert tier reached 41.3 tok/s at 4 streams (+55%), but quality A/B is pending. Prefill fidelity failed the inherited guard (top-1 0.9856 vs 0.988 required, mean KL 0.0068 vs 0.00103 allowed).

generation:

82.7 tokens/s

quant:

QAT (W4A16)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Gemma 4 12B at 82.7 tok/s decode on vLLM and 85.0 tok/s on Ollama on one RTX 3090 24 GB capped at 300 W. Setup is vLLM 0.30.0 with Google's W4A16 QAT checkpoint and Ollama 0.34.4 with gemma4:12b-it-qat, temperature 0, 256 tokens out, one request at a time. At sixteen concurrent requests vLLM reached 506.8 tok/s against Ollama's 142.3 tok/s; first token was 213.0 ms on vLLM against 441.9 ms on Ollama. Board energy at sixteen requests was 590 joules per 1,000 tokens on vLLM and 1,384 on Ollama. On 108 questions the two agreed on 63 strictly and 98 leniently.

generation:

84.2-88.8 tokens/s

prompt processing (prefill):

3401-3413 tokens/s

quant:

W4A16 (safetensors)

kv:

BF16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next on two RTX 3090s with 128 GB RAM, reaching 3,401–3,413 tok/s prefill and 84.2–88.8 tok/s decode on a 131,099-token prompt. Setup is vLLM with W4A16 AutoRound weights, BF16 KV cache, FP8 PLE table, MTP3 speculative decoding, and CPU expert offload. The 64 GB RAM profile runs both cards with 64 GB system RAM, keeping 88 experts per layer on each GPU and streaming the rest from RAM; prefill uses 8,192-token chunks. A single RTX 3090 with 64 GB RAM reaches 2,106 tok/s prefill and 47.4 tok/s decode at 131K context.

generation:

127.0 tokens/s

prompt processing (prefill):

1440.0 tokens/s

kv:

KVarN 4/2-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 127 tok/s single-stream decode on one RTX 3090 at 250 W. Setup is vLLM 0.30.0 with 38 custom patches, int8 tensor-core GEMMs, speculative decoding with MTP or DFlash2 drafting at 7 drafts per pass, and KVarN 4/2-bit KV cache at 150k-262k context. Decode reaches 381 tok/s at 25k context when the answer quotes the prompt. At 64 concurrent requests the aggregate is about 1,035 tok/s decode and 948 end to end. Prefill is about 1,440 tok/s, or 1,850-1,940 tok/s depending on int8 activation settings. The card is shared with training jobs.

generation:

114.8 tokens/s

prompt processing (prefill):

1785.0 tokens/s

quant:

INT8 W8A16 (compressed-tensors)

kv:

fp8_e4m3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 113.7 to 114.8 t/s decode and ~1,785 t/s prefill on 2x RTX 3090. Setup is vLLM v0.29.0 with INT8 W8A16 compressed-tensors weights, fp8_e4m3 KV cache, 262,144 context, tensor parallel 2, DFlash2 W4A16 drafter with 7 speculative tokens, and a 64 GiB CPU offload tier. User notes NVLink gives +12% decode and +18% prefill over PCIe P2P off, aggregate decode is 302.5 t/s at 4 concurrent, and a W8A8 build prefilled at 2,377 t/s at 46K.

generation:

153.5-186.7 tokens/s

prompt processing (prefill):

5123-5410 tokens/s

quant:

W4A16 (AutoRound)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next (125B MoE, ~6B activated) served on four RTX 3090s at full 262,144-token context, with an 806,792-token FP8 KV pool. Setup is vLLM 0.30.0 (v2.2.0 release, 58 commits over the public tag) with a W4A16 AutoRound checkpoint, TP2 x PP2 plus expert parallel, MTP K=3, and FP8 KV cache with a scale sidecar. Cold prefill runs 5,123-5,410 t/s from 10K to 200K tokens; decode with thinking on is 153.5-186.7 t/s (median of three prompts per depth, one boot). The release is a reliability update at parity with v2, not faster: decode lands at 0.973-1.040 of v2 in every cell. On a rented 4x RTX 3090 without peer-to-peer, a cold 54,713-token prompt took 13.8 s (about 4,000 t/s).

generation:

193.3 tokens/s

quant:

W4A16 (AutoRound)

kv:

fp8_e5m2

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 193.3 tok/s median on 4x RTX 3090 (96 GB VRAM, 31 GB host RAM). Setup is vLLM with Intel AutoRound W4A16 experts (Marlin), BF16 attention, FP8 n-gram table, fp8_e5m2 KV cache, 262,144-token context, 6 seats, and albucino's INT4 MTP draft with 3 tokens and expert parallel. Without the draft the same lane measured 55.8 tok/s; real prompts ran 109.5 tok/s prose, 108.4 chat, 145.8 code. Six parallel coding agents sustained 317 tok/s aggregate (570 peak burst, 53 tok/s per agent, TTFT 0.47 s); at gmu 0.95 six agents gave 252.6 tok/s aggregate and 48 tok/s per stream. A llama.cpp lane on the same box measured 96 to 102 tok/s on copy/edit tasks and 40 to 49 tok/s freeform, with ~311 tok/s prefill and two 262,144-token slots.

generation:

127.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at 127 tok/s single-stream on a single RTX 3090 at 250 W. Setup is vLLM with a 150k token context and an OpenAI-compatible API, in single-user mode. Aggregate decode is ~1,035 tok/s at 64 concurrent in batch mode, and 381 tok/s when the answer quotes its own prompt.

generation:

116.6 tokens/s

quant:

W4A16 (W4A16)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next 180B W4A16 at 116.6 t/s decode on four RTX 3090s capped at 220 W each. Setup is vLLM built from a pinned PR head with five patches, MTP speculative decoding with k=4, 65,536 context, and 76,046 KV tokens. Decode holds 115.1 t/s at 32k and 72.75 t/s at 128k; prefill reads 3,187 t/s at 32k and 2,835 t/s at 128k. The c8 aggregate is 180.7-230.3 t/s on the bf16 profile. Peak short-prompt decode reaches 198.5 t/s on code. Spread is plus or minus 15% run to run.

generation:

138.0 tokens/s

quant:

GPTQ-int4 (GPTQ)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at ~138 t/s single-request on an RTX 3090, up from ~124 t/s, using a custom vLLM 0.27.1 stack with DFlash2 speculative drafting. Setup is vLLM with GPTQ-int4 weights, fp8 KV cache, 64k context, and a W4A16 DFlash2 drafter; the 3090 is power limited to 250W. The same stack reaches 942 t/s at 64 concurrent requests. User notes DFlash2 is best for 1-4 concurrent users and MTP wins at 8+ concurrent; quality is unchanged with perplexity 8.09 and GSM8K 96.5%.

generation:

70.0 tokens/s

quant:

INT4

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at over 70 t/s single-stream decode on 2x RTX 3090, with close to 200 t/s decode at 3 to 4 concurrency and over 10k t/s prefill. Setup is stock vLLM v0.29 on Linux serving the RedHatAI INT4 quant with an FP8 KV cache, MTP speculative decoding with 3 tokens, tensor parallel size 2, and the full 262k context. The user notes the setup is standard and that the RedHat quant is overlooked for Ampere hardware.

prompt processing (prefill):

250.0 tokens/s

quant:

Q4

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B running a local agent loop for about 21 days on a single RTX 3090, with prefill around 250 t/s. Setup is vLLM serving the Q4 model with a Q8 KV cache at 200k context, using a deepseek harness and a written rulebook; llama.cpp reaches about 700 prefill t/s on the same card. The run used 180 subagents, about 230M tokens in and out, 1.7B cache-read tokens, and 699 compactions totaling about 83 hours, roughly 17% of calendar time, with a typical compact taking about 7 minutes on a 160k+ token prompt.

generation:

80.0 tokens/s

prompt processing (prefill):

3701.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at ~80 t/s on 3x RTX 3090 at 1M context, dropping to ~60 t/s once QSA reaches its 2048-token budget. Setup is vLLM with most of the KV cache offloaded to system RAM, no KV cache quantization. Prefill at 248k reaches 3,701 t/s, and throughput is ~150 t/s at 4 concurrent requests. Decode speed stays flat as total context grows because only 12 of 48 layers hold a KV cache and QSA bounds the per-step read to 48 MiB per token.

generation:

40.0 tokens/s

prompt processing (prefill):

1104.0 tokens/s

quant:

IQ4_XS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at ~40 t/s decode on dual RTX 3090 (sm_86) with the IQ4_XS GGUF, stable from 1k to 126k context. Setup is a vLLM 0.29.0 GGUF-capable fork with patches and env toggles; 32k fresh prefill measured at ~1104 t/s. Q4_K_XL is kept experimental: ~32.3 t/s decode at 1k, ~854 t/s prefill at 8k, and unstable at long context. An older ~6665 t/s prefill figure was found non-reproducible.

generation:

100.0 tokens/s

quant:

Q4

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at an average 100 t/s on an RTX 3090, compared to 20 t/s on a MacBook M5 Pro 48GB. Setup is vLLM with a Q4 model on Linux, served over an OpenAI endpoint. The Mac run used llama.cpp with MTP and an Unsloth Q6 quant; Q4 on Mac made little speed difference. User notes tool calls fail more often but the model self-corrects, and considers the setup good enough to replace a Claude subscription.

generation:

38.4 tokens/s

prompt processing (prefill):

871.9 tokens/s

quant:

INT4 AutoRound (AutoRound)

kv:

fp8_e4m3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8 27B at 38.39 t/s generation and 871.93 t/s prefill on an RTX 3090 24GB, averaged over 3 hours of constant running. Setup is vLLM 0.27.1 with the Club 3090 patch, INT4 AutoRound weights, FP8 E4M3 KV cache, 147,456 token context, single sequence, and AOT compilation on WSL2. At 10K prompt the run reached 42.3 t/s generation and 1000.26 t/s prefill; at 90K prompt it reached 34 t/s generation and 743.59 t/s prefill. The user notes llama.cpp with Q5 UD GGUF at 81,920 context plus MTP gave 25-30 t/s, and that 154K context was the practical ceiling.

generation:

60.0 tokens/s

quant:

FP4

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports a 285B MoE at 60+ t/s on 10 RTX 3090s and 120+ t/s on 12 RTX 3090s. Setup is FP4 experts with FP8 attention and 157GB of weights, using TP2xPP5 on 10 GPUs at a 240W cap and TP4xPP3 on 12 GPUs. The 60+ t/s figure is on 10 GPUs. Context runs to 1M without offload and 4M with RAM offload, with long-context prefill around 3500 t/s when batched. Vision, speculative decoding and tool calls work. A Docker image and repo are provided.

quant:

W4A16

User compares a W4A16 quant of Nemotron 3.5 Lightning 30B-A3B against an IQ4_XS GGUF on an RTX 3090. Setup is vLLM for W4A16 and llama.cpp for IQ4_XS. The two are near-parity in instruction following benchmarks, with about 4.5x throughput by B16. The user describes the model as fast and reliable, and suitable for batch labelling and agentic responses.