Local model performance on your hardware

generation:

86.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3-8B on an RTX PRO 6000 Blackwell 96GB, reporting 86 tok/s single-stream decode in BF16 under vLLM. Setup is vLLM 0.27.1 with BF16 weights, greedy decoding, concurrency 32 for aggregate figures; SGLang 0.5.9 and llama.cpp CUDA builds were also tested. Aggregate throughput at concurrency 32 was 1,725 tok/s for vLLM, 1,327 for SGLang and 428 for llama.cpp; single-stream decode sat in an 83 to 96 tok/s band across all stacks. An FP8 checkpoint on vLLM reached 130 tok/s single-stream and 2,597 tok/s aggregate, with p50 latency down from 0.74 s to 0.49 s.

generation:

113.0 tokens/s

prompt processing (prefill):

2800.0 tokens/s

quant:

MXFP4 (MXFP4)

kv:

BF16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports MiniMax-M3 at ~113 tok/s single-stream decode on 4x RTX PRO 6000 Blackwell (96 GB each). Setup is vLLM with the MXFP4 checkpoint (~256 GB, ~60 GB per card), BF16 KV cache, tensor parallelism 4, and 250K context; cold prefill runs ~2,800 tok/s flat from 8K to 131K. Four concurrent requests reach ~184 tok/s aggregate. A second NVFP4/SGLang configuration on the same host decodes at ~87 tok/s single-stream with a ~643K-token KV pool and image/video support.

generation:

90.6 tokens/s

quant:

NVFP4 (NVFP4)

kv:

BF16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.1 at 90.6 t/s single-stream decode on 8x RTX PRO 6000 Blackwell (96 GB each, 768 GB total). Setup is vLLM with an NVFP4 checkpoint (lukealonso/GLM-5.1-NVFP4-MTP), BF16 KV cache, DCP1, MTP speculative decoding on, 371456 KV cache tokens, max model length 202752. Aggregate decode reaches 1165.1 t/s at 64 concurrent requests; cold prefill is 4333 t/s at 8k context and 3170 t/s at 128k. DCP2 and DCP4 trade decode speed for larger KV cache, and DCP8 MTP-off was unstable at long context.

generation:

101.6 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-Flash-Next (125B-A6B) on a single RTX PRO 6000 Blackwell 96GB with 64GB host RAM, comparing vLLM, ExLlamaV3 and SGLang. vLLM with NVFP4/FP8 mixed checkpoint, INT4 PLE offload and MTP reaches 101.6 tok/s at 1k context and 98.9 tok/s at 32k, with prefill around 28k tok/s. ExLlamaV3 EXL3 is faster at short single-user chat (168.9 tok/s at 1k) but fails most requests at 32k concurrency. SGLang with a file-backed n-gram table decodes at 37-51 tok/s. The user notes the 27B incumbent stays in production due to larger KV capacity and reliable tool_choice enforcement.

generation:

112.5 tokens/s

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Kimi-K2.6 at 112.5 tok/s single-stream decode on 8x RTX PRO 6000 Blackwell. Setup is vLLM with fp8 KV cache, TRITON_MLA attention, TP=8, DCP=8, and Eagle3 MTP speculative decoding with 3 tokens using the lightseekorg/kimi-k2.5-eagle3-mla draft model. The 112.5 tok/s is the DCP=1 + MTP=3, 0-context, concurrency-1 cell from an aggregate decode matrix; the page also gives DCP=1 no-MTP (91.5), DCP=4 MTP (101.5), DCP=8 MTP (89.4) and DCP=8 no-MTP (75.5) at the same cell, plus prefill of 7892 tok/s at 8k and 5604 tok/s at ~83k. An erratum notes the old 128k rows were actually ~83k prompt tokens.

generation:

179.5 tokens/s

quant:

EXL3-K3 (EXL3)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.3 Flash at 179.50 tok/s single-stream decode on two RTX PRO 6000 Blackwell 96GB GPUs. Setup is vLLM with EXL3-K3 quant and FP8 KV cache, TP2 + EP2 + DCP2, DFlash2 speculative decoding, 1,048,576-token context ceiling. Aggregate decode reaches 1,045.47 tok/s at concurrency 16; prefill is 4,107-4,258 tok/s across 8K-128K prompts. Exact one-million-token six-needle retrieval passed 6/6 in 284.859 seconds.

generation:

106.1 tokens/s

prompt processing (prefill):

13036.0 tokens/s

quant:

FP4+FP8

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4-Flash at 106.1 tok/s single-request decode on 2x RTX Pro 6000 Blackwell (192GB VRAM). Setup is vLLM with FP4+FP8 quantization and FP8 KV cache, no prompt caching and no speculative decoding, 1,024 output tokens per request. Single-request prefill at 8K context is 13,036 tok/s; decode falls to 96.3 tok/s at 1024K context. System throughput with 8 concurrent requests is 367.8 tok/s at 1K context, 170.0 tok/s at 32K and 56.5 tok/s at 128K.

generation:

246.0 tokens/s

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports MiMo-V2.6-Flash-RL at 246 tok/s decode at C1 (90.0 decode steps/s) on two RTX PRO 6000 Blackwell GPUs. Setup is vLLM with FP8 KV cache, TP2 at 0.985 utilization, DFlash drafter with 7 tokens, video off, 1.31M token KV cache. Decode is 42.3 steps/s at C8 and 30.6 at C16; 246 tok/s at C1 on mixed real tasks. Prefill is 9.7K / 9.1K tok/s at 8K / 32K. First start autotunes b12x for about 15 minutes.

generation:

51.0 tokens/s

prompt processing (prefill):

15359.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Gemma 4 31B IT NVFP4 Turbo at 51 tok/s single-stream decode and 15359 tok/s prefill on an RTX PRO 6000. Setup is vLLM with NVFP4 weights, FP8 KV cache, and 16384 tokens of context, using 18.5 GiB of GPU memory. Batched decode reaches 1244 tok/s and concurrency 6.22 req/s at 1K input / 200 output tokens; the same benchmark on an RTX 5090 performed identically because at 16k context performance is not limited by GPU memory.

generation:

60-88 tokens/s

quant:

FP8 (FP8)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports running Qwen 3.8 27B FP8 on a single RTX PRO 6000 Blackwell 96GB with vLLM, shared by a small group of developers for coding agents. Setup uses vLLM with prefix caching and speculative decoding. In its first week the node processed 2B tokens, with two users alone accounting for roughly 950M and 900M tokens. Per-user throughput while sharing the node is 60-88 tok/s. The user reports 0 preemptions, 0 queued requests and 38.6% peak KV cache usage, and notes the node is not yet saturated.

User describes an open-source project (jevm) that replaces the lm_head of Qwen3.8-27B with ordinal cumulative-link heads on several layers, trained with LoRA, to produce five calibrated answers in one prefill for Chinese A-share news screening. No model results are reported: only a smoke test on Qwen3-0.6B has been run, and the 27B is planned for training on 8× RTX PRO 6000. The repo contains label validation and baselines, including a text-blind GBM at 0.55–0.57 AUC on price reactions and 0.68–0.70 on volume/propagation. A plain LoRA plus single-letter-answer baseline is served with vLLM logprobs for comparison.

generation:

57.0 tokens/s

quant:

BF16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Muse Glimmer 30B at ~57 t/s with speculative decoding using DFlash, against ~25 t/s without speculation, a 2.3x speedup. Setup is vLLM with six patches applied to the image. Mean acceptance length is ~2.5 tokens per verification step, with overall draft acceptance ~10%. Per-position acceptance is ~73% at position 0, ~40% at 1, ~15% at 2, and near zero past 5. The recipe's 3.1x figure was greedy decoding with K-quant on llama.cpp under different conditions. The predicted ceiling is ~65 t/s at 2.6 mean acceptance.

quant:

BF16

rating:

4/5

User reports Qwen 3.6 27B abliterated BF16 running locally via vLLM with llama-swap and MTP speculative decoding, scoring 8/10 on a Terminal-Bench 2.0 pilot. The same setup beats DeepSeek-V4 IQ2 at 7/10 and comes close to an FP8 API at 9/10. It is the only configuration to pass the cancel-async-tasks hard task, and it missed build-cython-ext and sqlite-db-truncate on timeout.

generation:

114.8 tokens/s

quant:

FP8

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares Qwen3.8-27B-FP8 against Qwen3.6-27B-FP8 on an RTX PRO 6000 Blackwell with vLLM. In an MTP sweep, Qwen3.8 peaks at MTP 6 with 114.8 tok/s and Qwen3.6 peaks at MTP 7 with 131.6 tok/s. Qwen3.8 is 5-20% slower across MTP steps, though quality is comparable or better at some steps. A pre-sweep manual run at MTP 5 showed Qwen3.8 faster, at 108.9 vs 103.6 tok/s.

generation:

56.0 tokens/s

prompt processing (prefill):

3000.0 tokens/s

quant:

8-bit

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at about 56 t/s decode and about 3K t/s prefill on a single RTX PRO 6000 96GB GPU. Setup is vLLM with an 8-bit quant at about 262K context, serving three concurrent users. The figures are for three concurrent users, and the user offers free community access rather than presenting a benchmark.

generation:

110.0 tokens/s

quant:

NVFP4 (safetensors)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at roughly 110 t/s, ranging from 76 to 125 t/s, on a single RTX Pro 6000 Blackwell with 96 GB, at 165,000 to 170,000 tokens of context. Setup is vLLM with the NVFP4 quant and INT4 quantized PLE n-grams, MTP speculative decoding, prefix caching, and PLE CPU offload. The user praises quality for coding and agentic tasks.

User reports DeepSeek V4 Flash on 2x RTX PRO 6000 with vLLM reaches Sonnet quality and faster wall-clock time than Sonnet or Opus via API. User mentions Qwen 3.6 models as anchors.

User benchmarks Qwen3.6 27B BF16 and Qwen3.6 35B BF16. For the 35B, best generation throughput is 3500 t/s at 128 concurrency with MTP off, and prompt throughput is 30000 t/s. The 27B was also tested with MTP on and off.

User reports FastDMS, a DMS KV-cache compression implementation, decoding 1.5-2x faster than vLLM BF16/FP8 with 5-8x less KV memory. Benchmarks cover Llama-3.2-1B and Qwen3-8B DMS checkpoints, with KLD and token match comparable or better than vLLM's FP8/TurboQuant. Training took about 20 min on an RTX Pro 6000 Blackwell.