Local model performance on your hardware

generation:

86.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 86 tok/s single-stream on 2x DGX Spark with TP=2, and 60 tok/s on a single DGX Spark. Setup is vLLM with NVFP4 experts and MXFP8 dense layers, 262K context, MTP with 4 probabilistic drafts, and a retrained drafter. Aggregate throughput at 8 concurrent requests is 195 tok/s on 2x and 137 tok/s on 1x; with thinking off on 36 coding prompts it is 113 tok/s single-stream and 317 tok/s aggregate on 2x. User's own requant beats the publisher's vLLM image (51.8 tok/s c1 defaults, 96.6 tok/s c8) by 1.67x to 2.02x. IFEval strict 87.4% vs 87.1% shows no quality loss. Two independent 1x replicas (DP=2) reach 241 tok/s at c16.

generation:

118.6 tokens/s

quant:

NVFP4 / MXFP8 mixed (safetensors)

kv:

FP8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 at 118.6 t/s on two NVIDIA DGX Spark (GB10 Grace Blackwell) machines at TP=2. Setup is vLLM with NVFP4 experts and MXFP8 dense/attention paths, FP8 KV cache, probabilistic MTP with 4 draft tokens, and a configured max_model_len of 262144. The figure is the mean of 3 boot-level means of per-pass median request decode rates over 36 coding prompts run one at a time, with a between-boot sample SD of 0.353 t/s; the workload is favorable to MTP and is not a general conversational speed estimate.

generation:

78.6 tokens/s

quant:

NVFP4 / MXFP8 mixed (safetensors)

kv:

FP8

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 at 78.6 t/s on one NVIDIA DGX Spark (GB10 Grace Blackwell, 128 GB unified RAM, TP=1). Setup is vLLM with NVFP4 experts and MXFP8 dense/attention paths, FP8 KV cache, probabilistic MTP with 4 draft tokens, and a configured max_model_len of 262144. The figure is the mean of 6 boot-level means of per-pass median request decode rates, one request at a time, with a between-boot sample SD of 0.487 t/s; it is not pooled tokens/time or an end-to-end completion rate.

generation:

48.2 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Gemma 4 26B-A4B-it NVFP4 at 48.2 t/s for a 1000-token generation on an NVIDIA DGX Spark (GB10 Blackwell, 128GB unified memory), against 23.3 t/s for the BF16 original. Setup is vLLM with an NVFP4 (W4A4) modelopt checkpoint and FP8 KV cache, single DGX Spark, Marlin MoE backend, 256K max context; the model loads at 15.7 GiB. TTFT is 53 ms versus 97 ms for BF16, and the checkpoint is ~16.5 GB on disk versus ~49 GB. Quality retention averages 97.6% across GSM8K and IFEval, with GSM8K down about 4 percentage points.

generation:

39.8 tokens/s

quant:

FP8 (FP8)

kv:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at 39.8 t/s mean decode on two DGX Sparks, with final code decode at roughly 74 t/s mean. Setup is vLLM with FP8 weights and NVFP4 KV cache, TP=2 across two DGX Sparks, DSpark speculative decoding with 3 proposed tokens, single concurrency. User also tried DeepSeek V4 Flash on the same setup but it failed benchmarks due to non-completion and excessive token use (30-45k tokens on an easy task), unlike cloud versions of the same model. User is happy with Qwen3.8 Flash-Next but concerned about overthinking tendency.

generation:

38.0 tokens/s

prompt processing (prefill):

902.0 tokens/s

quant:

MXFP4 (MXFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash at 37.95 tok/s single-stream decode across 8 categories and 902 tok/s cold prefill at 2K tokens on four DGX Sparks. Setup is vLLM TP4 across four DGX Sparks over ConnectX-7 RoCE, MXFP4 experts and MXFP8 dense weights, DSpark speculative decoding with k=5, the whole model in unified memory (56.6-68.9 GiB per Spark), 300K context window. Per category the single-stream decode was code 73.8 tok/s, JSON 52.1, math 50.9 and prose 24.4. Six streams reached 131.86 tok/s aggregate. Cold prefill was 1,539 tok/s at 32K and 1,194 at 64K.

generation:

13.9 tokens/s

prompt processing (prefill):

539.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash 162B (REAP K144 pruned) at 13.899 tok/s decode and 538.958 tok/s prefill on a single DGX Spark. Setup is vLLM with NVFP4/MXFP4 expert weights and FP8 KV cache, 200K context, 14 GB KV cache, no speculative decoding. Validation used a 186,390-token prompt with 345.834 s TTFT and needle retained; the watchdog logged a low-memory kill at teardown, so the profile is proof-of-concept. K160 prefills at about 514 tok/s and decodes at about 24 tok/s with MTP.

generation:

9.8 tokens/s

prompt processing (prefill):

500.0 tokens/s

quant:

2-bit expert planes

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4-Flash-0731 at ~9.8 tok/s decode on a single DGX Spark (GB10, 128GB unified memory). Setup is a vLLM-based stack (vLLM-Moet adapted for unified memory and sm_121) with 2-bit expert planes and an FP4 tier, serving 256K context with a KV pool of ~660K tokens. Decode uses the built-in DSpark speculative head at k=2; the recommended k=7 was slower at 5.1 tok/s. Prefill is ~500 tok/s chunked at 2048, with a 28K-token prompt giving ~53 s TTFT. First boot took ~35 min for one-time expert quantization, later boots ~10 min via persistent cache. Tool calling and per-request thinking toggle work out of the box.

generation:

80.1 tokens/s

quant:

NVFP4 (NVFP4)

kv:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek-V4-Flash-Vision-Exp at 80.1 tok/s on count-to-100, 51.8 tok/s on code and 33.2 tok/s on prose, on 2x DGX Spark at TP2. Setup is vLLM with NVFP4 weights and nvfp4_ds_mla KV cache, 1M context, DSpark speculative decoding at k=5, KV pool 2.79M tokens / 19.12 GiB at gpu_memory_utilization 0.85. Without the Patch 4 shared-expert fix the same workloads measure 50.7, 47.2 and 30.4 tok/s. The 0731 text checkpoint reaches 78.4 tok/s peak at 98.9% acceptance with Patch 4, versus 66.1 peak and 60.2% acceptance without. The user notes a ~30% cold-start penalty and that realistic mixed agent traffic at concurrency 4 gives ~88.6 tok/s aggregate and ~22.3 tok/s per stream.

generation:

41.0 tokens/s

prompt processing (prefill):

1785.0 tokens/s

quant:

FP8 (FP8)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4-Flash at ~41 tok/s single-stream and ~350 tok/s aggregate at c=32 on 2x DGX Spark (GB10, 128 GB unified) via vLLM TP=2 over QSFP56 RoCE. Setup is vLLM with official FP8 weights, 1M context profile, shared ~1.1M-token KV pool; prefill ~1785 t/s. Long-context sweep: 8k 40 t/s, 100k 40 t/s, 200k 38 t/s, 500k 32 t/s decode; prefill ~1970 down to ~1380 t/s. Cross-machine: RTX PRO 6000 ds4 46.9 t/s, M2 Ultra 29.7 t/s, single Spark ds4 IQ2_XXS ~14 t/s. llama.cpp GGUF suite up to 67 t/s (35B-A3B), 172B MoE 31 t/s.

generation:

30.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-Flash-Next on a DGX Spark (GB10, 121 GB unified memory) against Claude Opus 5.5 across 21 graded coding tasks. Local model matches Opus on 11 everyday tasks (11/11 each) but is roughly 5-8x slower, and scores 7/10 on hard tasks versus Opus's 10/10. Setup is vLLM 0.30 with NVFP4 weights and MTP speculative decoding, about 30 tok/s on long generations. Harnesses tested were a minimal 3-tool loop, Claude Code 2.1.285, and Qwen Code 0.24.7. User notes the failure mode was over-verification rather than inability, and that harness choice moved the hard-tier score as much as settings did. A real-world test built a 150-line spec wedding microsite in 88 minutes, about 5.4k lines across 30 files.

generation:

79.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Ornith-1.5-35B-A3B at 79 tok/s single-stream on one NVIDIA DGX Spark (128GB unified memory). Setup is vLLM 0.24 with the official 4-bit NVFP4 checkpoint (about 22 GiB) and multi-token prediction speculative decoding, serving the full 262K context. At 20 concurrent users each stream gets about 25 tok/s, 490 tok/s total; GPU memory utilization had to be capped at 0.70 and throughput dropped past 20 users. Reasoning scores cited: MATH-500 97%, GSM8K 98%, MMLU-Pro 81%.

generation:

40.0 tokens/s

prompt processing (prefill):

1500.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports GLM-5.3 Flash at ~40 t/s decode and ~1500 t/s prefill on 2x DGX Sparks. Setup is vLLM with NVFP4 weights and DFlash2 speculative decoding, tensor parallel across 2 cards, 260k context via Claude Code scaffold; decode figure quoted at 100k context. User says the model feels between GLM 5.1 and 5.3 for coding, has good vision and 3D understanding, and calls it a "good enough" home setup.

generation:

20.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Kimi K3 at 20 t/s on 16 DGX Sparks at 300k context, up from an unusable 7 t/s at 100k context. Setup is vLLM across two joined 8-node DGX Spark clusters. The user also ran Qwen 397B at 30 t/s on 4x GB10 and 50-60 t/s on 16x3090, Qwen 235B at 10-12 t/s and DeepSeek 671B at 8 t/s on a Threadripper with 512GB DDR4 and experts offloaded to RAM.

generation:

78.3 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 78.3 tok/s generation on an ASUS Ascent GX10 with NVIDIA GB10 and 128 GB unified memory. Setup is vLLM with NVFP4 weights at 131,072-token context, one model at a time on the machine. The run scored 86/86 hidden tests in 21:01 wall clock, the fastest perfect run in the collection, at 57.9 end-to-end tokens per second. The same model at FP8 scored 68/86 at 50.0 tok/s, a difference the user attributes to one all-or-nothing task rather than precision.

generation:

53.3 tokens/s

quant:

MXFP4 (MXFP4)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports MiMo-V2.6-Flash-RL at 53.31 tok/s per stream at C1 on two DGX Sparks (GB10, 121.7 GiB unified memory each) with vLLM tensor parallel 2 and DFlash speculative decoding (7 draft tokens). Setup is vLLM with MXFP4 experts, fp8 KV cache, 300K max context, marlin MoE backend, GPU memory utilization 0.90, max-num-seqs 8, KV pool 1,835,052 tokens. Aggregate throughput is 155.77 tok/s at six streams; cold prefill ranges from 1,946.6 tok/s at 2K to 656.4 tok/s at 248K; TTFT 0.367 s at C1. User notes tool-call storms in agent use and that prose/narrative decode is slow due to low DFlash acceptance.

generation:

69.3 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8 e4m3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash at 69.3 tok/s single-stream coding peak on four NVIDIA DGX Spark (GB10) nodes at tensor-parallel 4. Setup is vLLM with NVFP4 weights and FP8 e4m3 KV cache, DFlash2 block-diffusion speculative decoding with a 5-layer Qwen3 drafter proposing 7 tokens per step; ~45 GiB of weights per node. Mixed-prompt sustained decode is 36.7 tok/s at c1 (42.2 tok/s at temp=0), 56.1 tok/s aggregate at c2 and 84.7 tok/s at c4; prefill holds ~1,700 tok/s from 4K to 128K context. A v3 config with pinned KV cache and chunked prefill regressed 25% to 31.5 tok/s.

generation:

30.1 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks GLM-5.3-Flash on two GB10 nodes (DGX Spark-class, 121 GiB unified memory each, tensor-parallel 2 over RoCE), comparing NVFP4 and EXL3 quantizations. Single-stream decode at 8k context is 30.1 tok/s for NVFP4 and 28.8 tok/s for EXL3. Setup uses vLLM with DFlash2 speculative decoding (k=7) and 8k context. NVFP4 collapses under concurrency (7.1 tok/s per stream at 4 streams) while EXL3 holds steady (26.7 tok/s). User reports NVFP4 prefills ~1.8x faster cold (1667.7 vs 925.9 tok/s at 8k) but has zero prefix cache hits, while EXL3 achieves 6.9x faster TTFT on reuse. Agent sessions are warm and concurrent, favoring EXL3. Later updates show k=3 recovers +87% for NVFP4 at C4, and TensorFold TP=2 pairs reach 113.2 tok/s structured single-stream.

generation:

43.9 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 43.9 tok/s median single-stream on one DGX Spark (GB10, 128 GB unified memory). Setup is vLLM nightly 8a728663 with the official NVIDIA NVFP4 checkpoint, FP8 KV cache, MTP3 speculative decoding, 6 sequences, 4096-token prefill chunk, 262,144-token context, gpu-memory-utilization 0.80; the 47.68 GiB FP8 n-gram table stays on NVMe and is read on demand, giving a KV pool of 1,027,392 tokens. Under load the same boot gives 33.4 tok/s per stream at x2 (43.9 aggregate), 26.4 at x4 (47.1), 21.7 at x6 (68.8). Cold prefill is 1,269 tok/s at 7K, 1,757 at 28K and 1,760 at 113K. The user also compares TP2 and TP4 lanes, where TP2 SPEED reaches 53.7 tok/s median single-stream and TP4 CONTEXT 40.5.

generation:

27.8 tokens/s

prompt processing (prefill):

1500-1730 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User compares vLLM and TensorFold running Qwen3.8-Flash-Next on a single DGX Spark (GB10, 128 GB unified memory). vLLM with an NVFP4 checkpoint, FP8 KV cache, and MTP with 3 draft tokens reaches 27.8 tok/s for a single request, while TensorFold with a 4-bit MLX checkpoint and int8 KV cache reaches 52.2 tok/s. At 5 concurrent requests, vLLM gives 78.5 tok/s and TensorFold 110.2 tok/s. Prompt processing is 1,500-1,730 tok/s for vLLM and 2,090-2,260 tok/s for TensorFold.