NVIDIA DGX Spark reports

generation:

40.0 tokens/s

prompt processing (prefill):

1500.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports GLM-5.3 Flash at ~40 t/s decode and ~1500 t/s prefill on 2x DGX Sparks. Setup is vLLM with NVFP4 weights and DFlash2 speculative decoding, tensor parallel across 2 cards, 260k context via Claude Code scaffold; decode figure quoted at 100k context. User says the model feels between GLM 5.1 and 5.3 for coding, has good vision and 3D understanding, and calls it a "good enough" home setup.

generation:

20.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Kimi K3 at 20 t/s on 16 DGX Sparks at 300k context, up from an unusable 7 t/s at 100k context. Setup is vLLM across two joined 8-node DGX Spark clusters. The user also ran Qwen 397B at 30 t/s on 4x GB10 and 50-60 t/s on 16x3090, Qwen 235B at 10-12 t/s and DeepSeek 671B at 8 t/s on a Threadripper with 512GB DDR4 and experts offloaded to RAM.

generation:

72-87 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Nemotron 3.5 Lightning 30B-A3B at 72 to 87 tok/s single-stream decode and about 2,600 tok/s prefill on a DGX Spark (GB10, 128GB unified memory). Setup is Ollama 0.32.9 with the Q4_K_M GGUF at the default 262,144 token context, 26GB resident at 100% GPU, with the built-in MTP speculative decoding active. Decode depends on the workload: 72 tok/s on prose and 84 to 87 tok/s on JSON and summaries. The same model served with vLLM using the NVFP4 checkpoint and DSpark draft model reached 108 tok/s decode and about 5,400 tok/s prefill. On one agent prompt the model answered in 485 tokens and 5.9s against 1,953 tokens and 26.0s for qwen3.5:35b-a3b.

generation:

78.3 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at 78.3 tok/s generation on an ASUS Ascent GX10 with NVIDIA GB10 and 128 GB unified memory. Setup is vLLM with NVFP4 weights at 131,072-token context, one model at a time on the machine. The run scored 86/86 hidden tests in 21:01 wall clock, the fastest perfect run in the collection, at 57.9 end-to-end tokens per second. The same model at FP8 scored 68/86 at 50.0 tok/s, a difference the user attributes to one all-or-nothing task rather than precision.

generation:

53.3 tokens/s

quant:

MXFP4 (MXFP4)

kv:

fp8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports MiMo-V2.6-Flash-RL at 53.31 tok/s per stream at C1 on two DGX Sparks (GB10, 121.7 GiB unified memory each) with vLLM tensor parallel 2 and DFlash speculative decoding (7 draft tokens). Setup is vLLM with MXFP4 experts, fp8 KV cache, 300K max context, marlin MoE backend, GPU memory utilization 0.90, max-num-seqs 8, KV pool 1,835,052 tokens. Aggregate throughput is 155.77 tok/s at six streams; cold prefill ranges from 1,946.6 tok/s at 2K to 656.4 tok/s at 248K; TTFT 0.367 s at C1. User notes tool-call storms in agent use and that prose/narrative decode is slow due to low DFlash acceptance.

generation:

69.3 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8 e4m3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-5.3-Flash at 69.3 tok/s single-stream coding peak on four NVIDIA DGX Spark (GB10) nodes at tensor-parallel 4. Setup is vLLM with NVFP4 weights and FP8 e4m3 KV cache, DFlash2 block-diffusion speculative decoding with a 5-layer Qwen3 drafter proposing 7 tokens per step; ~45 GiB of weights per node. Mixed-prompt sustained decode is 36.7 tok/s at c1 (42.2 tok/s at temp=0), 56.1 tok/s aggregate at c2 and 84.7 tok/s at c4; prefill holds ~1,700 tok/s from 4K to 128K context. A v3 config with pinned KV cache and chunked prefill regressed 25% to 31.5 tok/s.

generation:

30.1 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks GLM-5.3-Flash on two GB10 nodes (DGX Spark-class, 121 GiB unified memory each, tensor-parallel 2 over RoCE), comparing NVFP4 and EXL3 quantizations. Single-stream decode at 8k context is 30.1 tok/s for NVFP4 and 28.8 tok/s for EXL3. Setup uses vLLM with DFlash2 speculative decoding (k=7) and 8k context. NVFP4 collapses under concurrency (7.1 tok/s per stream at 4 streams) while EXL3 holds steady (26.7 tok/s). User reports NVFP4 prefills ~1.8x faster cold (1667.7 vs 925.9 tok/s at 8k) but has zero prefix cache hits, while EXL3 achieves 6.9x faster TTFT on reuse. Agent sessions are warm and concurrent, favoring EXL3. Later updates show k=3 recovers +87% for NVFP4 at C4, and TensorFold TP=2 pairs reach 113.2 tok/s structured single-stream.

generation:

69.9 tokens/s

prompt processing (prefill):

2657.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks the Gemma 4 family on an NVIDIA DGX Spark (GB10 Grace Blackwell SoC) with 128 GB unified memory, served via llama.cpp in a Docker container. Setup is llama.cpp build 5d3a4a7 with GGUF weights, flash attention enabled, batch size 2048, and a 256K context window. The 26B-A4B MoE model uses Q4_K_M and reaches 69.9 t/s generation and 2657 t/s prefill at 512 tokens. The 31B dense model reaches 11.0 t/s generation and 743 t/s prefill. At 32K context depth, the 26B drops to 52.1 t/s generation and 1877 t/s prefill. Batched concurrency on the 26B reaches 537 t/s aggregate generation at 32 concurrent requests and 3015 t/s aggregate prefill at batch 16.

generation:

43.9 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 43.9 tok/s median single-stream on one DGX Spark (GB10, 128 GB unified memory). Setup is vLLM nightly 8a728663 with the official NVIDIA NVFP4 checkpoint, FP8 KV cache, MTP3 speculative decoding, 6 sequences, 4096-token prefill chunk, 262,144-token context, gpu-memory-utilization 0.80; the 47.68 GiB FP8 n-gram table stays on NVMe and is read on demand, giving a KV pool of 1,027,392 tokens. Under load the same boot gives 33.4 tok/s per stream at x2 (43.9 aggregate), 26.4 at x4 (47.1), 21.7 at x6 (68.8). Cold prefill is 1,269 tok/s at 7K, 1,757 at 28K and 1,760 at 113K. The user also compares TP2 and TP4 lanes, where TP2 SPEED reaches 53.7 tok/s median single-stream and TP4 CONTEXT 40.5.

generation:

41.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

fp8_e4m3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B in NVFP4 on a single NVIDIA DGX Spark (GB10, 128 GB unified memory), reporting a greedy median of 41.0 tok/s with SGLang + DSpark speculative decoding. Setup is SGLang with NVFP4 weights, fp8_e4m3 KV cache, 262144 max context, and DSpark block-drafter speculation; vLLM + MTP is compared on the same harness. Per-workload SGLang figures range from 37.4-41.8 tok/s on code, 40.1-44.5 on reasoning, 40.3-52.9 on math, and 18.0-19.6 on free prose. vLLM single-stream reaches 20.0 tok/s on random 512/512 and 18.0 on ShareGPT; both engines converge to roughly 60 tok/s aggregate at concurrency 4. SGLang serves 4x the context (262K vs 65K) and has lower TTFT and TPOT.

quant:

NVFP4 (NVFP4)

kv:

fp8_e4m3

User benchmarks Qwen3.8-27B dense at 38.6 tok/s code generation on a single NVIDIA DGX Spark (GB10, 128 GB unified memory). Setup is SGLang with NVFP4 weights, fp8_e4m3 KV cache, 262144 context and DFlash2 speculative decoding with 8 draft tokens; the same machine also ran vLLM NVFP4 + MTP k3 at 24.5 tok/s and Ollama Q4_K_M at 10.3 tok/s. User reports speculative decoding degrades with context (accept rate 0.36-0.45 short vs 0.10-0.16 long reasoning, decode 27-33 vs 13-16 tok/s), that FP8 is 3x slower than Q4 on dense models (7.8 vs 23 tok/s), and that chat speed is only 15.9 tok/s versus 38.6 on code.

generation:

27.8 tokens/s

prompt processing (prefill):

1500-1730 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User compares vLLM and TensorFold running Qwen3.8-Flash-Next on a single DGX Spark (GB10, 128 GB unified memory). vLLM with an NVFP4 checkpoint, FP8 KV cache, and MTP with 3 draft tokens reaches 27.8 tok/s for a single request, while TensorFold with a 4-bit MLX checkpoint and int8 KV cache reaches 52.2 tok/s. At 5 concurrent requests, vLLM gives 78.5 tok/s and TensorFold 110.2 tok/s. Prompt processing is 1,500-1,730 tok/s for vLLM and 2,090-2,260 tok/s for TensorFold.

generation:

19.3-19.7 tokens/s

prompt processing (prefill):

200-370 tokens/s

quant:

UD-IQ2_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4-Flash-0731 at 19.3 to 19.7 tokens/second single-stream on one NVIDIA DGX Spark (GB10, 128GB unified memory). Setup is llama.cpp build 10235 with Unsloth UD-IQ2_M dynamic 2-bit weights, 131072 tokens of context per slot, flash attention on, and no speculative decoding. Prefill runs 200 to 370 tok/s depending on request shape; four concurrent slots give 52 tok/s aggregate.

generation:

2.7 tokens/s

prompt processing (prefill):

337.0 tokens/s

quant:

FP4 (FP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash at 2.68 tok/s median decode (2.64-2.71 tok/s range) on a single NVIDIA DGX Spark (GB10) with 128 GB unified memory. Setup uses native FP4 experts, a 73.8 GB arena holding 25.6% of routed experts with the rest streamed from NVMe, DSpark speculative decoding, and MAX_SEQ 32768. Prefill is 337 tok/s on a 5,014-token prompt (14.9 s). Without DSpark decode is 1.75 tok/s. Expert hit rate is about 0.83 and about 0.9 GB is read from NVMe per token.

generation:

72.4 tokens/s

kv:

fp8_ds_mla

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek V4 Flash at 72.4 tok/s on two NVIDIA DGX Spark nodes. Setup is vLLM with fp8_ds_mla KV cache, tensor parallel across both nodes over 200 GbE, prefix cache disabled, DSpark speculative drafting. Also measured Qwen3.8-Flash-Next NVFP4 on SGLang with NEXTN speculation at 42.6 tok/s and GLM-5.3-Flash on SGLang with DFlash2 speculation at 41.1 tok/s.

generation:

34.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at about 34 tok/s real-world on a single DGX Spark (ASUS Ascent GX10, GB10, 128GB unified memory). Setup is SGLang with an NVFP4 W4A4 checkpoint and DSpark block-speculative decoding, batch-1 single-stream. User also measured 38.0 tok/s average on eval-style workloads and 46.7 tok/s peak on GSM8K-style prompts. On the same machine, llama.cpp UD-Q4_K_XL with MTP reached about 27 tok/s real-world and 24-30 tok/s on eval-style workloads, while vLLM 0.27 NVFP4 with MTP reached about 24.5 tok/s.

generation:

60-70 tokens/s

quant:

W4A16 AutoRound (AutoRound)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash at 60-70 (peak 74.1) tok/s peak single-stream decode on one DGX Spark (GB10), with 60 to 70 tok/s on normal requests and 212.2 tok/s aggregate across 8 streams. Setup is vLLM with W4A16 AutoRound for the MoE experts, FP8 side layers and INT8 lm_head, full 262K context, using the model's own MTP draft head with a draft vocab cut from 248K to 65K ids. Cold prefill reaches 4,016 tok/s at 16K, 2,426 tok/s at 64K and 2,213 tok/s at 128K, versus 1,171 / 1,071 / 1,065 tok/s for the original recipe. Cached coding prompts start replying in about 0.57 s. Quality scored 93.1% and 93.3% on a fixed 492-question suite over two seeds. The model takes about 71 GiB with a 16 GB KV pool; CJK drafting is slower due to the English/code draft vocab.

generation:

5.2 tokens/s

prompt processing (prefill):

120.0 tokens/s

quant:

UD-Q2_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Nemotron-3-Ultra-550B-A55B at ~5.2 tok/s decode and ~120 tok/s prefill across 2x DGX Spark (GB10, 128GB unified each) linked over 200GbE ConnectX-7 RoCE. Setup is llama.cpp upstream master with UD-Q2_K_XL GGUF (~188 GiB, 6 shards), layer-split via RPC with --tensor-split 1,1, --no-mmap and -fit off, ~95 GiB per node. User notes decode is RPC-round-trip-bound and slower per-token than a single node; the model only fits across two boxes. A 600-token reasoning sample took 117s.

generation:

34.0 tokens/s

prompt processing (prefill):

2700-4000 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at ~34 tok/s single-stream decode on a single NVIDIA DGX Spark (GB10, 128 GB unified pool). Setup is vLLM v0.30.0 with NVIDIA's NVFP4 checkpoint, hybrid fp8 side layers, MTP=2 speculative decoding, prefix caching, deterministic top-k, and YaRN 500k context. Prefill is ~2,700-4,000 tok/s (8k-32k prompts) with a ~505-520k-token KV pool. The repo patches vLLM to serve the 48 GiB n-gram PLE table from NVMe via mmap, fixing prefix caching and non-deterministic top-k on GB10. A separate concurrency sweep on the same hardware reports 266.8 aggregate tok/s at 48 concurrent streams.

generation:

24.5 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 24.5 tok/s decode on code without MTP on a single DGX Spark (GB10, 121 GB unified). Setup is llama.cpp (danielhanchen qwen4exp/mtp branch) with UD-Q4_K_XL GGUF weights (103.7 GiB), 131,072 context, flash attention on and CUDA graphs active, with about 95 GB used serving. With the Q8_0 shared MTP draft at --spec-draft-n-max 4, decode reaches 47.1 tok/s on code, 36.9 on thinking and 24.2 on prose. A 77,527-token voxel-pagoda run finished in 51 min at 25.3 tok/s end to end.