Local model performance on your hardware

generation:

87.7 tokens/s

quant:

MXFP4 (safetensors)

kv:

FP8

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek-V4.1-Flash at 87.7 tok/s single-stream decode on four NVIDIA DGX Sparks (TP=4). Setup is SGLang with native MXFP4 experts and FP8 dense weights, DSpark speculative decoding (k=5), 1M context configured, 8M-token KV pool pinned, ~77 GiB resident weights per rank. Aggregate decode reaches 342.7 tok/s at 16 streams (22.2 per stream); cold prefill is 4,059-5,925 tok/s at 4k-262k context, 4.9k-5.0k tok/s on real text at 16k-128k. A 1,011,084-token needle test passed. The 3-Spark TP=3 profile is also documented at 37.9 tok/s single-stream.

generation:

41.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

fp8_e4m3

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.8-27B in NVFP4 on a single NVIDIA DGX Spark (GB10, 128 GB unified memory), reporting a greedy median of 41.0 tok/s with SGLang + DSpark speculative decoding. Setup is SGLang with NVFP4 weights, fp8_e4m3 KV cache, 262144 max context, and DSpark block-drafter speculation; vLLM + MTP is compared on the same harness. Per-workload SGLang figures range from 37.4-41.8 tok/s on code, 40.1-44.5 on reasoning, 40.3-52.9 on math, and 18.0-19.6 on free prose. vLLM single-stream reaches 20.0 tok/s on random 512/512 and 18.0 on ShareGPT; both engines converge to roughly 60 tok/s aggregate at concurrency 4. SGLang serves 4x the context (262K vs 65K) and has lower TTFT and TPOT.

quant:

NVFP4 (NVFP4)

kv:

fp8_e4m3

User benchmarks Qwen3.8-27B dense at 38.6 tok/s code generation on a single NVIDIA DGX Spark (GB10, 128 GB unified memory). Setup is SGLang with NVFP4 weights, fp8_e4m3 KV cache, 262144 context and DFlash2 speculative decoding with 8 draft tokens; the same machine also ran vLLM NVFP4 + MTP k3 at 24.5 tok/s and Ollama Q4_K_M at 10.3 tok/s. User reports speculative decoding degrades with context (accept rate 0.36-0.45 short vs 0.10-0.16 long reasoning, decode 27-33 vs 13-16 tok/s), that FP8 is 3x slower than Q4 on dense models (7.8 vs 23 tok/s), and that chat speed is only 15.9 tok/s versus 38.6 on code.

generation:

34.0 tokens/s

quant:

NVFP4 (NVFP4)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-27B at about 34 tok/s real-world on a single DGX Spark (ASUS Ascent GX10, GB10, 128GB unified memory). Setup is SGLang with an NVFP4 W4A4 checkpoint and DSpark block-speculative decoding, batch-1 single-stream. User also measured 38.0 tok/s average on eval-style workloads and 46.7 tok/s peak on GSM8K-style prompts. On the same machine, llama.cpp UD-Q4_K_XL with MTP reached about 27 tok/s real-world and 24-30 tok/s on eval-style workloads, while vLLM 0.27 NVFP4 with MTP reached about 24.5 tok/s.

generation:

35.0 tokens/s

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8 Flash-Next at about 35 t/s on a single DGX Spark. Setup is SGLang with an NVFP4 quant at 262,144 context, used through VSCode Copilot in autopilot mode. The session produced about 10k lines of code and consumed about 800k tokens over 8 hours of planning, coding, and testing.

generation:

51.1 tokens/s

quant:

NVFP4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 51.06 t/s single-stream and 184.03 t/s aggregate at c6 on 2x DGX Spark in TP2. Setup is SGLang with QSA, NEXTN and FlashInfer GDN. Disabling NEXTN gives 26.4 t/s at c1 and 111.8 t/s at c6. Medium thinking scored 22/24 on a 24-task LiveBench-style pilot.