Local model performance on your hardware

generation:

66.0 tokens/s

prompt processing (prefill):

2827.0 tokens/s

quant:

2.75 bpw EXL3 (EXL3)

User reports DeepSeek-V4.1-Flash at 66 tok/s single-stream on prose with thinking off, on two DGX Sparks. Setup is vLLM with tensor parallelism across the two Sparks at 2.75 bpw EXL3, with Engram tables held in system RAM on a separate EPYC/3090 box and served over RDMA, and a DSpark speculative drafter on an NVLinked pair of 3090s. KV cache pool is about 1.93M tokens, plus a prefix cache on NVMe. Prefill runs 2,827 tok/s at 4K down to 2,618 tok/s at 256K, where the first token takes 100 s. Four-stream totals reach 322 tok/s on structured JSON and 192 tok/s on code; decode after a 128K prompt is 62 tok/s. A warm first token comes back in 2.58 s against 58.6 s cold. Sparks ran at 2300 MHz, medians of 3 reps.

prompt processing (prefill):

3556.7 tokens/s

quant:

Q4_K_M (GGUF)

User benchmarks a llama.cpp CUDA PR that fuses MoE gate/up matmuls and GLU in MMQ, reporting prompt-processing throughput on DGX Spark (GB10) and RTX 5090. On DGX Spark with Qwen3.6-35B-A3B Q4_K_M, prompt processing rises from 3235.8 to 3556.7 t/s (+9.9%); the Qwen3-30B-A3B Q4_K_M row goes from 2911.4 to 3410.8 t/s (+17.2%). On RTX 5090 the same Qwen3.6-35B-A3B Q4_K_M row goes from 15000.3 to 15852.0 t/s (+5.7%), with UD-Q4_K_M at 15030.2 to 15947.8 t/s (+6.1%). No generation speeds are given.

generation:

102.9 tokens/s

quant:

MXFP4 (MXFP4)

User benchmarks an unnamed MoE model on an NVIDIA DGX Spark, reaching 102.88 tok/s decode with MTP speculative decoding (NEXTN, steps=3, topk=1, draft_tokens=4) versus a 36.09 tok/s baseline. The setup stacks three optimizations: a Humming MXFP4 MoE backend, online FP8 LM head quantization, and MTP speculation. Each step is measured: Humming MoE alone gives 39.45 tok/s (+9.3%), adding the FP8 LM head gives 43.94 tok/s (+21.7%), and MTP brings it to 102.88 tok/s (+185.1%).

generation:

62.2 tokens/s

User reports DeepSeek V4.1-Flash decoding at a median 62.2 t/s on four DGX Spark (GX10) units in the original IKEA Kallax setup, with prefill of 4,255 t/s at 8K and 4,268 t/s at 250K. Decode uses speculative decoding, which swings about ±10 t/s between runs, so decode was left out of the cooling comparison. Each run is a single 8-minute sweep from 1K to 250K tokens with the model already loaded. Across six cooling setups the hottest GPU peaked at 82°C with no throttling; clocks stepped down about 3% from roughly 2525 MHz at 53°C to 2450 MHz at 82°C, and 250K prefill stayed between 4,225 and 4,353 t/s. The user's V3 open-faced air duct enclosure was the coolest setup at 65.1°C average under load.

generation:

106.0 tokens/s

quant:

EXL3 3.5bpw (EXL3)

User reports DeepSeek V4.1 Flash at 106 t/s per stream at concurrency 1 on four DGX Sparks, averaging 93 t/s aggregate across eight categories. Setup is TensorFold's Zig engine with an EXL3 3.5bpw pack, 1M context, a 2M-position KV pool, DSpark speculative drafts on and tensor parallel across all four GB10 nodes over RoCE v2. The sweep runs to 16 streams: per-stream decode falls to 20 t/s while aggregate rises to 250 t/s at C16. Coding reaches 139 t/s per stream at C1 and 343 t/s aggregate at C16; prose 81 and 224. Cold prefill was skipped, previously measured at 1.9-2.3K tok/s.

prompt processing (prefill):

899.9 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

User benchmarks a llama.cpp PR that fuses FFN gate/up matmuls and GLU in MMQ, on a DGX Spark (GB10) and an RTX 5090. Setup is llama.cpp with llama-bench, -ngl 999, flash attention on, 16384-token prompts, F16 KV cache, batch 2048, 5 repetitions. On the DGX Spark, Qwen3.8-27B Q4_K_M prompt processing rises from 840.1 to 899.9 tok/s (+7.1%). On the RTX 5090 the same model and quant goes from 3688.4 to 3790.9 tok/s. The table also covers Qwen3.6-27B and Llama-3.1-8B across Q4_K_M, Q5_K_M, Q6_K, Q8_0 and IQ4_XS; only prompt processing was measured (-n 0).

generation:

58.1 tokens/s

User reports gpt-oss-120b at 58.1 tok/s on two DGX Sparks (128 GB each) with tensor parallel over RPC in their Ollama fork, up from 42.7 tok/s on one node and 33.8 tok/s with a layer split. Setup is Ollama driving llama.cpp RPC with two servers, each holding half of every layer and doing the all-reduce between themselves, over a direct 200 Gb/s RDMA cable. Measured with ollama cluster bench at 32k context, tg128, warm, mean of 3. Qwen3.8 27B with an MTP draft reached 34.7 tok/s (24.4 one node, 21.4 layer split) and DeepSeek-V4-Flash MXFP4 at 156 GB reached 21.4 tok/s (14.5 layer split, does not fit on one node). Prompt processing stays about 10% below one node for models that fit, and load time is a few seconds longer than the layer split. vLLM with TP=2 gave 55.7 tok/s for gpt-oss-120b on the same pair in eager mode, single request. Limits are exactly two machines, one tensor model at a time.

generation:

45.8 tokens/s

prompt processing (prefill):

1717.0 tokens/s

quant:

NVFP4 (NVFP4)

kv:

FP8

User benchmarks Qwen3.8-Flash-Next served four ways on one DGX Spark (GB10, 121 GB unified memory), reporting 45.8 t/s decode and 1,717 t/s prefill for their own fast recipe. Setup is vLLM with NVIDIA's NVFP4 FP4-Local checkpoint, FP8 KV cache and MTP 3, at a 262,144-token window with thinking off and temperature 0. The other recipes reach 54.3 t/s (hibrid48) and 51.6 t/s (TensorFold) decode; quality mode drops to 44.4 t/s and 1,362 t/s prefill. Under concurrent load hibrid48 delivers 82 t/s with 16 requests of 8k against 51 for the user's recipe. TensorFold returns HTTP 400 on response_format and scores 6/12 on structured output versus 10/12 for the vLLM recipes.

generation:

42.2 tokens/s

prompt processing (prefill):

2158.0 tokens/s

quant:

NVFP4 (NVFP4)

User reports Qwen3.8-27B at 42.2 tok/s single-user on one DGX Spark, with prefill at 2,158 tok/s. Setup is vLLM with unsloth's NVFP4 weights on the GB10's FP4 tensor cores, plus DFlash2 speculative decoding with 10 drafts from an NVFP4 drafter and a small vLLM plugin. At 16k context it reaches 33.4 tok/s and at 100k 29.5 tok/s; the board's 16k two-user test gives 60.6 tok/s total, first among single-Spark entries. Quality checks against BF16 recover 99.3% on gsm8k, 101.1% on ifeval, 99.4% on humaneval and 101.2% on NarrativeQA.

generation:

56.8 tokens/s

quant:

EXL3 2.9bpw (EXL3)

User reports DeepSeek-V4.1-Flash-1017 at 56.8 tok/s decode on text and 92.2 tok/s on code with TensorFold on two DGX Spark, TP=2 over the CX7 cable. Setup is TensorFold with EXL3 2.9bpw weights and 1M context, using a speculative two-node decoder; the same pair with the vLLM kit reached 27.0 tok/s text and 39.9 tok/s code. Four concurrent code requests total 140 tok/s, and a 64K-token needle test took 35.9 s versus 80.4 s on the kit. Long-prompt prefill is 30-37% below the published numbers (1,707 vs 2,421 tok/s at 32K), and the first long prompt after a start costs about 6 s extra.

generation:

86.0 tokens/s

prompt processing (prefill):

2892.0 tokens/s

User reports Qwen3.8-Flash-Next at about 86 tok/s decode on 2x DGX Spark, with prefill improved to 2892 tok/s at pp2048 c1. Setup is a 2x Spark tensor-parallel recipe (v2.2.0) with a 16K cold TTFT of 5.49 s; the 1x release (v2.4.0) decodes about 60 tok/s and prefills 1916 tok/s. Prefill gains come from reading GDN prefill inputs as views and running one projection as a single BF16 GEMM; decode is unchanged and both releases passed the same quality gate.

generation:

60.8 tokens/s

prompt processing (prefill):

2574.0 tokens/s

User reports DeepSeek V4.1 Flash at 60.8 t/s decode and 2,574 t/s prefill on a 1K prompt using a tuned SGLang build on 4x DGX Spark. Setup is SGLang TP4 with knapcio's recipe, DSpark speculative decoding, Engram tables streamed from NVMe, 262K context, and 100 GbE switch. The model weights are 511 GB and do not fit comfortably on 4 Sparks; memory allocation above 0.92 per Spark crashes. User compares against Patrick Moorhead's results: 62.5 t/s vs ~58 t/s single-request prose, 79.1 t/s vs ~96 t/s single-request code, and 293 t/s vs 494 t/s peak code at 16 vs 32 concurrent requests. User could not reproduce 1M context, 32 slots, or ~0.2 s first token.

generation:

52.9 tokens/s

prompt processing (prefill):

1689.5 tokens/s

quant:

MXFP4 (MXFP4)

User reports 52.87 t/s generation and 1689.47 t/s prompt processing. Source evidence (verbatim): Source: https://jetsonhacks.com/wp-content/uploads/2025/10/spark-llamacpp-bench.html # llama.cpp on NVIDIA DGX Spark - Benchmarks Benchmarks generated with llama.cpp tools on an NVIDIA DGX Spark (NVIDIA GB10, compute capability 12.1, VMM: yes). llama.cpp build b6767 (commit 5acd45546), improved decode via PR 16585. Settings: llama-bench with -fa 1, -p 2048, -n 32, -ub 2048, at context depths of 0, 4096, 8192, 16384 and 32768. All values are tokens per second. Model gpt-oss-120b MXFP4 MoE, 59.02 GiB, 116.83B params. At depth 0: pp2048 1689.47 t/s (+/- 107.67), tg32 52.87 t/s (+/- 1.70). At depth 4096: pp2048 1733.41, tg32 51.02. At depth 8192: pp2048 1705.93, tg32 48.46. At depth 16384: pp2048 1514.78, tg32 44.78. At depth 32768: pp2048 1221.23, tg32 38.76. Single stream llama-bench, no speculative decoding.

generation:

15.3 tokens/s

quant:

UD-IQ3_XXS (GGUF)

User reports Qwen3.5-397B at 15.3 t/s across two DGX Sparks (128 GB unified each) over llama.cpp RPC, a model that does not fit on one node. Setup is a forked Ollama 0.40.3 with llama.cpp b11351, UD-IQ3_XXS quant (141 GB), 256 output tokens, warm run, two nodes linked by a 200 Gb/s cable. Splitting across nodes costs speed: fast small models lose 30-60% (gemma4:e2b 106.6 to 44.1 t/s), large models lose 15-20%, and load time is 3-5x longer. The default policy keeps a model on one machine and only splits when it does not fit.

generation:

59.4 tokens/s

prompt processing (prefill):

3633.8 tokens/s

quant:

Q4_K_M (GGUF)

User reports 59.42 t/s generation and 3633.84 t/s prompt processing. Source evidence (verbatim): Source: https://blog.majid.info/llama-bench/dgx1.txt # NVIDIA DGX Spark 128GB / GB10: original Gemma4 E4B Q4_K_M llama-bench log First-hand author log linked from https://blog.majid.info/llama-bench/ (April 13, 2026 article). Measured hardware: NVIDIA DGX Spark 128GB / GB10. Gemma4 E4B Q4_K_M GGUF, llama.cpp ff5ef8278 (8763), backend CUDA. pp512 3633.84 ±201.59 tokens/s; tg128 59.42 ±0.96 tokens/s. One GB10, ngl99; 122502 MiB runtime device VRAM visibility. 128GB is unified system capacity, not dedicated VRAM or resident use. Command names --hf-repo unsloth/gemma-4-E4B-it-GGUF; exact tested quantization repository https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF ; tested revision unknown. Log reports 4.62 GiB model size and 7.52B counted parameters; size is not resident memory. Context depth, KV quantization, measured power, driver version and explicit speculation/concurrency declarations unreported. Raw source: https://blog.majid.info/llama-bench/dgx1.txt .

generation:

86.0 tokens/s

quant:

NVFP4 (NVFP4)

User reports Qwen3.8-Flash-Next at 86 tok/s single-stream on 2x DGX Spark with TP=2, and 60 tok/s on a single DGX Spark. Setup is vLLM with NVFP4 experts and MXFP8 dense layers, 262K context, MTP with 4 probabilistic drafts, and a retrained drafter. Aggregate throughput at 8 concurrent requests is 195 tok/s on 2x and 137 tok/s on 1x; with thinking off on 36 coding prompts it is 113 tok/s single-stream and 317 tok/s aggregate on 2x. User's own requant beats the publisher's vLLM image (51.8 tok/s c1 defaults, 96.6 tok/s c8) by 1.67x to 2.02x. IFEval strict 87.4% vs 87.1% shows no quality loss. Two independent 1x replicas (DP=2) reach 241 tok/s at c16.

generation:

118.6 tokens/s

quant:

NVFP4 / MXFP8 mixed (safetensors)

kv:

FP8

mtp (multi-token prediction):

on

User reports Qwen3.8 at 118.6 t/s on two NVIDIA DGX Spark (GB10 Grace Blackwell) machines at TP=2. Setup is vLLM with NVFP4 experts and MXFP8 dense/attention paths, FP8 KV cache, probabilistic MTP with 4 draft tokens, and a configured max_model_len of 262144. The figure is the mean of 3 boot-level means of per-pass median request decode rates over 36 coding prompts run one at a time, with a between-boot sample SD of 0.353 t/s; the workload is favorable to MTP and is not a general conversational speed estimate.

generation:

78.6 tokens/s

quant:

NVFP4 / MXFP8 mixed (safetensors)

kv:

FP8

mtp (multi-token prediction):

on

User reports Qwen3.8 at 78.6 t/s on one NVIDIA DGX Spark (GB10 Grace Blackwell, 128 GB unified RAM, TP=1). Setup is vLLM with NVFP4 experts and MXFP8 dense/attention paths, FP8 KV cache, probabilistic MTP with 4 draft tokens, and a configured max_model_len of 262144. The figure is the mean of 6 boot-level means of per-pass median request decode rates, one request at a time, with a between-boot sample SD of 0.487 t/s; it is not pooled tokens/time or an end-to-end completion rate.

generation:

14.4 tokens/s

quant:

Q4_K (GGUF)

User reports Nemotron-3-Super 120B at ~14.4 t/s on a DGX Spark GB10 with 128GB unified memory. Setup is llama.cpp built natively for sm_121 (commit 463b6a963, CUDA 13.0, driver 580.126.09) with the ggml-org Q4_K GGUF (66GB). Ollama's Q4_K_M build reached ~14.2 t/s but its MoE GGUF blobs are incompatible with upstream llama.cpp (blk.1.ffn_down_exps.weight shape mismatch, expected 4096 got 1024). The ggml-org Q4_K file saves about 20GB versus Ollama's 86GB. User notes an OOM pitfall on load that requires dropping the page cache first.

generation:

19.9 tokens/s

prompt processing (prefill):

768.8 tokens/s

quant:

Q4_K (GGUF)

flash attention:

on

User benchmarks Nemotron-3-Super-120B-A12B at 19.94 t/s generation and 768.84 t/s prompt processing on an NVIDIA DGX Spark. Setup is llama.cpp with Q4_K GGUF, 65.10 GiB model size, flash attention enabled, 99 GPU layers, n_ubatch 2048. Batched benchmark shows aggregate throughput up to 56.16 t/s generation and 771.69 t/s prompt at 32 concurrent requests. A second model, Nemotron-3-Nano-4B Q8_0, was also benchmarked at 52.85 t/s generation and 2761.90 t/s prompt processing.