Local model performance on your hardware

generation:

14.4 tokens/s

quant:

Q4_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Nemotron-3-Super 120B at ~14.4 t/s on a DGX Spark GB10 with 128GB unified memory. Setup is llama.cpp built natively for sm_121 (commit 463b6a963, CUDA 13.0, driver 580.126.09) with the ggml-org Q4_K GGUF (66GB). Ollama's Q4_K_M build reached ~14.2 t/s but its MoE GGUF blobs are incompatible with upstream llama.cpp (blk.1.ffn_down_exps.weight shape mismatch, expected 4096 got 1024). The ggml-org Q4_K file saves about 20GB versus Ollama's 86GB. User notes an OOM pitfall on load that requires dropping the page cache first.

generation:

19.9 tokens/s

prompt processing (prefill):

768.8 tokens/s

quant:

Q4_K (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Nemotron-3-Super-120B-A12B at 19.94 t/s generation and 768.84 t/s prompt processing on an NVIDIA DGX Spark. Setup is llama.cpp with Q4_K GGUF, 65.10 GiB model size, flash attention enabled, 99 GPU layers, n_ubatch 2048. Batched benchmark shows aggregate throughput up to 56.16 t/s generation and 771.69 t/s prompt at 32 concurrent requests. A second model, Nemotron-3-Nano-4B Q8_0, was also benchmarked at 52.85 t/s generation and 2761.90 t/s prompt processing.

generation:

6.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Gemma 4 31B at about 6.0 t/s on a DGX Spark without MTP, and 11.4 to 19.3 t/s with MTP speculative decoding enabled (--spec-draft-n-max 4). Setup is llama.cpp with the am17an/Gemma4-31B-it-GGUF build. With MTP the draft accept rate was 0.5881 over 9 requests. The user notes no speed-up for the MoE variant, over 2x average speedup on the dense model, and that AIME-26 results (~87%) were replicated.

generation:

7-11 tokens/s

kv:

turbo4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6 27B at 7-11 tok/s with stock llama.cpp on an NVIDIA DGX Spark (GB10), consistent across every scenario tested. Setup is stock llama.cpp on one GB10 with no speculative decoding and a q8_0 KV cache, the whole model in unified memory; the post does not state the target quant for the stock run. With DFlash speculative decoding (spiritbuun fork, Q4_K_M target, Q8_0 draft, turbo4 KV) the author gives per-task ranges only: 38-40 tok/s HTML/JS, 24-25 Python, 23-25 short chat, 20-22 medium context and 27-29 sustained over 2048 tokens. Q8_0 and BF16 targets are slower. vLLM with MTP reached around 30 tok/s. The 35B-A3B MoE runs 60-66 tok/s stock and up to 113 tok/s on HTML/JS with DFlash.

generation:

69.0 tokens/s

quant:

Q6_K / Q5_K / Q3_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Ornith-1.5-35B-A3B APEX i-quality at 69.0 t/s on a DGX Spark (GB10). Setup is llama.cpp with Q6_K/Q5_K/Q3_K banded GGUF at 131072 context, single slot, no speculation. A second no-drafter run measured 69.9 t/s. With MTP draft-mtp at n_max 1 the model reached 73.52 t/s (0.674 acceptance); deeper draft depths were slower. i-compact measured 73.9 t/s and i-mini 80.8 t/s via llama-cli. The agentic coding suite passed 9 of 9 tasks and 154 of 154 tests; vision passed 5 of 5 with the image and 0 of 5 blind.

generation:

69.9 tokens/s

prompt processing (prefill):

2657.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks the Gemma 4 family on an NVIDIA DGX Spark (GB10 Grace Blackwell SoC) with 128 GB unified memory, served via llama.cpp in a Docker container. Setup is llama.cpp build 5d3a4a7 with GGUF weights, flash attention enabled, batch size 2048, and a 256K context window. The 26B-A4B MoE model uses Q4_K_M and reaches 69.9 t/s generation and 2657 t/s prefill at 512 tokens. The 31B dense model reaches 11.0 t/s generation and 743 t/s prefill. At 32K context depth, the 26B drops to 52.1 t/s generation and 1877 t/s prefill. Batched concurrency on the 26B reaches 537 t/s aggregate generation at 32 concurrent requests and 3015 t/s aggregate prefill at batch 16.

generation:

19.3-19.7 tokens/s

prompt processing (prefill):

200-370 tokens/s

quant:

UD-IQ2_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports DeepSeek-V4-Flash-0731 at 19.3 to 19.7 tokens/second single-stream on one NVIDIA DGX Spark (GB10, 128GB unified memory). Setup is llama.cpp build 10235 with Unsloth UD-IQ2_M dynamic 2-bit weights, 131072 tokens of context per slot, flash attention on, and no speculative decoding. Prefill runs 200 to 370 tok/s depending on request shape; four concurrent slots give 52 tok/s aggregate.

generation:

5.2 tokens/s

prompt processing (prefill):

120.0 tokens/s

quant:

UD-Q2_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Nemotron-3-Ultra-550B-A55B at ~5.2 tok/s decode and ~120 tok/s prefill across 2x DGX Spark (GB10, 128GB unified each) linked over 200GbE ConnectX-7 RoCE. Setup is llama.cpp upstream master with UD-Q2_K_XL GGUF (~188 GiB, 6 shards), layer-split via RPC with --tensor-split 1,1, --no-mmap and -fit off, ~95 GiB per node. User notes decode is RPC-round-trip-bound and slower per-token than a single node; the model only fits across two boxes. A 600-token reasoning sample took 117s.

generation:

24.5 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.8-Flash-Next at 24.5 tok/s decode on code without MTP on a single DGX Spark (GB10, 121 GB unified). Setup is llama.cpp (danielhanchen qwen4exp/mtp branch) with UD-Q4_K_XL GGUF weights (103.7 GiB), 131,072 context, flash attention on and CUDA graphs active, with about 95 GB used serving. With the Q8_0 shared MTP draft at --spec-draft-n-max 4, decode reaches 47.1 tok/s on code, 36.9 on thinking and 24.2 on prose. A 77,527-token voxel-pagoda run finished in 51 min at 25.3 tok/s end to end.

generation:

15.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports approximately 15 tokens/s generation speed running Qwen3.8-27B on a DGX Spark (GB10) with llama.cpp. Setup is llama.cpp with UD-Q4_K_XL GGUF quant and q8_0 KV cache, using MTP speculative decoding with draft-mtp and spec-draft-n-max 2, batch 4096, ubatch 2048, flash attention on, 99 GPU layers, 8 threads. User notes that NVIDIA forum users reported similar performance with vLLM, but an official SGLang post reported 38 tokens/s decode throughput on DGX Spark, and asks whether a configuration or optimization is missing or whether the model/platform is not yet fully optimized in llama.cpp.

generation:

35.7 tokens/s

quant:

Q5_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User compares generation throughput on a 128 GB DGX Spark at three starting-context depths for two deployment paths. Official INT4 on a vLLM fork reaches 38.3 t/s short, 7.9 t/s at ~45K and 4.6 t/s at ~90K. Q5_K_M on llama.cpp reaches 35.7 t/s short, 33.6 t/s at ~45K and 33.2 t/s at ~90K. The config is 262,144 tokens of max context with ~103 GB total memory used, and streaming separates TTFT from decode. These are the creator's measurements, not an independent rerun, and two variables change at once (runtime and quantization), so no pure engine effect is isolated. The later 131,072-context serving default should not be attached to this table.

generation:

24.0 tokens/s

quant:

UD-IQ3_XXS

kv:

bf16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports DeepSeek-V4-Flash-0731 UD-IQ3_XXS at roughly 23-27 t/s on a DGX Spark GB10, stabilizing at about 24 t/s. Setup is llama.cpp with DSpark speculative decoding. The user notes it is faster than Qwen3.6-27B-NVFP4 for codegen.

generation:

36-38 tokens/s

prompt processing (prefill):

390.0 tokens/s

quant:

K-Quant-Dynamic (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports retrieval at 832K tokens with 3/3 success. DFlash speculative decoding gives roughly 3x speedup. RPC split is slower than single-node.