Local model performance on your hardware

generation:

14.4 tokens/s

quant:

Q4_K (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Nemotron-3-Super 120B at ~14.4 t/s on a DGX Spark GB10 with 128GB unified memory. Setup is llama.cpp built natively for sm_121 (commit 463b6a963, CUDA 13.0, driver 580.126.09) with the ggml-org Q4_K GGUF (66GB). Ollama's Q4_K_M build reached ~14.2 t/s but its MoE GGUF blobs are incompatible with upstream llama.cpp (blk.1.ffn_down_exps.weight shape mismatch, expected 4096 got 1024). The ggml-org Q4_K file saves about 20GB versus Ollama's 86GB. User notes an OOM pitfall on load that requires dropping the page cache first.

generation:

19.9 tokens/s

prompt processing (prefill):

768.8 tokens/s

quant:

Q4_K (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks Nemotron-3-Super-120B-A12B at 19.94 t/s generation and 768.84 t/s prompt processing on an NVIDIA DGX Spark. Setup is llama.cpp with Q4_K GGUF, 65.10 GiB model size, flash attention enabled, 99 GPU layers, n_ubatch 2048. Batched benchmark shows aggregate throughput up to 56.16 t/s generation and 771.69 t/s prompt at 32 concurrent requests. A second model, Nemotron-3-Nano-4B Q8_0, was also benchmarked at 52.85 t/s generation and 2761.90 t/s prompt processing.

generation:

43.0 tokens/s

quant:

~3 bits per weight

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Nemotron 3 Super (120B, 12B active) at 43 tok/s on a single RTX 4090. Setup uses the glyd engine with a ~3 bits per weight quant (48.7 GB), hot experts on GPU and the rest computed on CPU from RAM; on 32 GB machines the remainder streams from SSD. Same 4090 with llama.cpp and Unsloth Q2_K_XL does 17 tok/s. Also reports 37 tok/s on RTX 3090, 35 tok/s on a 16 GB card, and 13-18 tok/s on a 32 GB RAM PC. Cites GSM8K 97% and MMLU-Pro 77%.