llama.cpp on the NVIDIA DGX Spark
On the NVIDIA DGX Spark, llama.cpp runs Nemotron 3 120B · 12B active at 4-bit at a median 17.2 tokens per second for generation, from 2 plain runs (context not stated).
Speed by model on the NVIDIA DGX Spark
Plain runs only: one device, one request, no speculative decoding, the whole model in memory. Each row is one model size at one bit class, with the other engines' plain runs of the same model on this card beside it. Context length and build still differ between runs, and each side shows its own.
| Model | llama.cpp | Other engines |
|---|---|---|
| Nemotron 3 120B · 12B active 4-bit | 17.2 t/smedian of 2, context not stated | no other engine reported |
| Gemma 4 26B · 4B active 4-bit | 69.9 t/sone report, 256K context | vLLM 48.2 t/sone report, context not stated |
| Ling-3.0 124B · 5.1B active 5-bit | 35.7 t/sone report, 256K context | no other engine reported |
| Ornith1.5 35B · 3B active 6-bit | 69.0 t/sone report, 128K context | no other engine reported |
| Qwen3.8 125B · 6B active 4-bit | 24.5 t/sone report, 128K context | SGLang 35.0 t/sone report, 256K context |
11 llama.cpp reports on the NVIDIA DGX Spark
Newest first. A run on several cards, with speculative decoding, with part of the model in system RAM or with batched requests says so, since each describes a different setup.
- Nemotron 3 120B · 12B active · Q4_K14.4 t/s
- Nemotron 3 120B · 12B active · Q4_K19.9 t/s
- Gemma 4 31B6.0 t/s
- Ornith1.5 35B · 3B active · Q6_K / Q5_K / Q3_K69.0 t/s
- Gemma 4 26B · 4B active · Q4_K_M69.9 t/s
- Nemotron-3-Ultra 550B · 55B active · UD-Q2_K_XL5.2 t/s(2 cards)
- Qwen3.8 125B · 6B active · UD-Q4_K_XL24.5 t/s
- Qwen3.8 27B · UD-Q4_K_XL15.0 t/s(speculative decoding)
- Ling-3.0 124B · 5.1B active · Q5_K_M35.7 t/s
- DeepSeek V4 Flash 284B · 13B active · UD-IQ3_XXS24.0 t/s(speculative decoding)
- Muse 30B · K-Quant-DynamicPP 390.0 t/s(2 cards, speculative decoding)
Frequently asked
How fast is llama.cpp on the NVIDIA DGX Spark?
On the NVIDIA DGX Spark, llama.cpp runs Nemotron 3 120B · 12B active at 4-bit at a median 17.2 tokens per second for generation, from 2 plain runs (context not stated).
Is llama.cpp faster than other engines on the NVIDIA DGX Spark?
Same card, same model size and bit class, plain runs only: Gemma 4 26B · 4B active at 4-bit gives llama.cpp 69.9 t/s (1 run) and vLLM 48.2 t/s (1). Context length and engine build differ between the runs, so treat it as a lead, not a benchmark.