llamaperf

Nemotron 3.5 Lightning

NVIDIA · 9 reports

Nemotron 3.5 Lightning VRAM requirements by size and quant →

How does Nemotron 3.5 Lightning run on your hardware?

Pick your GPU or Mac, then read the speeds people reported for Nemotron 3.5 Lightning on it, or estimate memory fit and speed.

Free to use. No account needed. Memory estimates and community measurements are labelled separately.

Run Nemotron 3.5 Lightning yourself? Add your numbers.

Your coding agent can run a speed test on your machine and send it here with a free account. Or paste a result you've already posted.

Reported performance for Nemotron 3.5 Lightning

Filter this model’s reports by setup →

Speeds are as each source reported them. GPU count, offloading and concurrent requests change them, and prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing two reports.

Nemotron 3.5 Lightning 30B (3B active) Ternary

Unknown GPU · llama.cpp · 368,000 ctx

generation:
172.0 tokens/s
prompt processing (prefill):
7000.0 tokens/s
quant:
ternary 1.58-bit (GGUF)

User reports a ternary (1.58-bit) Nemotron 3.5 Lightning 30B-A3B GGUF running on a 12 GB GPU with 368K context, at 172 tok/s decode and 7000+ tok/s prefill. Setup is stock llama.cpp (validated on b11555+, b11796) and also the PrismML fork / Bonsai (validated on b10793), one file for both engines. The card does not name the specific GPU model.

Oct 26, 2026
Apple hardware
generation:
42.3 tokens/s
quant:
UD-Q3_K_XL (GGUF)

User reports Nemotron 3.5 Lightning 30B-A3B at 42.3 t/s on an Apple Mac mini M6 with 32 GB unified memory. Setup is LM Studio with its llama.cpp GGUF backend, UD-Q3_K_XL quant, and 32,768 tokens of loaded context. The 42.3 t/s is the median of three English-prose runs (Ruhr economy prompt), client-observed streaming decode throughput, not an overall mean or prefill rate.

Oct 13, 2026
Apple hardware
generation:
45.1 tokens/s
quant:
Q4_K_M (GGUF)

User reports Nemotron 3.5 Lightning 30B-A3B at 45.1 t/s on an Apple Mac mini M6 with 32 GB unified memory. Setup is LM Studio with its llama.cpp GGUF backend, GGUF Q4_K_M, and a loaded context of 32,768 tokens. The 45.1 t/s is the median of three English-prose runs and is client-observed streaming decode throughput, not an overall mean, peak, or prefill rate.

Oct 13, 2026

Nemotron 3.5 Lightning 30B (3B active)

Jetson AGX Orin 64GB · vLLM

generation:
80.0 tokens/s
agenticlong-context

User reports the current vLLM setup for Nemotron 3.5 Lightning 30B-A3B on a Jetson AGX Orin 64GB runs at around 80 tok/s while occupying roughly 62 GB of unified memory. The post is a benchmark plan: it proposes comparing SGLang and llama-server against the vLLM baseline, measuring cold and warm-prefix TTFT, decode tok/s, and resident memory across 4K to 128K prompts, a 32K shared-prefix four-agent workload, and constrained budgets of 32, 40 and 48 GB. No SGLang or llama-server results are given yet; the 80 tok/s figure is the existing vLLM baseline.

Oct 9, 2026
Tone: positiveNVIDIA hardware
generation:
110.0 tokens/s
quant:
Q4_0 (GGUF)
kv:
F16
codingagenticlong-context

User reports Nemotron-3.5-Lightning-30B-A3B at 110 tok/s writing code on two Tesla P100 16GB cards. Setup is llama.cpp b10970 with Q4_0 weights, F16 KV cache, tensor split across both cards, and the model's built-in MTP draft head at n-max 2. The same model reaches 95 tok/s on prose, 50 tok/s at 128k context, 39 tok/s at 256k, and 16 tok/s at 1M tokens. The study covers 545 speed measurements of 29 models from 2B to 122B parameters, all weights and KV cache in VRAM with no system RAM offload. A 119B MoE model runs 39 tok/s against 4.3 tok/s for a 70B dense model on the same cards. Tensor split makes dense models from 8B up 21-44% faster. A single P100 throttles to 906 MHz and loses 25% under sustained load, while two cards share the heat and lose 5.5%.

Oct 6, 2026
Tone: positiveNVIDIA hardware
generation:
72-87 tokens/s
quant:
Q4_K_M (GGUF)
agentictool-use

User reports Nemotron 3.5 Lightning 30B-A3B at 72 to 87 tok/s single-stream decode and about 2,600 tok/s prefill on a DGX Spark (GB10, 128GB unified memory). Setup is Ollama 0.32.9 with the Q4_K_M GGUF at the default 262,144 token context, 26GB resident at 100% GPU, with the built-in MTP speculative decoding active. Decode depends on the workload: 72 tok/s on prose and 84 to 87 tok/s on JSON and summaries. The same model served with vLLM using the NVFP4 checkpoint and DSpark draft model reached 108 tok/s decode and about 5,400 tok/s prefill. On one agent prompt the model answered in 485 tokens and 5.9s against 1,953 tokens and 26.0s for qwen3.5:35b-a3b.

Oct 4, 2026
Tone: positiveIntel hardware
generation:
91.9 tokens/s
prompt processing (prefill):
1760.0 tokens/s
quant:
Q4_0 (GGUF)
agenticcoding

User reports Nemotron 3.5 Lightning 30B-A3B at 91.91 tok/s decode on 2x Intel Arc Pro B60 24GB. Setup is llama.cpp SYCL with Q4_0 weights and an MTP Q8_0 drafter at --spec-draft-n-max 7, 22.18 GiB VRAM, 1,760 tok/s prefill at 12K context. MTP acceptance is 99.5-100% at every n-max; the model needs the whole card with no co-residence.

Oct 3, 2026
Tone: positiveNVIDIA hardware
generation:
80.0 tokens/s
prompt processing (prefill):
4737.5 tokens/s
quant:
NVFP4 (GGUF)
kv:
q8_0

User reports Nemotron 3.5 Lightning 30B-A3B at 79.98 t/s generation and 4737.46 t/s prompt processing on 2x RTX 5060 Ti 16GB. Setup is llama.cpp with NVFP4 GGUF weights and q8_0 KV cache, 1048576 token context, flash attention on, no MTP layers. The run processed a 54025-token prompt and generated 2156 tokens; the user notes it fits in 32 GB VRAM with no expert-layer offloading to RAM.

Sep 28, 2026
Tone: positiveNVIDIA hardware
quant:
W4A16

User compares a W4A16 quant of Nemotron 3.5 Lightning 30B-A3B against an IQ4_XS GGUF on an RTX 3090. Setup is vLLM for W4A16 and llama.cpp for IQ4_XS. The two are near-parity in instruction following benchmarks, with about 4.5x throughput by B16. The user describes the model as fast and reliable, and suitable for batch labelling and agentic responses.

Sep 7, 2026
Engines people run Nemotron 3.5 Lightning with
EngineReports
llama.cpp4
LM Studio2
vLLM2
Ollama1

Get a weekly email of new Nemotron 3.5 Lightning reports on any GPU.

Email me new reports

Reported configurations

Individual observations from this page, not expected speeds or a ranking. Select a model to open its report, with the source and offloading conditions. Reported context may be a limit rather than actual input length.

Model / hardwareQuant / engineReported contextGeneration
Nemotron 3.5 Lightning 30B (3B active)
M6 32GB
UD-Q3_K_XL
LM Studio
32,76842.3 tokens/s
Nemotron 3.5 Lightning 30B (3B active)
M6 32GB
Q4_K_M
LM Studio
32,76845.1 tokens/s
Nemotron 3.5 Lightning 30B (3B active)
2× NVIDIA Tesla P100 16GB
Q4_0
llama.cpp
Not reported110.0 tokens/s
Nemotron 3.5 Lightning 30B (3B active)
2× Intel Arc Pro B60 24GB
Q4_0
llama.cpp
Not reported91.9 tokens/s
Nemotron 3.5 Lightning 30B (3B active)
2× NVIDIA RTX 5060 Ti 16GB
NVFP4
llama.cpp
1,048,57680.0 tokens/s