llamaperf

Nemotron 3.5 Lightning 30B (3B active) Ternary

on Unknown GPU · llama.cpp · 368,000 ctx

Oct 26, 2026
Throughput
172.0 t/s gen · 7000.0 tokens/s prompt processing (prefill)Prompt processing is how fast the input is read, not how long you wait for the first word. How to measure time to first token.
Quant
ternary 1.58-bit (GGUF)
VRAM reported
12 GB

Summary

User reports a ternary (1.58-bit) Nemotron 3.5 Lightning 30B-A3B GGUF running on a 12 GB GPU with 368K context, at 172 tok/s decode and 7000+ tok/s prefill. Setup is stock llama.cpp (validated on b11555+, b11796) and also the PrismML fork / Bonsai (validated on b10793), one file for both engines. The card does not name the specific GPU model.

Compare this run

Medians of plain runs: one device, one request, no speculative decoding, the whole model in the device's memory. The report above is one setup, so read it beside them and not as one of them.

Every Nemotron 3.5 Lightning report →