Nemotron 3.5 Bolt 74B (9B active)
NVIDIA RTX 5090 · llama.cpp · 32,768 ctx
- generation:
- 343.0 tokens/s
- quant:
- IQ2_XXS (GGUF)
- kv:
- q8_0
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Nemotron 3.5 Bolt 74B-A9B at 343 t/s generation on an RTX 5090 32GB, median of 15 runs (303 to 354). Setup is llama.cpp b11750 with NVIDIA's IQ2_XXS GGUF, q8_0 KV cache, 32K context, single stream, and the LiLiCorr drafter at 55% acceptance; 25.0 GiB of VRAM with the drafter and projector. Without the drafter it runs 166 t/s at 22.7 GiB, and 21.0 GiB without drafter or projector. llama-bench gives pp512 5,166 t/s and tg128 172 t/s. On a 33-task DevOps tool-use eval it passed 26 of 33 (79%, score 0.88) with zero safety failures, 54% unrecorded tool calls and 28.3 GiB VRAM at four slots.