Inkling 2 Mini 26B (3.4B active)
on NVIDIA RTX 5090 · NInfer · 10,000 ctx
Use cases
vision
Summary
User reports Inkling 2 Mini at 182 t/s decode and 6,664 t/s prefill on an RTX 5090, compared with 97 t/s decode and 2,318 t/s prefill for llama.cpp on the same GGUF.
Setup is the user's NInfer-all fork with Unsloth's UD-IQ4_XS GGUF, a 10K-token prompt, and the DSpark drafter enabled; llama.cpp was Unsloth's build because mainline does not load this GGUF.
The model does not fit in 16 GB or less, so on those cards experts sit in system RAM and the GPU caches the most-used ones; reading experts straight from disk is slow, about 9 t/s on the RTX 5060. The post also lists results for the RTX 4090, 3090, 5080, 3060 and 5060.
Compare this run
Medians of plain runs: one device, one request, no speculative decoding, the whole model in the device's memory. The report above is one setup, so read it beside them and not as one of them.
Inkling 2 Mini 26B · 3.4B active at 4-bit by hardware
- AMD RX 7900 XT20 GB108.9 t/s · one run
Bars are medians on one scale, 0 to 108.9 tokens/s. Press ▶ to watch text arrive at a row's speed.
Models most run on the NVIDIA RTX 5090
- Qwen3.8 27B4-bit80.0 t/s · 7 runs
- 215.6 t/s · 2 runs
- Qwen3.8 27B6-bit69.1 t/s · 2 runs
- 243.9 t/s · one run
- Gemma 4 31B4-bit64.0 t/s · one run
- 563.1 t/s · one run
Bars are medians on one scale, 0 to 563.1 tokens/s. Each dot is one run and opens its report. Press ▶ to watch text arrive at a row's speed.
Everything reported on the NVIDIA RTX 5090 →Every Inkling 2 Mini report →Estimate Inkling 2 Mini on this card →