Inkling 2 Mini 26B (3.4B active)
NVIDIA RTX 5090 · NInfer · 10,000 ctx
- generation:
- 182.0 tokens/s
- prompt processing (prefill):
- 6664.0 tokens/s
- quant:
- UD-IQ4_XS (GGUF)
User reports Inkling 2 Mini at 182 t/s decode and 6,664 t/s prefill on an RTX 5090, compared with 97 t/s decode and 2,318 t/s prefill for llama.cpp on the same GGUF. Setup is the user's NInfer-all fork with Unsloth's UD-IQ4_XS GGUF, a 10K-token prompt, and the DSpark drafter enabled; llama.cpp was Unsloth's build because mainline does not load this GGUF. The model does not fit in 16 GB or less, so on those cards experts sit in system RAM and the GPU caches the most-used ones; reading experts straight from disk is slow, about 9 t/s on the RTX 5060. The post also lists results for the RTX 4090, 3090, 5080, 3060 and 5060.