llamaperf

Inkling 2 Mini

on NVIDIA RTX 4070 Ti · llama.cpp · 65,536 ctx

Tone: positiveNVIDIA hardware
Nov 22, 2026
Throughput
51-64 t/s gen
Quant
mixB (GGUF)
KV cache
q8_0
VRAM reported
12 GB

Use cases

tool-usevision

Summary

User reports Inkling 2 Mini at 51-64 tok/s decode and ~1,500 tok/s prefill at 64K context on a single RTX 4070 Ti 12 GB. Setup is a forked llama.cpp (inkling-fast) with a custom 2-3 bit mixB GGUF (10.3 GiB) and q8_0 KV cache, fitting in 11.6 GB VRAM including the Windows desktop; layers 22-23 run on CPU via -ot. Decode is 61-64 tok/s on short prompts and 51-53 tok/s at 22K; llama-bench pp512/tg128 gives 63.5-63.8 tok/s decode and 2,183-2,191 tok/s prefill. Quality is 90.9% top-1 vs Q8_0, KLD 0.048, PPL ratio 1.034.

Compare this run

Medians of plain runs: one device, one request, no speculative decoding, the whole model in the device's memory. The report above is one setup, so read it beside them and not as one of them.

Models most run on the NVIDIA RTX 4070 Ti

Bars are medians on one scale, 0 to 118.0 tokens/s. Press ▶ to watch text arrive at a row's speed.

Everything reported on the NVIDIA RTX 4070 Ti →Every Inkling 2 Mini report →Estimate Inkling 2 Mini on this card →