Inkling 2 Mini
NVIDIA RTX 4070 Ti · llama.cpp · 65,536 ctx
- generation:
- 51-64 tokens/s
- quant:
- mixB (GGUF)
- kv:
- q8_0
User reports Inkling 2 Mini at 51-64 tok/s decode and ~1,500 tok/s prefill at 64K context on a single RTX 4070 Ti 12 GB. Setup is a forked llama.cpp (inkling-fast) with a custom 2-3 bit mixB GGUF (10.3 GiB) and q8_0 KV cache, fitting in 11.6 GB VRAM including the Windows desktop; layers 22-23 run on CPU via -ot. Decode is 61-64 tok/s on short prompts and 51-53 tok/s at 22K; llama-bench pp512/tg128 gives 63.5-63.8 tok/s decode and 2,183-2,191 tok/s prefill. Quality is 90.9% top-1 vs Q8_0, KLD 0.048, PPL ratio 1.034.