Inkling 2 Mini 26B (3B active) abliterated
NVIDIA RTX 4090 · Xolotl · 8,192 ctx
- generation:
- 59.7 tokens/s
- prompt processing (prefill):
- 1162.0 tokens/s
- quant:
- XQ (GGUF)
User reports Inkling 2 Mini at 59.7 t/s decode and 1,162 t/s prefill at 8K on a single RTX 4090. Setup is a custom engine written in Xolotl with a 13.9 GB GGUF quant of the abliterated model, which fits on the 24 GB card. The model is a 26B MoE with about 3B active parameters and a looped middle of 12 layers run 3 times per token. Held-out KL against the parent is 0.013 with 96.3% top-1 agreement; refusals were 4/100 on a harmful prompt set and 1/100 on harmless ones. The model's MTP drafter is an optional separate file and was not used for these figures. User expects 100+ t/s after fusing the many small per-token kernels.