Inkling-2 Mini
on NVIDIA RTX 4090 · vLLM · 35,000 ctx
Summary
User reports Inkling 2 Mini running on a single RTX 4090 at about 190-220 tokens/s with vLLM, up from roughly 117 out of the box.
Setup uses NVFP4 experts (W4A16) plus FP8 W8A16 attention, head and embeddings, with FP8 KV cache, fitting in 24 GB at a 35k context. The run uses the model's built-in drafter and required small patches to vLLM, which does not yet support 4090s for this model.
An initial quality check on sample prompts reported about a 0.4% loss in perplexity; the user plans further testing.