llamaperf

Inkling-2 Mini

on NVIDIA RTX 4090 · vLLM · 35,000 ctx

Tone: positiveNVIDIA hardware
Nov 21, 2026
Throughput
190-220 t/s gen
Quant
NVFP4 experts (W4A16) + FP8 W8A16 attention/head/embeddings (NVFP4)
KV cache
FP8
VRAM reported
24 GB

Summary

User reports Inkling 2 Mini running on a single RTX 4090 at about 190-220 tokens/s with vLLM, up from roughly 117 out of the box. Setup uses NVFP4 experts (W4A16) plus FP8 W8A16 attention, head and embeddings, with FP8 KV cache, fitting in 24 GB at a 35k context. The run uses the model's built-in drafter and required small patches to vLLM, which does not yet support 4090s for this model. An initial quality check on sample prompts reported about a 0.4% loss in perplexity; the user plans further testing.