Llama 3.2 1B
on NVIDIA RTX Pro 6000 Blackwell · vLLM · 8,192 ctx
May 5, 2026
Summary
User reports FastDMS, a DMS KV-cache compression implementation, decoding 1.5-2x faster than vLLM BF16/FP8 with 5-8x less KV memory.
Benchmarks cover Llama-3.2-1B and Qwen3-8B DMS checkpoints, with KLD and token match comparable or better than vLLM's FP8/TurboQuant.
Training took about 20 min on an RTX Pro 6000 Blackwell.