Bonsai 27B
on AMD RX 9070 · Bonsai runtime · 120,000 ctx
Summary
User reports 43.4 t/s with a q8_0 KV cache and 73.0 t/s with a kvarn4 KV cache at 120K context.
The q8_0 cache used about 13.1 GB total and the kvarn4 cache about 9.8 GB total.
The kvarn4 cache is 68% faster and uses 3.3 GB less VRAM.