Bonsai 1.7B Ternary
on Unknown GPU · ik_llama.cpp
Summary
User compares low-bit GGUF runtimes on CPU (ARM Neoverse-N1 server and AVX2 x86 VMs) against reference outputs. Ternary-Bonsai 1.7B prompt processing reaches 71 t/s with PQ2_0 in ik_llama.cpp with -rtr, versus 28 t/s in Prism and 2 t/s for Q2_0 g64 in Prism.
Setup is ik_llama.cpp with -rtr, which is off by default; without it ik_llama.cpp is often the slowest. llama.cpp merged an AVX2 Q2_0 kernel on Oct 13 (~6x faster) and ik_llama.cpp added Q2_0 the same day (~15x faster than Prism).
For the 27B, the smaller PTQ1_0 file is ~4.4x slower than PQ2_0 for prompt processing in Prism on AVX2. The Q2_0 file marked recommended on the Ternary-Bonsai-1.7B card does not load in any of the three runtimes. BitNet b1.58 2B4T on bitnet.cpp main is wrong on CPU, with perplexity ~6x the HF transformers reference on x86 and ???? output on ARM.