Bonsai 2 4B Ternary
on Unknown GPU · ik_llama.cpp
Summary
User benchmarks Ternary-Bonsai-2-4B Q2_0 on an Intel Core i5-13420H (4 P-cores, AVX-VNNI) under ik_llama.cpp, CPU only. The offline-repacked Q2_0_R8 build reaches 49.08 t/s prompt processing (pp512) and 11.55 t/s generation (tg128).
Native Q2_0 gives 34.46 t/s pp512 and 9.56 t/s tg128; runtime repack with -rtr gives 49.99 t/s pp512 and 11.64 t/s tg128. The PR reports roughly +45% prompt processing and +22% token generation.
Perplexity at n_ctx=128 over 3 chunks is 13.0721 native, 13.0763 with -rtr and 13.0763 for Q2_0_R8.