Bonsai 2 27B Ternary
on Unknown GPU · llama.cpp
Summary
User reports running a 27B ternary model across two phones over a USB-C cable at ~2.8 tok/s. The setup uses llama.cpp RPC over a localhost tunnel created via Android USB accessory mode, with no Wi-Fi, root, or adb.
The model is Ternary-Bonsai-2 (27B) in a Hadamard-rotated basis, run with a custom llama.cpp patch that applies rotation to activations. The two-phone pair is not yet faster than the fastest single phone, which achieves 5.62 tok/s on a Redmi, compared to 0.44 tok/s with the vendor's runtime.
The requantization increases perplexity to 14.26 from a 14.05 reference (+1.5%). A lossless repacker from 2-bit ternary GGUF to GGML's TQ1_0 base-3 container reduces the model size from 6.5 GiB to 5.3 GiB (−18%).