Bonsai 2 27B Ternary
on Unknown GPU · llama.rn
Summary
User reports Ternary-Bonsai-2-27B at 5.6 tok/s text generation on a Xiaomi 15 phone, and about 2.8 tok/s when clustered across two Android phones (Xiaomi Redmi 100 Pro Max + ROG 7) over USB with a custom RPC split.
Setup is a custom C++ engine forked from llama.rn, with weights compressed to 2.06 bpw (6.52 GiB) in GGUF and kept ternary in the Hadamard-rotated basis, transformed at runtime on the ARM CPU.
The cluster run splits layers 4.36 GiB / 2.15 GiB and is bottlenecked by the slower phone. Perplexity is 14.26 against a 14.05 reference, and max temperature sits around 30°C.