Bonsai 1.7B
on Unknown GPU · BitRun
Summary
User reports Bonsai 1.7B at ~73 tokens/sec generation on an Apple M4 Pro CPU-only, pulling 11.7 Watts (about 0.31 Joules per token).
Setup is BitRun v0.2.0-beta with GGUF ternary weights, running entirely on CPU memory.
On a Core i5-12600K, Bonsai 1.7B generates at 85 tokens/sec pulling around 22 Watts; the 8B Bonsai model reaches 21 tokens/sec on the M4 Pro at the same 11.7 Watts, and BitNet 70M clocks almost 600 tokens/sec.