BitNet b1.58 2B
on Unknown GPU · Project Zero
Summary
User reports BitNet b1.58-2B-4T at 36 t/s on a Xeon with Project Zero, a self-written C99 inference engine, about 1.8x faster than bitnet.cpp.
The engine also runs SmolLM2 F16 at ~100 t/s on an i5-11300H, where llama.cpp is about 7% faster, and DeepSeek Q4_K at 1.9 t/s versus 13.7 t/s for llama.cpp.
It is built with GCC and make, uses AVX-512 kernels for ternary packing, and exposes an OpenAI-compatible API with SSE streaming.
Compare this run
Medians of plain runs: one device, one request, no speculative decoding, the whole model in the device's memory. The report above is one setup, so read it beside them and not as one of them.