- generation:
- 29-30 tokens/s
- prompt processing (prefill):
- 650.0 tokens/s
- quant:
- UD-Q4_K_M (GGUF)
User reports Inkling-2-Mini decoding at ~29-30 tok/s on Strix Halo with a from-scratch HIP engine written as a Gufo extension, against ~25 tok/s for llama.cpp on the same machine. Setup is Gufo with UD-Q4_K_M GGUF weights in the Mainline_Support layout, single machine on gfx1151; prefill is ~650 tok/s and decode falls to 26.5 tok/s at 20k depth. Continuous batching gives 56 tok/s aggregate across 4 concurrent requests versus 30 single. DSpark speculative decoding is lossless but slower at 17.6 tok/s with 10-30% acceptance. A 31k-token needle test took 71 s versus 255 s for llama.cpp. The author notes the hyperloom implementation is about 50% faster, placing this between llama.cpp and loom.