Inkling-2 Mini
on AMD Strix Halo 128GB · Gufo
Use cases
visiontool-uselong-context
Summary
User reports Inkling-2-Mini decoding at ~29-30 tok/s on Strix Halo with a from-scratch HIP engine written as a Gufo extension, against ~25 tok/s for llama.cpp on the same machine.
Setup is Gufo with UD-Q4_K_M GGUF weights in the Mainline_Support layout, single machine on gfx1151; prefill is ~650 tok/s and decode falls to 26.5 tok/s at 20k depth.
Continuous batching gives 56 tok/s aggregate across 4 concurrent requests versus 30 single. DSpark speculative decoding is lossless but slower at 17.6 tok/s with 10-30% acceptance. A 31k-token needle test took 71 s versus 255 s for llama.cpp. The author notes the hyperloom implementation is about 50% faster, placing this between llama.cpp and loom.
Compare this run
Medians of plain runs: one device, one request, no speculative decoding, the whole model in the device's memory. The report above is one setup, so read it beside them and not as one of them.
Models most run on the AMD Strix Halo 128GB
- 26.6 t/s · 8 runs
- 60.0 t/s · 7 runs
- 69.9 t/s · 6 runs
- 50.4 t/s · 4 runs
- 52.6 t/s · 4 runs
- 71.7 t/s · 4 runs
Bars are medians on one scale, 0 to 77.0 tokens/s. Each dot is one run and opens its report. Press ▶ to watch text arrive at a row's speed.
Everything reported on the AMD Strix Halo 128GB →Every Inkling-2 report →Estimate Inkling-2 on this card →