Inkling-2 small
on AMD Strix Halo 128GB · libosmos
Summary
User reports Inkling-2-small running on a single AMD Strix Halo at 14-24 tok/s decode with drafting and over 310 tok/s prefill.
Setup is libosmos v0.8.0 with a patch pack based on the Unsloth UD-IQ1_S quant, about 156GB.
The pack reports a KLD of 0.196 and top-1 agreement of 85.8%, with more optimizations promised in later versions.