Flint 40B (2B active)
on AMD Strix Halo 128GB · halogen · 2,048 ctx
Summary
User reports Flint, a 40B MoE with about 2B active parameters, decoding at ~98 tok/s on a Ryzen AI Max+ 395 with 128 GB of unified memory.
Setup is the halogen server (0.18.14-flint-preview image) with int4 weights, at 1-2k context.
Decode falls to ~67-70 tok/s at 32k context, where prefill is ~2,450 tok/s against ~3,700 tok/s on a 2k prompt. Four concurrent requests total ~190 tok/s, and follow-up turns resume from cache with ~0.4 s to first token at 32k.