Unknown family 26B (4B active)
on AMD Radeon 780M · llama.cpp · 16,384 ctx
Use cases
agenticlong-context
Summary
User reports a fully local Italian voice assistant on a mini PC with a Radeon 780M iGPU and 64GB DDR5 shared memory, reaching a median ~2.2s from end of speech to first audio. The LLM is a 26B-A4B MoE at Q4 QAT running in llama.cpp on Vulkan at ~25 tok/s, with 16k context, one slot, flash attention and cache reuse.
Prefill of the turn costs ~1.0s (~0.35s fixed per request plus ~3ms/token), ASR ~0.75s and first sentence plus TTS first chunk ~0.55s. The user puts the calculated floor on this box at ~1.2-1.5s.
Smaller dense models were benchmarked and rejected for factual fidelity, and KV-cache warming during speech was net neutral-to-negative on one slot. The user asks what else could be done before buying a 24GB dGPU.
Compare this run
Medians of plain runs: one device, one request, no speculative decoding, the whole model in the device's memory. The report above is one setup, so read it beside them and not as one of them.
Models most run on the AMD Radeon 780M
- 19.7 t/s · 2 runs
- 25.0 t/s · one run
Bars are medians on one scale, 0 to 25.0 tokens/s. Each dot is one run and opens its report. Press ▶ to watch text arrive at a row's speed.