Unknown family 26B (4B active)
AMD Radeon 780M · llama.cpp · 16,384 ctx
- generation:
- 25.0 tokens/s
- quant:
- Q4 QAT (GGUF)
- flash attention:
- on
User reports a fully local Italian voice assistant on a mini PC with a Radeon 780M iGPU and 64GB DDR5 shared memory, reaching a median ~2.2s from end of speech to first audio. The LLM is a 26B-A4B MoE at Q4 QAT running in llama.cpp on Vulkan at ~25 tok/s, with 16k context, one slot, flash attention and cache reuse. Prefill of the turn costs ~1.0s (~0.35s fixed per request plus ~3ms/token), ASR ~0.75s and first sentence plus TTS first chunk ~0.55s. The user puts the calculated floor on this box at ~1.2-1.5s. Smaller dense models were benchmarked and rejected for factual fidelity, and KV-cache warming during speech was net neutral-to-negative on one slot. The user asks what else could be done before buying a 24GB dGPU.