MTP off, ngram on

on AMD Strix Halo 128GB · EngramHalo.cpp · 20,480 ctx

Oct 13, 2026

Throughput

21.4 t/s gen · 122.9 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.

Quant

UD-IQ4_XS (GGUF)

KV cache

Flash Attention

MTP (Multi-Token Prediction)

off

System RAM

128 GB

Summary

User reports Qwen3.8 at 21.4 t/s generation and 122.9 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is EngramHalo with UD-IQ4_XS, q8_0 target K/V cache, Flash Attention, eight slots of 20480 tokens, and one participating ROCm0 GPU. The headline is the first one-user request: 71 input tokens and 256 output tokens, with 0.6 s TTFT. The 21.4 t/s is the displayed native TG mean, not client aggregate throughput, which was 20.4 t/s.