Local model performance on your hardware

generation:

157.0 tokens/s

quant:

EXL3 3.5bpw (EXL3)

kv:

Q8

mtp (multi-token prediction):

on

User reports Qwen3.8-27B at 157 tok/s decode on an RX 7900 XTX 24GB with DFlash2 speculative decoding at 2K context. Setup is a ROCm port of ExLlamaV3 with EXL3 3.5bpw weights, Q8 KV cache, and a DFlash2 5.0bpw draft model, served through TabbyAPI. Decode drops to 147 tok/s at 32K and 107 tok/s at 99K; MTP gives 108 tok/s at 2K. Prefill is ~1180-1200 tok/s at 32K. Through the TabbyAPI OpenAI endpoint, code reaches ~186 tok/s and prose ~78 tok/s. The same GPU with llama.cpp and an IQ3_XXS GGUF decoded at 40-53 tok/s.