Ornith1.5 35B (3B active)
on AMD Radeon AI PRO R9700 32GB · WHIRL · 131,072 ctx
Use cases
codingagenticvisionlong-context
Summary
User reports Ornith-1.5-35B-A3B at 244.5 tok/s decode on an AMD Radeon AI PRO R9700 32GB, compared against llama.cpp b11214 at 118.8 tok/s.
Setup is WHIRL v0.1.0 with an MXFP4 GGUF quant (4.46 bpw, 18.45 GiB loaded) and F16 KV cache; the R9700 is connected as a USB4 eGPU. The 244.5 tok/s figure uses MTP + n-gram speculative decoding on 7 Chinese coding prompts (800 tokens, server); plain decoding without MTP gives 166.7 tok/s.
Prefill at 8,192 tokens is 10,858 tok/s versus 4,637 tok/s in llama.cpp. Four concurrent requests aggregate 396.7 tok/s versus 191.9 tok/s. Perplexity on a mixed Chinese/English code corpus is 5.5192 ± 0.0653 versus 5.3206 ± 0.0619 for the official Q4_K_M, 3.7% higher.