llamaperf

Ornith1.5 35B (3B active)

on AMD Radeon AI PRO R9700 32GB · WHIRL · 131,072 ctx

AMD hardware
Oct 4, 2026
Throughput
244.5 t/s gen · 10858.0 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
MXFP4 (GGUF)
KV cache
F16
VRAM reported
32 GB

Use cases

codingagenticvisionlong-context

Summary

User reports Ornith-1.5-35B-A3B at 244.5 tok/s decode on an AMD Radeon AI PRO R9700 32GB, compared against llama.cpp b11214 at 118.8 tok/s. Setup is WHIRL v0.1.0 with an MXFP4 GGUF quant (4.46 bpw, 18.45 GiB loaded) and F16 KV cache; the R9700 is connected as a USB4 eGPU. The 244.5 tok/s figure uses MTP + n-gram speculative decoding on 7 Chinese coding prompts (800 tokens, server); plain decoding without MTP gives 166.7 tok/s. Prefill at 8,192 tokens is 10,858 tok/s versus 4,637 tok/s in llama.cpp. Four concurrent requests aggregate 396.7 tok/s versus 191.9 tok/s. Perplexity on a mixed Chinese/English code corpus is 5.5192 ± 0.0653 versus 5.3206 ± 0.0619 for the official Q4_K_M, 3.7% higher.