llamaperf

Ornith1.5 35B (3B active)

on NVIDIA Tesla P40 24GB · llama.cpp

Tone: positiveNVIDIA hardware
Oct 28, 2026
Throughput
60.8 t/s gen · 1104.0 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
Q4_0 (GGUF)
KV cache
q8_0
VRAM reported
24 GB

Summary

User reports Ornith 1.5 35B-A3B at 60.8 t/s decode and 1104 t/s prefill at zero depth on a single Tesla P40 24GB. Setup is a custom llama.cpp fork with Q4_0 weights, q8_0 KV cache, full 262k context, on Windows over PCIe x1; the llama-bench table is without MTP. Decode falls to 46.0 t/s at 32k, 29.1 t/s at 100k and 18.8 t/s at 190k. With MTP (draft n=2) the real server gives 65-74 t/s on short prompts, and a 101k-token prompt reads at ~415 t/s and generates at 35-39 t/s. ECC off added 1.5 GB VRAM and ~9% decode; q4 KV gave 30.6 t/s versus 42.2 t/s for q8 at 32k.