Local model performance on your hardware

generation:

19.2 tokens/s

quant:

UD-IQ3_XXS (GGUF)

User benchmarks MoMoE, an MIT-licensed fork of Strata, against llama.cpp on Ornith-1.5-122B-A10B UD-IQ3_XXS on an RTX 4070 Ti 12 GB, reporting 19.2 t/s versus 4.6 t/s. MoMoE runs mixture-of-experts models on one GPU plus system RAM, so weights are partly offloaded. The post also lists Qwen3.8-Flash-Next at 65-77 vs 23.1 t/s and Ornith-1.5-35B-A3B at 132-154 vs 77.8 t/s on the same card, both with MTP drafting. User notes llama.cpp ties or wins when a model fits on the GPU, citing LFM2.5-24B at 105 vs 76 t/s and Flint at 103 vs 90 t/s on the PC, and says the fork is early and tested only on their own machines.

generation:

108.0 tokens/s

quant:

4-bit (MLX)

User reports Ornith 1.5 35B-A3B at 108 tok/s on an M2 Max 32GB, versus 66 tok/s with the macOS Aerial lock-screen animation playing. Setup is oMLX with a 4-bit MoE model, medians of 4-6 local runs over 512 output tokens. After an 8k-token prompt, generation was 89 vs 66 tok/s, and WindowServer CPU use rose from roughly 25% to 44% while the animation ran. Using caffeinate -i instead of -d let the display sleep and restored speed.

generation:

15.5 tokens/s

prompt processing (prefill):

548.0 tokens/s

quant:

Q4_K_XL (GGUF)

User reports Ornith-1.5-122B at 15.5 t/s decode and 548 t/s prefill on a single RTX 3090 at 32K context. Setup is llama.cpp with Q4_K_XL GGUF and routed experts kept in system RAM via --n-cpu-moe 38, on a cloud pod with 32 vCPU EPYC 7763 and 125 GB RAM. Decode ranged 12-22 t/s across runs; Q3 reached 26 t/s and 128K context fits with --n-cpu-moe 40. Q4 peaked at 55 GiB RAM and the pod cost $1.53.

generation:

86-91 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

flash attention:

on

User reports Ornith-1.5-35B-A3B Q4 at 86–91 t/s generation and about 1,030 t/s prompt processing on an RTX 4070 Ti SUPER 16 GB, at a roughly 3–6k-token prompt. Setup is llama.cpp b11370 (CUDA) in router mode with Q4_K_M weights, q8_0 KV cache, 65536 context, flash-attn on, and n-cpu-moe 15, using about 15.1–15.6 GB of the 16 GB card. The figures are given as ranges, so no single throughput is recorded. The run is compared against a previous RTX 3060 Ti 8 GB setup; the host is an i7-14700K with 32 GB RAM on Windows 11.

generation:

20.0 tokens/s

User asks for advice on upgrading a local stack with a €1000 budget, currently a Ryzen 5 9600, RTX 5060 8GB and 32GB DDR5. They report running Ornith 1.5 35B A3B with offload at an average of 20 tk/s, alongside Ornith 1.5 9B, and embeddings, TTS and STT on system RAM. Options considered are a used RTX 3090 at 600-700, a second GPU such as an RTX 3060, or a 16GB Mac Mini M6 at 850; no benchmark of those parts is given.

generation:

61.5 tokens/s

User reports Ornith1.5-35B-A3B Splash at 61.5 t/s on an Apple Mac mini M6 with 32GB unified memory. Setup is LM Studio with the Splash backend and 32,768 tokens of loaded context, one model loaded alone. The 61.5 t/s is the median of three English-prose runs, client-observed streaming decode throughput rather than a pooled rate or peak; the three runs measured 64.7, 59.4 and 61.5 t/s.

generation:

43.7 tokens/s

quant:

Q4_K_M (GGUF)

User reports Ornith1.5 35B-A3B at 43.7 t/s on an Apple Mac mini M6 with 32 GB unified memory. Setup is LM Studio with its llama.cpp GGUF backend, GGUF Q4_K_M, and a loaded context of 32,768 tokens. The 43.7 t/s is the median of three English-prose runs (Ruhr economy prompt) and is client-observed streaming decode throughput, not an overall mean, peak speed, or prefill rate.

generation:

97.5 tokens/s

prompt processing (prefill):

1870.0 tokens/s

quant:

IQ4_XS (GGUF)

User reports Ornith 1.5 35B-A3B at 97.5 t/s generation and 1870 t/s prompt processing on two NVIDIA RTX 3060 12GB cards. Setup is llama.cpp with an IQ4_XS GGUF quant; the author's own Ornith quant, similar to Unsloth UD-IQ4_XS, with a 16.94 GiB model file. The figures are separate llama-bench tests, PP512 and TG128, not one 512-input/128-output request; repetitions, KV cache type, context length and GPU split are unreported.

generation:

50.1 tokens/s

prompt processing (prefill):

563.2 tokens/s

quant:

Q6_K (GGUF)

User reports Ornith-1.5-35B-A3B at 50.1 t/s generation and 563.2 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp b11414 with Vulkan, Q6_K GGUF, one integrated Radeon 8060S, and 32,768 existing KV tokens. The two figures come from separate depth-32768 tests: prompt processing adds 4096 tokens to the existing cache, and generation covers 128 tokens at that starting depth, not one end-to-end request.

generation:

15.0 tokens/s

prompt processing (prefill):

200.0 tokens/s

kv:

Q8_0

User reports Ornith 1.5 9B at ~15 tok/s generation and ~200 tok/s prompt processing on an 8 GB RTX 4060. Setup is llama.cpp with Q8_0 KV cache and MTP speculative decoding via a draft model, flash attention on. Without MTP the user gets double these numbers, and reports a 0.95 draft acceptance rate with a mean accepted length of 2.27. The user also notes memory bandwidth and GPU utilisation are about half with MTP enabled.

generation:

79.0 tokens/s

quant:

NVFP4 (NVFP4)

User reports Ornith-1.5-35B-A3B at 79 tok/s single-stream on one NVIDIA DGX Spark (128GB unified memory). Setup is vLLM 0.24 with the official 4-bit NVFP4 checkpoint (about 22 GiB) and multi-token prediction speculative decoding, serving the full 262K context. At 20 concurrent users each stream gets about 25 tok/s, 490 tok/s total; GPU memory utilization had to be capped at 0.70 and throughput dropped past 20 users. Reasoning scores cited: MATH-500 97%, GSM8K 98%, MMLU-Pro 81%.

generation:

180.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports Ornith 1.5 35B-A3B at about 180 tok/s on two RTX 5070 Ti 16GB cards, roughly 3x the 55-70 tok/s of the Qwen3.8 27B it replaced. Setup is Ollama on Windows with Q4_K_M at 128K context, 24.4GB fully on the two cards; the agent runs on pi in WSL. Two runs gave 176 and 183 tok/s with a short prompt and thinking off. At 256K about 1.2GB spills to system RAM and speed drops to about 139 tok/s, so 128K stays the default. Both of the user's agent tests max out at 9/9 and 10/10, which the user notes only shows it is not worse than the 27B, not that it is smarter. Vendor numbers of 79 on SWE-bench Verified and 68.5 on Terminal-Bench 2.1 are cited but untested.

generation:

169.4 tokens/s

quant:

Q4_K_M (GGUF)

User reports Ornith-1.5-35B-A3B-BigBang-MTP at 169.4 tok/s with MTP speculative decoding on a single RTX 3090, up 29% from 131.0 tok/s without the draft. Setup is llama.cpp with Q4_K_M main weights and a Q8_0 draft, single RTX 3090, temp 0.1. The grafted Qwen3.6 MTP head accepts 0.55-0.75 of tokens (code high, chat lower) with mean accepted runs of 3.2-4.0 tokens, versus ~0.13 acceptance for stock Ornith-1.5's random-init head. Stock Ornith-1.5 measured 133.9 tok/s without a draft. Perplexity on mixed code and prose was 3.34 versus 3.41 for stock, and both scored 15/15 on a 15-task Python pass@1. The author calls this a small local eval, not a benchmark suite.

generation:

72.6 tokens/s

prompt processing (prefill):

1504.9 tokens/s

quant:

Q4_0_ROCMFP4 (GGUF)

User reports Ornith-1.5-35B-A3B at 72.6 t/s generation and 1504.9 t/s prefill on AMD Strix Halo 128GB. Setup is llama.cpp-rocm with Q4_0_ROCMFP4 GGUF quantization, -ngl 99, benchmarked with llama-bench -p 256,2048 -n 256 -r 5. Seven ROCmFPX quant tiers are provided; the Q4_0_ROCMFP4_FAST tier is the bare baseline. The model is a 35B MoE with ~3B active parameters and supports vision via bundled mmproj and MTP speculative decoding.

generation:

69.0 tokens/s

quant:

Q6_K / Q5_K / Q3_K (GGUF)

User reports Ornith-1.5-35B-A3B APEX i-quality at 69.0 t/s on a DGX Spark (GB10). Setup is llama.cpp with Q6_K/Q5_K/Q3_K banded GGUF at 131072 context, single slot, no speculation. A second no-drafter run measured 69.9 t/s. With MTP draft-mtp at n_max 1 the model reached 73.52 t/s (0.674 acceptance); deeper draft depths were slower. i-compact measured 73.9 t/s and i-mini 80.8 t/s via llama-cli. The agentic coding suite passed 9 of 9 tasks and 154 of 154 tests; vision passed 5 of 5 with the image and 0 of 5 blind.

generation:

244.5 tokens/s

prompt processing (prefill):

10858.0 tokens/s

quant:

MXFP4 (GGUF)

kv:

F16

User reports Ornith-1.5-35B-A3B at 244.5 tok/s decode on an AMD Radeon AI PRO R9700 32GB, compared against llama.cpp b11214 at 118.8 tok/s. Setup is WHIRL v0.1.0 with an MXFP4 GGUF quant (4.46 bpw, 18.45 GiB loaded) and F16 KV cache; the R9700 is connected as a USB4 eGPU. The 244.5 tok/s figure uses MTP + n-gram speculative decoding on 7 Chinese coding prompts (800 tokens, server); plain decoding without MTP gives 166.7 tok/s. Prefill at 8,192 tokens is 10,858 tok/s versus 4,637 tok/s in llama.cpp. Four concurrent requests aggregate 396.7 tok/s versus 191.9 tok/s. Perplexity on a mixed Chinese/English code corpus is 5.5192 ± 0.0653 versus 5.3206 ± 0.0619 for the official Q4_K_M, 3.7% higher.

generation:

175.3 tokens/s

prompt processing (prefill):

9698.5 tokens/s

quant:

MXFP4 (MXFP4)

User reports Ornith-1.5-35B-A3B at 175.25 tok/s generation and 9,698.55 tok/s prompt processing at concurrency 1 on an Intel Arc Pro B70. Setup is GRIMOIRE, a native C++/SYCL inference engine, with MXFP4 weights and 4096-token prompts. At concurrency 8 the engine reaches 391.04 tok/s total generation and 10,154.18 tok/s total prompt processing. A separate v1.5 build measured 193.6 tok/s at c1 and 464.2 tok/s total at c8, with pp512 at c8 measuring 9,547 tok/s.

generation:

9.4 tokens/s

User reports Ornith 1.5 35B at around 9.4 t/s on an RTX 2050 4GB laptop with 16GB RAM, streaming the model mostly from SSD with a custom inference engine. For reference, GPT-OSS 20B gets around 6 t/s using GPU+CPU in LM Studio on the same machine. User asks for pruned or distilled MoE models or Q4 quants under 80GB for web development.

generation:

108.9 tokens/s

quant:

Q4_K (GGUF)

kv:

FP16

User reports Ornith-1.5-35B-A3B decoding at 108.9 tok/s on an AMD MI50 32GB, 1.9x the llama.cpp baseline of 57.2 tok/s on the same GGUF and GPU. Setup is a custom HIP engine with Q4_K weights and FP16 KV cache, short context of 273-529 tokens, model load 27.6 s, sclk 1725 MHz under load. The engine reproduces llama.cpp's 8-token greedy continuation 8/8 teacher-forced and free-running; the MTP layer is not loaded.

prompt processing (prefill):

300-400 tokens/s

User reports Ornith 1.5 35B-A3B at 30-40 t/s generation and 300-400 t/s prompt processing on an RTX 5060 Laptop 8GB with 32GB DDR5 RAM. Setup uses llama.cpp via OpenCode, with generation at 30 t/s for 100k-140k context and up to 40 t/s below 100k context. User also ran Qwen 3.8 27B on the same hardware and compares the two models for coding tasks, noting Ornith handles contained tasks well but struggles with complex multi-file changes.