Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6 35B-A3B at 60 t/s on two BC-250 ex-mining APUs with 64k context.
Setup is llama.cpp with Vulkan and RPC on Bazzite, the two boards providing roughly 27GB of combined GPU memory over 1gb Ethernet.
The boards cost $115 each, around $300 total including PSU, and the user is pleased with the performance.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User asks what performance P100 owners get, having ordered one for $80, and reports their current baseline on an RX 6600 XT with 32GB DDR4 3600.
Current setup runs unsloth Qwen3.6 35B-A3B UD_Q4_K_XL in llama.cpp with MTP and --cpu-moe at 64k full-precision context, giving 30 t/s generation and 48-50 t/s decode, with prefill up to 800 t/s at 0 context and 700 t/s at 10k.
User hopes the P100 can match the decode numbers and plans to tune -b and -ub for its higher core count; the card will go into a dedicated inference machine.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6 35B-A3B at ~50 t/s on a 128GB Strix Halo APU running Nobara Linux.
Setup is llama.cpp with the Q6 quant.
User asks what newer models are recommended for the 128GB Strix Halo, either faster with the same capabilities or better with the same speed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports almost 70 t/s with Qwen3.6-35B-A3B Q4_K_XL on a budget build using three Nvidia Tesla P100 16GB GPUs.
The GPUs cost $80 each and are split across three nodes to manage thermals; the motherboard required a patched BIOS to enable Above 4G Decoding.
User is still testing and optimizing, and has published a GitHub repository for the build.
User reports a DIY Jev-like inference setup using unmodified open weight LLMs, evaluated on a 32,235-example benchmark.
Qwen3.6 35B-A3B reached 75.5% accuracy at ~5.3 req/s on a laptop RTX 5090 24GB, with Qwen3 27B at 75.3% ~2.9 req/s and Qwen3-4B at 65.0% ~27 req/s.
The approach uses boolean verification of candidate answers via true/false logits, batched through llama.cpp, with no NLI fine-tuning or classifier head. User notes the benchmark is not perfectly apples-to-apples and expresses skepticism about Jev hype.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen3.6 35B-A3B at 35 t/s generation and 1000 t/s prompt processing on an RTX 5070 12GB with an i5-10400F and 32GB RAM.
Setup is LM Studio with a 64k context, Q8 KV cache, flash attention, about 30 experts offloaded to CPU, and retain-thinking enabled. The run used about 10GB VRAM and 28GB system RAM at roughly 58K context.
User asks whether more performance can be squeezed from the rig and reports inconsistent context reporting between Cline and OpenCode.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
visionlong-context
User reports Qwen3.6-35B-A3B at 26.1 t/s generation and 233 t/s prompt eval at 131,072 context on an RTX A2000 12GB with a Xeon Silver 4314 and 128 GB DDR4.
Setup is llama.cpp master with the unsloth UD-Q5_K_M GGUF, q8_0 KV cache, --n-cpu-moe 32, MTP, and mmap on. Empty-context generation was 35-38 t/s; the 118k-deep figure was measured at --n-cpu-moe 31.
A 3000x2000 image request took 32 s, and the user notes that --n-cpu-moe 31 crashed on that image while 32 passed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6 35B A3B at 45 t/s on an RTX 4080 16GB, and Gemma 4 12B at 34 t/s.
Setup is llama.cpp with CUDA 13.3, MXFP4_MOE GGUF, 65535 context, q8_0 KV cache, and draft-MTP speculative decoding with n-max 4.
User finds MTP speculative decoding consistently reduces throughput rather than improving it, contrary to reports of 20-30% gains.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.6 35B-A3B at 77.92 t/s with MTP on an AMD Strix Halo 128GB, versus 53.83 t/s without MTP.
Setup is llama.cpp build b11009 with the unsloth UD-Q4_K_XL GGUF at 4096 context, using the Vulkan backend, flash attention, and draft-mtp speculative decoding with spec-draft-n-max 2.
The benchmark also covered Nemotron-3.5-Lightning-30B-A3B NVFP4, Qwen3.8 27B OBLITERATED Q4_K_M, GLM-4.5-Air Q4_K_M, Qwen3.8 27B UD-Q4_K_XL, Qwen3.8 Flash-Next UD-IQ4_XS, and Qwen3.6 35B-A3B BF16.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingsummarization
User reports Qwen3.6 35B-A3B at 600 t/s single request on an RTX Pro 6000 Blackwell.
Setup uses the Ninfer engine.
User describes it as a drudgework model for read-and-find or code tasks, noting it may use 20x more tokens but is still faster than many local models, and compares it favorably to Cerebras.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User asks whether speculative decoding (MTP) is still worth enabling for a 35B A3B MoE model offloaded across an 8GB RTX 5060 and 32GB of system RAM.
Current llama-server setup uses a Q4_K_XL GGUF with a Q8 KV cache, flash attention on, 4096 batch and ubatch, 16 CPU cores, 40 MoE layers on CPU and 99 GPU layers, yielding about 40 t/s generation and 500 t/s prompt processing without MTP.
User recalls earlier reports that MTP hurt prompt processing and wants to know if that is still the case and how others configure llama-server.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.6 35B-A3B at 19.7 t/s generation and 332.57 t/s prompt processing on a mixed GTX 1080 Ti and Radeon VII Vulkan setup.
Setup is llama.cpp Vulkan build b29c606e2 with the MXFP4 MoE GGUF and Flash Attention enabled, across two GPUs.
User also reports results for twelve other models including llama 7B, bailingmoe2 16B-A1B, gpt-oss 20B, Gemma 4 26B-A4B, Qwen3.5 27B, Qwen3-Coder 30B-A3B, granite 4.0, and Phi-3.5-MoE.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.6 35B A3B at 15-20 t/s on an M2 Max 32GB, with prompt processing taking 20 minutes for 40k tokens when the model fits in RAM.
Setup is Q4_XS GGUF at 60,000 context with a Q8 KV cache.
With SSD streaming the model runs at 4-10 t/s and prompt processing takes 50-60 minutes for 40-60k tokens. User asks whether to upgrade to an M1 Max 64GB, M3 Max 128GB, or a 24GB GPU build with 128GB RAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.6 27B at 7.4 t/s generation and 121.3 t/s prompt processing on an M3 Max 96GB.
Setup is oMLX with the MLX 8-bit model at pp1024/tg128, using 28.34 GB peak memory.
A pp4096/tg128 run reached 8.8 t/s generation and 133.8 t/s prompt processing. Continuous batching at 4x reached 19.9 t/s aggregate generation. User is new to LLMs and asks whether the slow speed is due to the dense model or a setup problem.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingsummarizationagentic
User compares ninfer-3090 against llama.cpp on an RTX 3090 for Qwen3.6-35B-A3B and Qwen3.8-27B.
The primary record is Qwen3.6-35B-A3B with ninfer-3090. llama.cpp with the UD-IQ4_XS GGUF reached 2439.7 t/s prefill, 148.6 t/s generation and 3410 ms TTFT. For Qwen3.8-27B, ninfer reached 187911.6 t/s prefill, 34.9 t/s generation and 27 ms TTFT, while llama.cpp reached 1012.7 t/s prefill, 38.5 t/s generation and 8160 ms TTFT.
The benchmark used 7 prompts (4 long at roughly 7.7k-12.9k tokens, 1 medium at roughly 2.3k, 2 short at roughly 400-600), 3 repeats each, max_tokens=1024, seed=42 and cache busting via nonce. The ninfer-3090 system is a Ryzen 5 2600 with 16 GB RAM; the llama.cpp system is a Threadripper PRO 3945WX with 128 GB RAM. The user notes the hardware is not apples-to-apples but that inference is GPU-contained.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User benchmarks multiple models on 16 GB of VRAM. Qwen3.6-35B-A3B-APEX-I-Quality reaches 96.7% HumanEval pass@1, 100% agentic success and 37.2 tok/s. Ornith-1.0-35B IQ4_NL runs at 38 tok/s.
Other models tested are Qwen3.8-27B variants, gemma-4-26B-A4B, KAT-Coder-V2.5-Dev and Qwen3.5-9B.
Ornith 1.0 35B A3B wins overall.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports llama.cpp PR #26563, which caches hot MoE experts on GPU, running Qwen3.6-35B-A3B on 8 GB of VRAM at 33.25 to 56.0 t/s with Q2_M and 17.34 to 35.93 t/s with Q5_K_P.
Setup uses autofit with --expert-hot-s -1.
The gains are 1.68x and 2.07x respectively. The user reports negative results for Qwen3.5-122B-A10B and Laguna-S-2.1. The change is CUDA only and applies to single-token decoding only.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.6-35B-A3B with MTP enabled on llama.cpp server, reaching 175,000-200,000 tokens of context with a Q8 KV cache.
Prompt processing runs 200-300 t/s and generation 25-40 t/s depending on draft hit rates.
VRAM is constrained at 10 GB with a single KV cache slot. The user runs a custom fork of the pi coding agent for sub-agents.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports 29 t/s on Qwen3.6-35B-A3B IQ4_NL with the llama.cpp Vulkan backend on a Strix Point HX370 with an 890M iGPU and 64 GB RAM.
MTP is enabled with acceptance rates of 70-90%, giving effective tokens/sec of 50-70.
User expresses interest in Qwen3.8 but notes current models may not perform as well.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
summarizationlong-context
User reports Qwen 27B Q4_K_M on Ollama at 100k context, running on an RTX 5060 Ti and 5070 Ti, with ~1050 t/s prompt processing and ~46 t/s generation on a ~35k token prompt.
Setup is Open WebUI with Open Terminal integration on Unraid, with inference on a separate machine.
The use case is legal document research and creation.