Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.5-35B-A3B at 14.85 t/s generation and 212.05 t/s prompt processing on an AMD Radeon 890M (gfx1150) iGPU.
Setup is llama.cpp (build a0ed91a44) with a Q4_K_XL GGUF, 99 layers offloaded, comparing Vulkan and ROCm backends with flash attention on and off.
ROCm reaches 227.54 t/s prefill and 12.68 t/s decode without flash attention, and 229.81 t/s prefill with 13.18 t/s decode with it. A second build with unified memory and ROCm flash attention gave similar results. The same machine also ran llama-2-7b Q4_0 at 17.14 t/s decode on Vulkan and 15.52 t/s on ROCm.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-9B INT4 at about 3.2 t/s generation and 6 t/s prefill on 2x SQRL FK33 FPGAs at 75 MHz.
Setup is a custom VHDL inference engine with a 256-token prompt, pipeline split across the two cards with the host carrying the residual between them; output was checked against llama.cpp layer by layer.
Generation falls to about 2.4 t/s at 2-3k context and prefill to about 5.5 t/s on a 2.3k-token prompt. The user also gives modelled, not yet run, estimates for Qwen3.8-27B INT4 on Jungle Cat and 4x VU35P boards, and an ASIC feasibility estimate.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5:9b at 17.42 tokens/s on an Apple M4 Mac mini with 24GB unified memory.
Setup is Ollama 0.33.0 with Q4_K_M at 4096 tokens of context, single request, no KV-cache quantization reported.
Generation speed is the median of 3 trials (range 17.33-17.57 t/s), measured with a ~88-token prompt and output capped at 300 tokens; model load time 2.3s, first-content latency 2.9s cold and 0.64s warm.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5-35B-A3B at 49.3 t/s on an M4 Pro Mac mini with 64GB unified memory.
Setup is mlx_lm with 8-bit weights, single request, 512-token context and 512 output tokens.
Figure is transcribed from SiliconBench (arXiv:2609.19169v1, Table 4), a single-node baseline, not a multi-node result; the 512-token context is the tested workload, not a maximum. Quantization scheme, runtime version, GPU core count, KV-cache precision and sampling settings are not established for this row.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5-9B at approximately 27 tokens/second decode on an Apple Mac mini M6 with 32GB unified memory.
Setup is llama.cpp with Q4_K_M weights (5.5GB file) at empty prior context.
The figure is from a Tweakers review sweep of prior context from empty to 90,000 tokens in 10,000-token increments; the sweep endpoint is not a verified maximum context capacity. Prefill, KV cache, offload and speculation settings are not reported for this result.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen 3.5 35B at 49.7 tok/s on a Strix Halo 128GB machine.
Setup is llama.cpp via llama-swap with a HuggingFace GGUF Q8 quant.
The same model completed the coding benchmark under Ollama but produced off-spec output; under llama-swap it completed with a full Rails app. Other local models on the same machine: Qwen 3 Coder Next 72.9 tok/s, GPT OSS 20B 78.3 tok/s, Gemma 4 27B 47.6 tok/s, GLM 4.7 Flash 47.4 tok/s, Qwen 3.5 122B 23.1 tok/s, Llama 4 Scout 17.5 tok/s, Qwen 3 32B 11.7 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-27B affine 4bit prefilling a 4096-token prompt at 461 tok/s on an M5 Pro 48GB (20-core GPU) with stock mlx 0.32.0 and mlx_lm 0.31.3 in a clean venv.
The post is a bug report against oMLX 0.5.0, whose bundled MLX build fails a NAX metallib scan, so its M5 dispatch never engages: the same model prefilled at only 180 tok/s at 4096 tokens there, and 262 tok/s with OMLX_NAX=1. No decode figure is given beyond saying decode is normal.
Qwen3.5-35B-A3B 4bit prefilled at 1,798 tok/s in the clean venv against about 900 on oMLX 0.5.0.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codingagenticlong-context
User reports Qwen3.5-122B-A10B at UD-IQ4_XS decoding at 18.75 tok/s on 2x RTX 4090 with expert offload to a Threadripper TRX50 with 128 GB DDR5 (2 of 4 memory channels populated).
Setup is llama.cpp on Windows with q8_0 KV cache; the 122B took 11.91 s per agent tool call against 3.40 s for a 27B, so the author shelved it.
The rest of the post covers the 27B as a coding subagent: at 64k per agent, Q8_K_XL ran 3 agents (112 s median wait), Q6_K_XL 5 (168 s) and Q4_K_M 9 (330 s), with aggregate prefill near 1,500 tok/s whatever the slot count. Qwen3.6-35B-A3B decoded at 78 tok/s and Qwen3.8-Flash-Next topped out at 23.7 tok/s with experts in system RAM.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-4B at 16.24 t/s generation and 438.97 t/s prefill on an Apple M6 Mac mini with 32GB unified memory.
Setup is MLX 0.32.2 with MLX-LM 0.31.3, a controlled affine 4-bit (Q4) MLX package with group size 64 and 16-bit KV cache, single request, 2048 input tokens and 128 generated tokens.
Measurements are from Precisit's published benchmark, not an independent rerun; decode excludes first-token production and model loading, and the 2048-token input is not a configured context limit.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-35B-A3B at 109.7 tokens/s generation and 1770.1 tokens/s prefill on an Apple Mac Studio M3 Ultra with 96 GB unified memory.
Setup is MLX with 4-bit weights loaded via mlx_lm.load() and run through mlx_lm.generate.stream_generate() without an HTTP server, at a 512-token context setting, generating 50 tokens at temperature 0.0.
Source reports 20.20 GB peak memory and 77.4 MB KV memory for this row. 512 is the tested context setting, not a configured maximum; runtime version, GPU core count, KV-cache precision and maximum context are not stated. This is a short-context standalone result, not an end-to-end server throughput measurement.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5 9B Q4_K_M decoding at 7.89 t/s on an AMD RX 9070 XT under a pinned llama.cpp Vulkan build (b10964), while LM Studio's Vulkan llama.cpp build reaches 29.89-30.46 t/s on the same GPU and byte-identical GGUF.
Setup is llama.cpp with Q4_K_M and q8_0 KV cache at 32768 context, flash attention on, single RX 9070 XT; llama-bench -p 128 -n 64 -r 1 gives 451.45 pp128 t/s and 7.89 tg64 t/s at -ngl 99.
-ngl 0 (CPU) decodes at 8.00 t/s and -ngl 99 at 7.83-7.89 t/s, so GPU offload buys nothing; the iGPU alone gives 3.04 t/s. Through the managed runtime decode is 7.41-7.49 t/s with prompt eval 98-222 t/s. LM Studio loaded the model at 255744 context and was still about 4x faster. Root cause not established.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5-9B at 82.53 tokens/second generation on a Mac Studio M4 Max with 64GB unified memory.
Setup is vllm-mlx with a 4-bit MLX quant, measured by llama-benchy over 2048 prompt tokens and 32 generated tokens across three runs. A generation peak of 87.99 tokens/second and a TTFT of 2.41 seconds are also reported.
Exact quantization scheme, model artifact, runtime versions, configured context limit, concurrency, speculative decoding status and resident memory usage are not stated. The 64GB is system memory capacity, not measured memory use.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
creative-writing
User reports Qwen3.5-9B at 60.1 tokens/s median decode on an Apple Mac mini M4 Pro with 48 GB unified memory.
Setup is Rapid-MLX 0.15.6 with mlx-community/Qwen3.5-9B-4bit, greedy decoding, thinking off, MTP plus prompt lookup speculation, and a short-story task. Baseline mlx-lm 0.32.0 server with speculation off reaches 47.1 tokens/s.
Three-run medians with fresh server and isolated cache per run. The whole-file rename task reaches 198.4 tokens/s versus 46.4 for the baseline, but that peak is task-specific and not a general generation rate.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5-9B at 107.5 t/s generation on an Apple Mac Studio with M3 Ultra and 256 GB unified memory.
Setup is mlx-lm 0.31.3 with the mlx-community 4-bit MLX quant, thinking off and no speculative decoding; peak memory was 5.24 GB.
The figure comes from a 34-token request that generated 1,156 tokens for a 900-word essay; the author ran two passes and reports variation of about 1 t/s or less. Configured context length is not stated for this run.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-35B-A3B at 31.4 tok/s decode and 2001 tok/s prefill on a Jetson AGX Orin 64GB.
Setup is vLLM 0.17.0 with a pre-built SM 8.7 wheel enabling Marlin GPTQ tensor core kernels, GPTQ-Int4 weights, 4096 context, single sequence.
Decode holds 30.9 tok/s at ~4,000 tokens and 29.2 tok/s at ~20,000 tokens. Compared against llama.cpp Q4_K_M at 22.5 tok/s decode and 523 tok/s prefill, and vLLM GPTQ without Marlin at 241 tok/s prefill. End-to-end 20k context plus 200 output tokens takes 17s versus 47s on llama.cpp.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingtool-usevisionlong-contextagentic
User reports Qwen3.5-9B at 49 t/s generation and 48 t/s prompt processing on an Intel Arc A770 16GB.
Setup is llama.cpp b9521 (Vulkan) with Q6_K weights, q4_1 KV cache, 256K context, flash attention, vision mmproj and MTP speculative decoding on a single slot.
The same model under WSL2 with Q8_0 KV cache and 128K context reached about 40 t/s without vision. Qwopus3.5-4B-Coder hit 64 t/s generation and 100 t/s prompt at 96K context, and Gemma 4 12B reached 22 t/s generation and 74 t/s prompt at 128K context.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agenticsummarizationcreative-writinglong-context
User benchmarks Qwen3.5-35B-A3B (thinking disabled) on Apple Silicon, reporting effective throughput of 71.3 tok/s on the ops-agent scenario with an M3 Max 128GB (40 GPU) running oMLX at MLX 4-bit; generation speed was 90.8 tok/s.
Setup is oMLX with a tiered KV cache; the same M3 Max run gives 61.4 effective tok/s on doc-summary, 22.6 on prefill-test and 90.1 on creative-writing. Generation speed is similar across MLX engines (~55-93 tok/s depending on hardware) but prefill varies widely: at 8K context LM Studio MLX takes 49s to prefill while oMLX takes 1.7s with its persistent SSD cache from a prior run.
Effective tok/s is output tokens divided by total wall-clock time including prefill. On the same M3 Max, LM Studio MLX reaches 37.1 effective tok/s; on M1 Max 64GB (24 GPU) oMLX 4-bit reaches 37.5 and oMLX fp16 47.3.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5-35B-A3B at ~45 t/s single user on a Strix Halo (Radeon 8060S, gfx1151) with 90112 MiB unified memory visible to the GPU.
Setup is llama.cpp HIP build (commit 4d99d45) with Q4_K_M GGUF, 131072 context, -ngl 999, flash attention on, and -dio, running in a Proxmox LXC with ROCm 7.2.0.
Aggregate throughput reaches ~168 t/s at 16 parallel slots, where the user notes memory bandwidth becomes the limit.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 2-3 t/s generation on an RX 6800S 8GB laptop with 40GB RAM and Ryzen 9 6900HS, across Qwen3.5 9B, Qwen3.8 27B, gpt-oss 20B and Gemma 4 31B, regardless of model size or whether the model fits fully in VRAM.
Setup is Ollama (Vulkan and ROCm builds both tried) with LM Studio also tested, which reached the higher end of 3 t/s. Offloading layers to VRAM reportedly halves throughput, and Ollama sometimes falls back to full CPU load.
User is asking for help diagnosing the bottleneck; no single throughput figure is given, only a 2-3 t/s range.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3.5-9B BF16 at 41.4 tok/s decode on a single RTX 3090.
Setup is Nano-vLLM with BF16 weights, CUDA Graph decode, batch 1, greedy decoding, 24 GB card at 0.9 GPU memory utilization.
Decode is memory-bandwidth bound at 24.2 ms/token; eager mode gives about 17 tok/s and 4 concurrent requests give 138 tok/s aggregate. Prefill latency is 238 / 254 / 378 / 899 ms for 16 / 128 / 512 / 2048 tokens. The repo also reports Qwen3.6-27B-FP8 at about 41 tok/s on 4x RTX 4090 and Qwen3.5-9B at about 98 tok/s on 4x RTX 4090.