Qwen3.5 35B · 3B active at 4-bit generates a median of 52.4 tokens per second on the AMD Strix Halo 128GB, the hardware it is most run on, from 2 plain runs (community reports, as of 31 Oct 2026).
As of 31 Oct 2026, Qwen3.5 35B · 3B active at 4-bit on the hardware it is most run on, with the median of plain runs (one device, one request, no speculative decoding, the whole model in its memory):
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3.5 4B at 133 tok/s on an M5 Max 36GB, part of a 32-model comparison across four Macs.
Setup is Rapid-MLX with 4-bit weights and no speculative decoding, 512-token prompts generating 128 tokens, median of 5 runs after one warmup.
The same model reaches 63 tok/s on an M2 Pro 32GB, 79 on an M4 Pro 48GB and 159 on an M3 Ultra 256GB. On MoE models the M5 Max matches or beats the M3 Ultra, while the dense Qwen3.8 27B at 3.5bpw is faster on the Ultra (37 vs 28 tok/s). One 1024x1024 image with Qwen-Image 2.1 took 38 s on the M5 Max and 83 s on the M3 Ultra. The author works on Rapid-MLX and notes other runtimes will give different absolute numbers.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5 27B at IQ3_S prefill of 1049.13 t/s (pp512) on an RX 9070 XT with the Lemonade B1356 ROCm build, against 460.03 t/s on B1357.
Setup is llama.cpp with IQ3_S weights (9.72 GiB), q8_0 KV cache, flash attention on, 99 layers offloaded and 512 batch.
Prefill falls from 1049.13 to 460.03 t/s at pp512 between B1356 and B1357, and rebuilding llama.cpp over B1356 returns the slow result. No generation figures are given.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
tool-use
User asks how to enforce a closed JSON schema (constrained decoding) for Qwen3.5-4B running on a Snapdragon X Plus X1P64100 NPU with 15.6 GB RAM via GenieX 0.9.3 (geniex serve --compute npu, GGUF Q4_0, ggml-hexagon backend).
They report roughly 13 tok/s on the 4B model and want streaming, constraint only after the thinking block, and no Python or compiler on the end user's PC.
No benchmark table is given; the questions cover whether geniex serve honours json_schema or grammar, whether a llama.cpp Hexagon build supports --grammar/--json-schema on Windows ARM64, whether Genie SDK or ONNX Runtime GenAI expose a constraint, and possible logit-bias or logprob workarounds.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User tests Qwen3.5-4B Q4_K_M on an RTX 3080 10GB with llama.cpp built from upstream PR 29827, flash attention on and ubatch 64, on a RULER-style multikey needle task. Prefill at a 709,596-token prompt (704K, YaRN x2.75) was about 662 tok/s, 18.4 minutes per run, peaking at 9,784 MiB VRAM.
Stating the question before the document as well as after fixed all five configs that failed with question-last, at 256K and at 512K/704K with YaRN. One run per config, so the user calls it a strong signal on a small sample rather than a rate.
In 10 paired trials at 256K with question last, q8_0 KV cache passed 7/10 at ~7,730 MiB peak and q4_0 passed 6/10 at ~5,680 MiB; 9 of 10 pairs gave identical answers and McNemar exact p = 1.0. The user notes 114-170 MiB showed as shared memory, f16 cache does not fit at 256K on this card, and the model rambles after answering.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports running Qwen3.5 35B-A3B and Gemma 4 26B-A4B on an M5 16GB Mac at about 20-40 t/s, with 128k of context.
Setup uses oMLX, which the user tested after seeing results, and goose for the CLI.
The speed is given only as a range. The user asks why others recommend smaller models such as Gemma 4 E4B or Qwen3.5 9B Q6_K instead.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
vision
User reports running Qwen3.5 9B on an iQOO Z9x phone with 12GB RAM at about 1 tok/s, on the phone's CPU with no GPU.
No engine, quant or context length is stated.
User says it is useful offline and that Qwen3.5, even the 0.8B model, is pretty good at vision.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports serving Qwen3.5-9B at 35-40 t/s per single user on a server with 8x T4 GPUs and 128 GB DDR4 RDIMM.
Setup is llama.cpp with Q5_K_XL at 87k context per slot, one llama.cpp instance per T4, 44 slots total behind an nginx round-robin load balancer. TTFT is about 3-5 seconds depending on context size.
The user is asking for advice on fine-tuning the model with QLoRA on roughly 500 GB of raw data, and lists other hardware (2x V620, 2x MI50 32GB, 2x RTX 3060 12GB) that has boot problems in several motherboards.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a 9B Qwen3.5 model at 30 t/s on a single P106-100 6GB card after extreme undervolting.
No engine, quant or context length is stated for the run.
Speed dropped to 25 t/s after 4k tokens were generated in one go. User is considering buying 12 of these cards at $20 each and asks whether 70B+ models would hold at least 10 t/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User reports Qwen3.5-9b-reasoning at about 50 tokens/s on an RTX 2060 8GB, passed through to a Proxmox server with an i9-10900KF and 32 GB RAM.
Setup is Ollama, accessed from the terminal and through Open WebUI.
The user calls the speed satisfactory for light coding and technical questions, and asks whether another engine, model or terminal front end would get more out of the hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
visiontool-use
User reports Qwen3.5-4B at 10.61 t/s decode on a Rockchip RK3588 NPU with the open_rknpu2_lm_infer engine, using MTP speculative decoding.
Setup is a custom Rust/C++ runtime that avoids Rockchip's proprietary inference libraries, with an OpenAI-compatible API, streaming, tool calling and prefix caching; prefill of a 512-token prompt ran at 147.83 tok/s.
User compares against RKLLM 1.3.1, which reached 5.54 tok/s decode after 512 tokens and 58.35 tok/s prefill; decode after 3,800 tokens was 9.90 tok/s versus 4.57 tok/s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentic
User reports Qwen3.5-9B running at about 25 tok/s without speculation and 33-37 tok/s with MTP speculative decoding on a fanless MacBook Air M5 16GB.
Setup is a custom MLX engine (QVex-3.5) with a self-made qvex-q4 4-bit imatrix quant, using Qwen's own MTP head as the draft model (about 490 MB).
The custom quant is 2.7x closer to bf16 than stock MLX 4-bit (KL 0.026 vs 0.069). A 3-bit build was about 13% faster but used 20-45% more energy per token, and MTP costs 10-25% more energy per token. With a 512-token thinking budget the model solved 24/24 tasks; the user calls this a sanity check rather than a benchmark, tested on one machine only.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports 140 t/s writing code with a custom inference engine built from scratch for Qwen3.5-27B on an RTX 3090 24GB, with CPU offload.
Setup uses the unsloth Q4_K_M GGUF weights with additional 8-bit quantization of the KV cache and hidden states.
Coding with a 4K-token file in context drops to 95 t/s, and prefill is about 1,600 tok/s. The user compares these figures against llama.cpp with MTP on the same hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-use
User reports Qwen3.5-35B-A3B at roughly 115-120 tok/s on a Mac Studio M1 Ultra 128GB plus an RTX 3090 24GB over a Thunderbolt 5 eGPU.
Setup is llama.cpp with Q4_K_M, about 17GB of the model on the 3090 and about 5GB on the Mac, with the KV cache quantized to Q8.
The user says the Q8 KV cache lets them run about 200K context entirely in VRAM, and that this model replaced a two-model agentic setup.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User reports Qwen3.5-35B-A3B at ~980 t/s prompt processing and 75-80 t/s generation on a single RTX 3090 24GB.
Setup is llama.cpp with the RPC backend, IQ4_NL GGUF, flash attention on, 64K context per agent, and all layers offloaded to the GPU. The host is an AMD EPYC 7663 with 128GB DDR4 RAM.
Generation is given only as a 75-80 t/s range. Two instances run simultaneously (planner and coder).
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3.5-27B Q4_K_M on an RTX 5090 at 30.09 t/s generation and 1515.83 t/s prompt processing with llama-bench (pp1024/tg400, 5 reps).
Setup is llama.cpp b10741 with CUDA 13.3 on Windows 11, -ngl 99 and flash attention on, 15.65 GiB GGUF.
A Qwen3 MoE 30B-A3B control on the same rig reaches 134.18 t/s tg, ruling out a global Windows/driver overhead. The user notes native Linux with MTP reaches about 76% of roofline and ~1.7x speedup, while the Windows/Ollama path runs 1.5-1.6x slower at every draft depth. PCIe Gen5 x1 vs x4 gave 27.95 vs 30.09 t/s, so decode is not host-link-bound.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 62 t/s generation.
Source evidence (verbatim):
Source: https://digtvbg.com/blog/llama-server-vulkan-rdna4-vllm-rocm-benchmark/
# Local LLM Inference on AMD RX 9070 XT: Vulkan vs ROCm Benchmarks on RDNA4
Hardware: AMD RX 9070 XT with 16GB VRAM on CachyOS. Model: Qwen3.5-9B-UD-Q6_K_XL (about 6.5 bits per weight). llama-server from llama.cpp built against Vulkan with --n-gpu-layers 999 -c 65536 --batch-size 2048 --ubatch-size 2048 --parallel 1: 62 tokens/second. vLLM on ROCm 7.2: 48 tokens/second. Same prompt, max tokens, temperature 0 and seed 42 for both; 10 measured runs per backend after 2 warmup runs. No speculative decoding is configured.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 77.8 t/s generation.
Source evidence (verbatim):
Source: https://terminalbytes.com/ollama-vs-llama-cpp-vs-mlx-mac-2026/
# Ollama vs llama.cpp vs MLX on a Mac Studio, measured
Hardware: Mac Studio with an M3 Ultra and 256GB unified memory. The author ran Qwen3.5 9B through every runtime with the same 34-token prompt, thinking off, one request at a time, two runs each, no speculative decoding. Results, generation speed: llama.cpp llama-bench build 10809 with unsloth Qwen3.5-9B Q4_K_M GGUF, tg128 77.8 tok/s; Ollama 0.32.13 GGUF qwen3.5:9b Q4_K_M 77.7 tok/s; Ollama 0.32.13 MLX NVFP4 90.5 tok/s; rapid-mlx 0.14.1 Qwen3.5-9B 4-bit 105.9 tok/s; mlx-lm 0.31.3 Qwen3.5-9B 4-bit 107.5 tok/s. The author wrote down the numbers from his own runs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3.5-35B-A3B at 73.43 tokens/s generation and 1808 tokens/s prefill on an Apple M3 Ultra 96GB.
Setup is llama.cpp build 8860 with Q8_0 GGUF weights and f16 KV cache, Metal backend, single machine.
The figures are separate synthetic tests (TG128 and PP512), not an end-to-end request. Historical TG128 85.48, turbo3 KV 63.23 and PP8192 1183 belong to other configurations and are kept for comparison only.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports about 40 tok/s on Qwen3.5 35B running CPU-only on a 16-core AMD Ryzen 9 9950X with 128GB of RAM.
No GPU or serving engine is named for the run.
The user is considering a Threadripper build but objects to ECC RAM pricing, citing $7,500 for 256GB of ECC RAM against $4k+ each for two 64-core 9980X CPUs, and compares a hypothetical $6,000 64-core system to a Mac Studio M5 Ultra with 256GB RAM at $10k.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 43.5 t/s generation.
Source evidence (verbatim):
Source: https://antekapetanovic.com/blog/qwen3.5-apple-silicon-benchmark/
# Ollama vs. llama.cpp vs. MLX with Qwen3.5 35B on Apple Silicon
Apple M4 Max, 128GB unified memory, macOS 26.3. Model Qwen3.5 35B-A3B (MoE). Ollama Q4_K_M, llama.cpp Unsloth Dynamic Q4_K_XL, context 65536 for both, MLX 4-bit group quantization. Engine versions not stated. Generation only, mean of 10 runs, single request, no speculative decoding stated. Engine-reported generation tok/s, thinking off, deterministic params: mlx-py 130.2 +/- 0.8, llama.cpp 71.0 +/- 0.1, Ollama 43.5 +/- 0.4. Client wall-clock: mlx-py 114.9, mlx-http 107.6, llama.cpp 65.8, Ollama 40.5. Prompt processing was not reported.
Individual observations from this page, not expected speeds or a ranking. Select a model to open its report, with the source and offloading conditions. Reported context may be a limit rather than actual input length.
The hardware Qwen3.5 35B · 3B active at 4-bit is most run on: AMD Strix Halo 128GB (52.4 t/s median, 2 runs); AMD Radeon 890M (14.9 t/s median, one run); AMD Radeon AI PRO R9700 32GB (149.2 t/s median, one run). Medians of plain runs, as of 31 Oct 2026.
Which inference engines do people use for Qwen3.5?
Reports of Qwen3.5 name llama.cpp (54 reports), MLX (8 reports), Ollama (7 reports), oMLX (3 reports), as of 31 Oct 2026.
Speed by hardware on Qwen3.5 35B · 3B active at 4-bit
The size and quant level the most devices have plain runs of: one device, one request, no speculative decoding, and the whole model in the device's memory. Every speed in this table is the same model.