Qwen3.5 reports

generation:

29.7 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5 4B at 29.7 t/s on a Mac mini M4 with 16 GB unified memory. Setup is llama.cpp with Q4_K_M and a configured context of 8,192 tokens; KV cache precision is not stated. The figure is the median of three requests after a short warm-up, with roughly 1% agreement among repeats; no prompt-processing rate or TTFT is given.

generation:

65.5 tokens/s

prompt processing (prefill):

305.9 tokens/s

quant:

Q4_K_M (GGUF)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 65.5 t/s generation and 305.9 t/s prompt processing on an M4 Max 64GB. Setup is llama.cpp with Q4_K_M, 4096 context, flash attention on, and prompt caching disabled. The headline Quick Chat run used 45 prompt tokens and 96 output tokens, with three measured samples per scenario; these are short inputs, not long-context benchmarks.

generation:

67.1 tokens/s

prompt processing (prefill):

546.3 tokens/s

quant:

Q4_K_M (GGUF)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 67.1 t/s generation and 546.3 t/s prompt processing on an M4 Max 64GB. Setup is ZINC with Metal, Q4_K_M, speculative decoding disabled, and an auto-sized context; the run used 48 prompt tokens and 96 output tokens. The stored 67.1 t/s is the median of three samples, with the source's own median listed as 67.05 t/s over a 67 to 67.34 t/s range.

generation:

86.5 tokens/s

prompt processing (prefill):

2369.3 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 86.5 t/s generation and 2369.3 t/s prompt processing on an M5 Max with 48GB unified memory. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

135.6 tokens/s

prompt processing (prefill):

4099.7 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 4B at 135.6 t/s generation and 4099.7 t/s prompt processing on an M5 Max 48GB. Setup is llama.cpp build 11146 with Metal, Q4_K_M weights, n_gpu_layers=99 and flash_attention=true. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

111.2 tokens/s

prompt processing (prefill):

3378.3 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-35B-A3B at 111.2 t/s generation and 3378.3 t/s prompt processing on a single Apple M5 Max with 48 GB unified memory. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, n_gpu_layers=99 and flash attention enabled. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput; context length is unspecified.

generation:

29.2 tokens/s

prompt processing (prefill):

730.5 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-27B at 29.2 t/s generation and 730.5 t/s prompt processing on a single Apple M5 Max with 48 GB unified memory. Setup is llama.cpp build 11146 with Metal, GGUF Q4_K_M weights, flash attention enabled and n_gpu_layers=99; the context length is unspecified. The figures are separate llama-bench microtests, tg128 and pp512, not one 512-input/128-output request or server concurrency throughput.

generation:

74.7 tokens/s

prompt processing (prefill):

2710.5 tokens/s

quant:

Q4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 122B-A10B at 74.7 t/s generation and 2710.5 t/s prompt processing on an M5 Ultra 256GB. Setup is MLX with a 4-bit quant, batch 1 on one device, at a 32768-token input and 128-token output. The figures are separate five-trial medians, not pooled across lengths; the same run peaked at 72.259GB of allocator high-water memory.

generation:

49.5 tokens/s

prompt processing (prefill):

784.5 tokens/s

quant:

Q4 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 122B-A10B at 49.5 t/s generation and 784.5 t/s prompt processing on an M3 Ultra 512GB. Setup is MLX with a 4-bit quant, batch 1 on one device, at a 32768-token input and 128-token output. The figures are separate five-trial medians, not pooled across lengths; the same run peaked at 76.32GB of allocator high-water memory.

generation:

30.3 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 30.3 t/s on an Apple Mac mini M6 with 32 GB unified memory. Setup is LM Studio with the MLX backend and a loaded context of 165,120 tokens, one machine with one model loaded alone. The 30.3 t/s is the median of three English-prose runs, a client-observed streaming decode rate rather than a pooled or peak figure; exact quantization is unknown.

generation:

112.9 tokens/s

prompt processing (prefill):

1234.5 tokens/s

quant:

Q6_K_L (GGUF)

kv:

F16

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 122B-A10B at 112.9 t/s generation and 1234.5 t/s prompt processing on one RTX PRO 5000 Blackwell 72 GB plus one RTX PRO 5000 Blackwell 48 GB. Setup is llama.cpp CUDA b10941 with Q6_K_L, f16 K/V cache, flash attention on, -ngl 999, batch 2048 and microbatch 512, split tensor 3/2 across the two cards. The 112.9 t/s is TG128 and the 1234.5 t/s is PP512, separate tests rather than one combined request, one repetition per cell; the same run's PP2048 and PP8192 reached 2092.1 t/s and 2262.2 t/s.

generation:

86.6 tokens/s

quant:

BF16 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks SGLang 0.5.17, vLLM 0.26.0 and llama.cpp b10333 on a 128GB GMKtec EVO-X3 with Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151). On Qwen3.5-0.8B BF16, custom vLLM reached 86.59 tok/s at one client and 311.59 tok/s aggregate at eight clients, beating SGLang and llama.cpp at every concurrency. Setup used ROCm 7.14, 4096-token API limit, deterministic 598-638 token prompts with 64 output tokens, prefix caching disabled. vLLM won all 20 matched concurrency points against SGLang across Qwen3.5-0.8B, Qwen3-8B, Qwen3-Coder-30B-A3B and a shared-prefix workload. For Qwen3.5-122B-A10B, custom llama.cpp with UD-Q4_K_XL reached 13.57 tok/s at one client and official llama.cpp 17.88 tok/s aggregate at two clients. DeepSeek V4 Flash on a specialised Lucebox ROCmFP3 service reached 23.70 tok/s median.

generation:

37.7 tokens/s

prompt processing (prefill):

1828.8 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 37.7 t/s generation and 1828.8 t/s prompt processing on an NVIDIA RTX 4060 Laptop 8GB. Setup is llama.cpp with Q4_K_M and full GPU offload via -ngl 99, run through native llama-bench CUDA on a Lenovo LOQ 15IRX9 with 24GB system RAM. The figures come from separate synthetic tests, pp512 and tg128, not a single 512-token request followed by a 128-token completion; context allocation and KV precision are not given.

generation:

15.6 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5-122B-A10B at 15.6 t/s generation on an M4 Max 128GB. Setup is llama.cpp with Q4_K_M and f16 K/V cache, all layers offloaded, batch 1024 and flash attention on. The figure is the 128K benchmark point after cache prefilling, not newly processed input tokens; the source's f16 KV series is 24.51 t/s at 64K, 22.25 at 81K and 15.59 at 128K, with a reported Metal/VRAM peak of 75.16 GB at 128K.

generation:

33.0 tokens/s

prompt processing (prefill):

386.4 tokens/s

quant:

4bit MLX (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 397B-A17B at 33 t/s generation and 386.4 t/s prompt processing on a single Mac Studio with an M3 Ultra and 512 GB unified memory. Setup is mlx-vlm with 4bit MLX, run at long context with 34,448 actual input tokens and 128 generated tokens per run, three runs, peak memory 234.9 GB. The figures are medians; prompt processing p10/p90 are 359.4 and 428.1 t/s, and median wall time was 105 seconds. KV quantization, flash attention and speculative decoding are not exposed, and cold-cache prefill is not established.

generation:

67.7 tokens/s

prompt processing (prefill):

2886.0 tokens/s

quant:

Q8_0

kv:

Q8

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 2B at 67.7 t/s generation and 2886 t/s prompt processing on an AMD Strix Halo 128GB box with a Radeon 8060S. Setup is llama.cpp on the rdna35-perf branch with Q8_0 weights, q8_0 KV cache, Flash Attention on and microbatch 2048, measured with llama-bench. The two figures are separate benchmark cells at a source-labeled 128k tokens already in the KV cache, not one end-to-end request; the prompt test processes 2048 additional tokens at that depth and is not TTFT.

generation:

77.8 tokens/s

prompt processing (prefill):

1010.3 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 77.8 t/s generation and 1010.3 t/s prompt processing on an M3 Ultra 256GB. Setup is llama.cpp with Q4_K_M, run through native llama-bench with -p 512 -n 128 -r 3. The 77.83 ± 0.11 t/s and 1010.32 ± 1.28 tokens/s are native benchmark rates for 512/128 test tokens, not configured maximum context; KV precision, flash attention and GPU-offload count are unreported.

generation:

58.9 tokens/s

prompt processing (prefill):

2317.5 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks a broad set of local models on a dual RTX 2000 Ada 16GB setup (32GB total VRAM) with 64GB system RAM, running Ollama in Docker on Debian 13. The headline result is Qwen3.5 35B-A3B Q4_K_M at 58.9 tok/s generation and 2317.5 tok/s prefill on a 16K-token prompt, though the user notes a reproducible CUDA illegal-memory-access crash with Flash Attention enabled on one prompt shape. Setup is Ollama with GGUF quants, 16K context for the headline run, and no KV cache quant stated. The user also reports a 100-task quality harness (250 points) and a separate speed harness with 512, 2K and 16K prompts. Other models tested include Gemma 4 26B-A4B Q4 at 42.5 tok/s, GPT-OSS 20B at 46.9 tok/s, Qwen3 30B-A3B IQ3 at 53.4 tok/s, and LFM2 24B-A2B Q4 at 102 tok/s. Dense models were slower: Mistral Small 3.2 24B Q4 at 12.6 tok/s and Qwen3.8 27B Q6_K_M at 8.8 tok/s. The user kept seven models and deleted the rest.

generation:

25-40 tokens/s

quant:

Q5_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User asks how to serve Qwen3.5-9B to 10-15 concurrent users on an Asus ESC4000 G3 with 8x T4 16GB and 128 GB DDR4 RAM. Currently runs 8 separate llama.cpp instances serving Qwen3.5-9B-Q5_K_XL at 87k context per slot with 44 slots, each slot generating 25-40 t/s and prefill at 1400 t/s. User is concerned about latency and asks whether vLLM or another engine would be faster, noting the T4s lack FP8 support and that vLLM with AWQ-INT8 was fast on their MI50 32GB.

generation:

19.2 tokens/s

quant:

UD-Q6_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen3.5 122B A10B at 19.17 t/s generation on a Strix Halo 128GB. Setup is llama.cpp with UD-Q6_K_XL GGUF, ROCm 7.2.3, full GPU offload, no MTP speculation. The model is 106.04 GiB. With MTP enabled the same model reaches 26.86 t/s. Prompt processing peaks at 266.48 t/s with batch size 512. A dense Gemma 4 31B Q4_K_M comparison runs at 10.48 t/s.