Qwen3.5 reports

prompt processing (prefill):

288.5 tokens/s

quant:

UD-Q6_K_XL (GGUF)

kv:

F16

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 122B-A10B at 288.5 t/s prompt processing on an AMD Ryzen AI MAX+ 395 with Radeon 8060S graphics. Setup is llama.cpp b9843 with UD-Q6_K_XL, f16 K/V cache and flash attention, one participating Radeon with ngl=999 and 16 CPU threads. The 288.5 t/s figure is the stock batch2048/ubatch512 arm at depth 512, measured as 288.503604 +/- 2.290730 t/s; the same arm falls to 119.312542 t/s at depth 100000. Generation throughput and TTFT were not measured.

generation:

80.9 tokens/s

prompt processing (prefill):

2779.0 tokens/s

quant:

MXFP4_MOE (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-35B-A3B at 80.9 t/s generation and 2779 t/s prompt processing on a Mac Studio M3 Ultra with 512 GB unified memory. Setup is llama.cpp b8783 with the MXFP4_MOE quantization, Metal backend, F16 K and V cache, and -ngl 999 offloading to one GPU. The figures come from llama-bench with -p 2048 -n 32 -ub 2048 -fa 1, five repetitions, batch 2048 and context depth 0; the source rates are 80.93 t/s and 2778.95 t/s.

generation:

48.4 tokens/s

prompt processing (prefill):

880.3 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-9B at 48.4 t/s generation and 880.3 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with Q8_0 GGUF, flash attention, F16 K/V, and 99 GPU layers requested on one Metal device. The headline is depth 0; at prefix depth 4096 generation is 47.519 t/s and at 16384 it is 45.097 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

78.8 tokens/s

prompt processing (prefill):

1567.5 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-4B at 78.8 t/s generation and 1567.5 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with Q8_0 GGUF, flash attention, F16 K/V, 99 GPU layers requested, single sequence, five repetitions. The headline is depth 0; at prefix depth 4096 generation is 76.142 t/s and at 16384 it is 70.660 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

68.0 tokens/s

prompt processing (prefill):

832.1 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-9B at 68 t/s generation and 832.1 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with the Q4_K_M GGUF, flash attention, F16 K/V cache, requested GPU layers 99, single sequence, five repetitions. The headline is depth 0; at prefix depth 4096 generation is 66.41 t/s and at 16384 it is 61.83 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

101.4 tokens/s

prompt processing (prefill):

1476.8 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-4B at 101.4 t/s generation and 1476.8 t/s prompt processing on an M4 Max 128GB. Setup is llama.cpp b10794 with Q4_K_M and F16 K/V cache, single sequence, 99 GPU layers requested, flash attention. The headline is depth 0; at prefix depth 4096 generation is 97.37 t/s and at 16384 it is 88.73 t/s. These are separate 512-token prefill and 128-token generation microbenchmarks, not an end-to-end conversation.

generation:

90.1 tokens/s

prompt processing (prefill):

2122.0 tokens/s

quant:

Q8_0 (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-0.8B at 90.1 t/s generation and 2122 t/s prompt processing on an M4 16GB Mac mini. Setup is llama.cpp 0.6.0 build 11429 with Q8_0 and Metal, flash attention enabled, single-stream. The same campaign's Homebrew llama.cpp 0.4.1 build 10964 baseline reached 87.3 t/s generation and 2049 t/s prompt processing; batched aggregate generation was 87.7 t/s at 1 stream and 322.4 t/s at 8 streams.

generation:

28.3 tokens/s

prompt processing (prefill):

352.4 tokens/s

quant:

Q4_K_M (GGUF)

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-4B at 28.3 t/s generation and 352.4 t/s prompt processing on a Mac mini M4 with 16 GB unified memory. Setup is llama.cpp 0.6.0 build 11429 with Metal, Q4_K_M GGUF, flash attention enabled, run via llama-bench with -p 512 -n 64 -d 0,4096 -r 3. The two figures are separate microbenchmarks, not one 512-input/64-output request; generation context depth is unknown, and PP512 after 4096 existing tokens is 321.5 t/s.

generation:

36.0 tokens/s

prompt processing (prefill):

1085.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 36 t/s generation and 1085 t/s prompt processing on one Strix Halo integrated GPU. Setup is llama.cpp Vulkan with Unsloth Dynamic UD-Q4_K_XL and 124 GB unified memory reported by the backend. The figures are separate llama-bench pp512 and tg128 phases, not a 512-token prompt followed by a 128-token chat, and total context capacity is unspecified.

generation:

60.0 tokens/s

prompt processing (prefill):

1992.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 4B at 60 t/s generation and 1992 t/s prompt processing on one Strix Halo integrated GPU. Setup is llama.cpp Vulkan with UD-Q4_K_XL, run as native llama-bench pp512 and tg128 phases. The figures are separate synthetic prompt-processing and token-generation phases, not a 512-token prompt followed by a 128-token chat, and total context capacity is unspecified.

generation:

88.9 tokens/s

prompt processing (prefill):

521.5 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 35B-A3B at 88.9 t/s generation and 521.5 t/s prompt processing on a single Apple M2 Max with 64 GB unified memory. Setup is MLX with 4-bit quantization, a requested 512-token prompt and 128-token generation cap at batch size 1, with no drafter passed. The figures are three-trial means after discarding Trial 1; the source means are 88.85 t/s generation and 521.47 t/s prompt, and peak memory was 20.218 in engine output units.

generation:

20.7 tokens/s

prompt processing (prefill):

91.4 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 27B at 20.7 t/s generation and 91.4 t/s prompt processing on an Apple M2 Max with 64GB unified memory. Setup is MLX with 4-bit quantization, batch size 1, a requested prompt length of 512 and generation cap of 128, run directly via python -m mlx_lm benchmark with no HTTP server. The figures are the mean of three retained trials after discarding Trial 1; the source reports 20.71 t/s TG and 91.36 t/s PP, and peak memory of 16.302 in engine output units.

generation:

62.7 tokens/s

prompt processing (prefill):

321.2 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 62.7 t/s generation and 321.2 t/s prompt processing on an Apple M2 Max with 64GB unified memory. Setup is MLX with 4-bit quantization, a requested prompt length of 512 and generation cap of 128, batch size 1, no HTTP server. The figures are three-trial means after discarding Trial 1; retained generation trials were 62.651, 62.747 and 62.627 t/s, and peak memory was 5.87 in engine output units.

generation:

81.1 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 81.1 t/s on a single Mac Studio with an M5 Max and 36 GB of unified memory. Setup is LM Studio llmster 0.0.25 with the MLX runtime 1.11.0, MLX 4-bit weights, and unknown context capacity, KV precision and Flash Attention. The figure is the source's rounded single-stream decode rate, client-observed streaming decode-phase throughput rather than engine-native timing; a separate four-stream aggregate result is 230.0 t/s.

generation:

78.0 tokens/s

quant:

4-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 78 t/s on a single Mac Studio with an M5 Max and 36 GB of unified memory. Setup is oMLX 0.7.0 with MLX 4-bit weights and an SSD cache directory enabled; configured context capacity, KV precision and speculative decoding are unknown. The figure is the source's rounded single-stream decode rate, client-observed streaming throughput rather than engine-native timing, and the source's homepage and prose give 77.9 t/s instead.

generation:

80.1 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 80.1 t/s on a single Mac Studio with an M5 Max and 36 GB of unified memory. Setup is Rapid-MLX 0.16.1 with MLX 4-bit and --no-spec-decode, in image-capable mode benchmarked with text prompts. The 80.1 t/s is the article's rounded single-stream decode rate, measured as client-observed streaming decode throughput rather than engine-native timing; a separate four-stream run reached 77.0 t/s as an aggregate wall-throughput figure.

generation:

115.5 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 115.5 t/s on a single Mac Studio with an M5 Max and 36 GB of unified memory. Setup is Rapid-MLX 0.16.1 with MLX 4-bit weights and MTP speculative decoding on the text-only lane, with image input disabled. The figure is the article's rounded single-stream decode rate, client-observed streaming decode-phase throughput rather than engine-native timing, and configured context capacity and KV precision are unknown.

generation:

54.4 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 54.4 t/s on one NVIDIA RTX 2060 Super with 8 GB of VRAM. Setup is llama.cpp with Q4_K_M and a context of 8192 tokens, all layers offloaded to the GPU. The figure is the median of three requests after a short warm-up, with roughly 1% agreement among repeats; the source reports 5.5 GB of graphics memory in use.

generation:

83.7 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5 4B at 83.7 t/s on one NVIDIA RTX 2060 Super with 8 GB of VRAM and an Intel Core i9-10900K host. Setup is llama.cpp server-cuda with Q4_K_M and a context of 8192 tokens, all layers forced onto the GPU. The figure is the median of three requests after a short warm-up, with roughly 1% agreement among repeats; the source reports 3.3 GB of graphics memory in use.

generation:

17.7 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 17.7 t/s on a Mac mini M4 with 16 GB unified memory. Setup is llama.cpp with Q4_K_M and a configured context of 8,192 tokens. The figure is the median of three requests after a short warm-up, with roughly 1% agreement among repeats; no prompt-processing rate, TTFT or end-to-end latency is supplied.