Qwen3.5 reports

generation:

29.2 tokens/s

quant:

6-bit (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 29.2 t/s generation. Source evidence (verbatim): Source: https://alooftwaffle.substack.com/p/96-hours-with-dual-dgx-sparks-and # First-hand Mac Studio M3 Ultra 512GB Qwen3.5-397B-A17B 6-bit mlx-vlm generation measurement Original first-hand report by AlooftWaffle, published March 27, 2026: https://alooftwaffle.substack.com/p/96-hours-with-dual-dgx-sparks-and . Historical source-reported measurements, not a new benchmark or independent reproduction. Selected configuration: one Mac Studio M3 Ultra with 512GB unified memory, running Qwen3.5-397B-A17B at publisher-described 6-bit precision using mlx-vlm. The shortest reported context row gives generation 29.2 tokens/s. The source labels this row 4K context; whether that label denotes actual occupied input or configured capacity is not established. Actual fresh input count, cached prefix, output count and runtime version are unknown. Leave structured context length unknown. Do not replace it with 4096 or infer zero cache. CPU/GPU core counts, sampling, thinking mode, speculative decoding, request concurrency, warmup, repeats, statistical aggregation, exact measurement date, OS, KV precision, Flash Attention, offload, temperatures, power and peak resident memory are unreported. The source's longer Mac rows report generation 28.6 tokens/s at the 8K label and 26.7 tokens/s at 32K; preserve these as separate source rows, not an average. The author also calls values 317/537/476 tokens/s 'prefill throughput (time to first token)'; native timing boundaries and counters are absent, so leave structured prompt throughput and TTFT unknown rather than converting or equating these metrics. Verified upstream model-family reference: https://huggingface.co/Qwen/Qwen3.5-397B-A17B . This is a base/reference repository, not the tested 6-bit artifact. The tested quantization repository, revision, file names, hashes and quantization recipe are unknown; do not assign a community pack or GGUF Q6 code. The source describes nearly 300GB of downloaded model files; that is download/storage volume, not measured resident RAM or a peak-memory measurement. The same article's dual-Spark INT4-AutoRound/vLLM measurements and embedding sentences/s are different hardware, quantization and workloads; none is the selected Mac generation speed or a controlled comparison isolating hardware. Full original source remains the URL above; no author contact or comments were made.

generation:

16.0 tokens/s

quant:

IQ1_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports about 16 tok/s running a 27B Qwen3.5 reasoning-distilled model at IQ1_S on an RTX 2070 8GB. Setup is Ollama with Open WebUI, an 8192-token context window, and an Intel Core i5-13600 CPU. The user notes the model can solve river puzzles but fails to count the vowels in "BANANA", and describes the setup as a desperate measure for an 8GB GPU.

generation:

69.8 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 69.8 t/s generation. Source evidence (verbatim): Source: https://www.fitmyllm.com/reports/rtx-3070-8gb # FitMyLLM original RTX3070 single-stream Qwen3.5 9B Q4_K_M measurement FitMyLLM first-party public report preview: measured on one rented NVIDIA GeForce RTX 3070 with 8GB VRAM, run finished 2026-08-29. Selected row Qwen3.5 9B Q4_K_M native decode 69.8 tokens/s, explicitly single stream (concurrent requests 1). llama.cpp build 91f8c9c5fb038c086e13e9cd823c29b33b07ba54, driver 595.71.05. The preview identifies seven successful models and two failed models, but exposes only five decode rates. Actual prompt/context depth, input/output token counts, repetitions, OS/CPU/core counts/system RAM, prefill rate, KVprecision and speculative-decoding settings are not disclosed for this selected row. Do not assign4K/8K/16K/32K depth sweep labels or1–8stream load tests from the paid report contents to this preview row. No exact tested HF repository, file checksum or model revision is linked; do not invent one or substitute a base-model reference. The runtime commit is a llama.cpp build revision, not a model revision. Original public source https://www.fitmyllm.com/reports/rtx-3070-8gb . Public methodology https://www.fitmyllm.com/methodology distinguishes first-party rented-hardware measurements from fitted estimates and describes headline rates as medians of independent runs; the specific repetition count and spread are not exposed in the preview. The methodology says raw JSON is freely accessible, while this report says its JSON requires purchasing the report; no raw API or paid material was accessed. Other public preview rows, separate deployments: Qwen3 4B Instruct2507 Q4_K_M 126; Llama3.1 8B Instruct Q4_K_M 80.9; Qwen3 8B Q4_K_M 78.4; Qwen3 8B Q3_K_M 73.3 tokens/s. These are not repetitions or ranges to average with the selected Qwen3.5 row.

generation:

15.3 tokens/s

quant:

UD-IQ3_XXS (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.5-397B at 15.3 t/s across two DGX Sparks (128 GB unified each) over llama.cpp RPC, a model that does not fit on one node. Setup is a forked Ollama 0.40.3 with llama.cpp b11351, UD-IQ3_XXS quant (141 GB), 256 output tokens, warm run, two nodes linked by a 200 Gb/s cable. Splitting across nodes costs speed: fast small models lose 30-60% (gemma4:e2b 106.6 to 44.1 t/s), large models lose 15-20%, and load time is 3-5x longer. The default policy keeps a model on one machine and only splits when it does not fit.

prompt processing (prefill):

16459.6 tokens/s

quant:

Q8_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks a GDN chunked kernel pull request in llama.cpp, reporting prompt-processing throughput on several GPUs and models. On an RTX PRO 6000 Blackwell with Qwen3.5-35B-A3B-Q8_0 at 2048 prompt tokens, the PR reaches 16459.56 tok/s against 13653.58 tok/s on master, a 20.6% gain. The table also covers RTX 5090, RTX 4090, RTX 3090 and Spark GB10 across Qwen3.5-35B-A3B, Qwen3.6-35B-A3B, Qwen3.8-27B and Qwen3.8-Flash-Next at 2048 and 4096 prompt tokens.

generation:

198.4 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 198.4 t/s generation. Source evidence (verbatim): Source: https://rapidmlx.com/compare/mlx-lm # Mac mini M4 Pro48GB Qwen3.5-9B speculative whole-file edit benchmark First-party engine-author measurements October4,2026. Mac mini AppleM4 Pro 48GB unified RAM, macOS26.5.1, ACpower, idle before starts; core counts unknown. Qwen3.5-9B4-bit, Rapid-MLX0.15.6 with MTP plus prompt-lookup versus mlx-lm0.32.0 default nonspeculative server, both MLX0.32.3. Whole-file rename task: 198.4 tok/s decode. Comparison baseline: 46.4 tok/s; median of3fresh-server runs, two discarded warmups, temperature0, thinkingoff. Exact prompt/output counts, contextcapacity, residentRAM and MTPcompanion artifact unknown. Decode=(completiontokens-1)/(last-firststreamedchunk), not end-to-end or prefill. Copy-heavy task benefits from prompt lookup; do not generalize peak. Testedquant repo verified: https://huggingface.co/mlx-community/Qwen3.5-9B-4bit . Sourceversions.json identifies shortened revision8b2b98c; fullSHA unknown and revisiontree read returned generic toolerror. Sources: https://rapidmlx.com/compare/mlx-lm ; https://rapidmlx.com/blog/assets/vs-mlx-2026-10/versions.json ; https://rapidmlx.com/blog/assets/vs-mlx-2026-10/raw/manifest.json . Source author builds tested engine; rawmanifest links individual runs. Selected run is Rapid-MLX with MTP plus prompt-lookup, generation 198.4 tok/s. Baseline mlx-lm measurement is a separate run. Prefill speed unknown.

generation:

22.6 tokens/s

prompt processing (prefill):

247.2 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 122B-A10B at 22.6 t/s generation and 247.2 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama-bench build b8171 with the Vulkan backend, UD-Q4_K_XL quant, ngl 99, and a Radeon 8060S Graphics (RADV GFX1151) device. The model is the 122B-A10B artifact at 63.65 GiB; PP512 is 512 benchmark prompt tokens and TG128 is 128 evaluation steps, so the two rates are not a same-request pair.

generation:

31.8 tokens/s

prompt processing (prefill):

747.7 tokens/s

quant:

UD-Q8_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 35B-A3B at 31.8 t/s generation and 747.7 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama-bench build b8171 with the Vulkan backend, UD-Q8_K_XL quant, ngl 99, and a Radeon 8060S Graphics (RADV GFX1151) device. The rates are separate synthetic phases: PP512 at 747.72 +/- 44.81 t/s and TG128 at 31.83 +/- 0.03 t/s, not a same-request prompt/output pair.

generation:

53.4 tokens/s

prompt processing (prefill):

865.7 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 35B-A3B at 53.4 t/s generation and 865.7 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama-bench build b8171 with the Vulkan backend, UD-Q4_K_XL quant, ngl 99, and a Radeon 8060S Graphics (RADV GFX1151) device. The rates are separate synthetic phases: PP512 at 865.72 +/- 59.59 t/s and TG128 at 53.39 +/- 0.08 t/s, not a same-request prompt/output pair.

generation:

10.7 tokens/s

prompt processing (prefill):

220.4 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 27B at 10.7 t/s generation and 220.4 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is the b8171 Vulkan build with the UD-Q4_K_XL GGUF, 26.90 B parameters and 16.40 GiB, ngl 99, on a Radeon 8060S (RADV GFX1151). The native figures are PP512 220.35 +/- 3.36 t/s and TG128 10.66 +/- 0.01 t/s, separate synthetic phases rather than a same-request pair; batch, threads, context depth and KV types are unreported.

generation:

31.1 tokens/s

prompt processing (prefill):

819.2 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5 9B at 31.1 t/s generation and 819.2 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama-bench build b8171 with the Vulkan backend, UD-Q4_K_XL quant, ngl 99, and a Radeon 8060S (RADV GFX1151) device. The rates are separate synthetic phases: PP512 at 819.24 +/- 55.72 t/s and TG128 at 31.09 +/- 0.05 t/s, not a same-request prompt/output pair.

generation:

28.0 tokens/s

prompt processing (prefill):

1259.1 tokens/s

quant:

UD-Q8_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-4B-UD-Q8_K_XL at 28 t/s generation and 1259.1 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp b8171 with the Vulkan backend, ngl 99, and the UD-Q8_K_XL quant; the native table lists type Q8_0, 4.21 B parameters and 5.53 GiB. The rates are separate synthetic phases, PP512 at 1259.14 +/- 3.82 t/s and TG128 at 27.95 +/- 0.07 t/s, not a same-request prompt/output pair.

generation:

48.5 tokens/s

prompt processing (prefill):

1388.9 tokens/s

quant:

UD-Q4_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-4B-UD-Q4_K_XL at 48.5 t/s generation and 1388.9 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama-bench build b8171 with the Vulkan backend, ngl 99, and the UD-Q4_K_XL quant, on a Radeon 8060S (RADV GFX1151) iGPU. The stored rates are rounded to 0.1 t/s from native PP512 1388.87 +/- 10.68 t/s and TG128 48.53 +/- 0.65 t/s, separate synthetic phases rather than a same-request pair.

generation:

19.9 tokens/s

prompt processing (prefill):

270.1 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-122B-A10B at 19.9 t/s generation and 270.1 t/s prompt processing on a Radeon 8060S plus an RTX 5070 Ti over OCuLink. Setup is llama.cpp Vulkan with UD-Q4_K_XL, full offload, flash attention, and a 95/5 Radeon-to-RTX tensor split. The headline is the first completed interior split; the same sweep reached 21.79 t/s generation at 80/20, and the all-APU 100/0 endpoint gave 19.46 t/s.

generation:

12.4 tokens/s

prompt processing (prefill):

281.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-27B at 12.4 t/s generation and 281 t/s prompt processing on a Minisforum MS-S1 Max with Radeon 8060S and an RTX 5070 Ti over OCuLink. Setup is llama-bench with UD-Q4_K_XL, -ngl 99 -fa 1, and a 9/1 Radeon-to-RTX tensor split, PP512 and TG128 phases. The 9/1 split measured 12.43 +/- 0.03 t/s TG128 and 280.95 +/- 10.11 t/s PP512; the all-APU 10/0 endpoint gave 11.89 t/s, and 0/10 failed with ErrorOutOfDeviceMemory.

generation:

22.4 tokens/s

prompt processing (prefill):

287.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-122B-A10B at 22.4 t/s generation and 287 t/s prompt processing on a single Radeon 8060S iGPU in an AMD Strix Halo system with 128 GB unified memory. Setup is llama.cpp with the Unsloth UD-Q4_K_XL quant, full offload, batch 2048 and flash attention, measured via llama-bench pp512 and tg128. The 287 t/s and 22.4 t/s are separate 512-token prompt and 128-token generation microbenchmarks, not one end-to-end request; the 72 GiB figure is artifact disk size, not measured resident memory.

generation:

12.1 tokens/s

prompt processing (prefill):

310.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-27B at 12.1 t/s generation and 310 t/s prompt processing on an AMD Strix Halo 128GB system with a single Radeon 8060S iGPU. Setup is llama.cpp with UD-Q4_K_XL, full offload, batch 2048 and flash attention, using the RADV Vulkan backend. The figures are separate 512-token prompt and 128-token generation microbenchmarks, not one end-to-end request; the AMDVLK backend gave 86.0 t/s prompt and 11.9 t/s generation.

generation:

35.7 tokens/s

prompt processing (prefill):

972.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-9B at 35.7 t/s generation and 972 t/s prompt processing on a single Radeon 8060S iGPU in an AMD Strix Halo system with 128 GB unified memory. Setup is llama.cpp with UD-Q4_K_XL, full offload (ngl 999), batch 2048, 32 threads and flash attention, using the RADV Vulkan backend. The figures are separate llama-bench pp512 and tg128 microbenchmarks, not one end-to-end request; the AMDVLK backend gave 289.0 t/s prompt and 35.2 t/s generation.

generation:

59.8 tokens/s

prompt processing (prefill):

1017.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-35B-A3B at 59.8 t/s generation and 1017 t/s prompt processing on a single Radeon 8060S iGPU in an AMD Strix Halo system with 128 GB unified memory. Setup is llama.cpp via RADV with the Unsloth UD-Q4_K_XL quant, full offload, batch 2048, flash attention and 32 threads. The figures are separate llama-bench pp512 and tg128 microbenchmarks, not one end-to-end request; the same campaign lists AMDVLK at 686.0 t/s prompt and 60.4 t/s generation.

generation:

37.9 tokens/s

prompt processing (prefill):

1375.0 tokens/s

quant:

Q8_0 (GGUF)

flash attention:

on

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen3.5-4B at 37.9 t/s generation and 1375 t/s prompt processing on a single Radeon 8060S iGPU in an AMD Strix Halo system with 128 GB unified memory. Setup is llama.cpp with Q8_0 and flash attention, full offload, batch 2048 and 32 threads; the source labels the run RADV Vulkan. These are separate 512-token prompt and 128-token generation microbenchmarks, not one end-to-end request. The same campaign reports AMDVLK at 510.0 t/s prompt and 40.0 t/s generation.