Local model performance on your hardware

generation:

133.0 tokens/s

quant:

4-bit (MLX)

User benchmarks Qwen3.5 4B at 133 tok/s on an M5 Max 36GB, part of a 32-model comparison across four Macs. Setup is Rapid-MLX with 4-bit weights and no speculative decoding, 512-token prompts generating 128 tokens, median of 5 runs after one warmup. The same model reaches 63 tok/s on an M2 Pro 32GB, 79 on an M4 Pro 48GB and 159 on an M3 Ultra 256GB. On MoE models the M5 Max matches or beats the M3 Ultra, while the dense Qwen3.8 27B at 3.5bpw is faster on the Ultra (37 vs 28 tok/s). One 1024x1024 image with Qwen-Image 2.1 took 38 s on the M5 Max and 83 s on the M3 Ultra. The author works on Rapid-MLX and notes other runtimes will give different absolute numbers.

prompt processing (prefill):

1049.1 tokens/s

quant:

IQ3_S (GGUF)

kv:

q8_0

User reports Qwen3.5 27B at IQ3_S prefill of 1049.13 t/s (pp512) on an RX 9070 XT with the Lemonade B1356 ROCm build, against 460.03 t/s on B1357. Setup is llama.cpp with IQ3_S weights (9.72 GiB), q8_0 KV cache, flash attention on, 99 layers offloaded and 512 batch. Prefill falls from 1049.13 to 460.03 t/s at pp512 between B1356 and B1357, and rebuilding llama.cpp over B1356 returns the slow result. No generation figures are given.

prompt processing (prefill):

662.0 tokens/s

quant:

Q4_K_M (GGUF)

User tests Qwen3.5-4B Q4_K_M on an RTX 3080 10GB with llama.cpp built from upstream PR 29827, flash attention on and ubatch 64, on a RULER-style multikey needle task. Prefill at a 709,596-token prompt (704K, YaRN x2.75) was about 662 tok/s, 18.4 minutes per run, peaking at 9,784 MiB VRAM. Stating the question before the document as well as after fixed all five configs that failed with question-last, at 256K and at 512K/704K with YaRN. One run per config, so the user calls it a strong signal on a small sample rather than a rate. In 10 paired trials at 256K with question last, q8_0 KV cache passed 7/10 at ~7,730 MiB peak and q4_0 passed 6/10 at ~5,680 MiB; 9 of 10 pairs gave identical answers and McNemar exact p = 1.0. The user notes 114-170 MiB showed as shared memory, f16 cache does not fit at 256K on this card, and the model rambles after answering.

prompt processing (prefill):

1600.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

8-bit

User reports 140 t/s writing code with a custom inference engine built from scratch for Qwen3.5-27B on an RTX 3090 24GB, with CPU offload. Setup uses the unsloth Q4_K_M GGUF weights with additional 8-bit quantization of the KV cache and hidden states. Coding with a 4K-token file in context drops to 95 t/s, and prefill is about 1,600 tok/s. The user compares these figures against llama.cpp with MTP on the same hardware.

generation:

75-80 tokens/s

prompt processing (prefill):

980.0 tokens/s

quant:

IQ4_NL (GGUF)

User reports Qwen3.5-35B-A3B at ~980 t/s prompt processing and 75-80 t/s generation on a single RTX 3090 24GB. Setup is llama.cpp with the RPC backend, IQ4_NL GGUF, flash attention on, 64K context per agent, and all layers offloaded to the GPU. The host is an AMD EPYC 7663 with 128GB DDR4 RAM. Generation is given only as a 75-80 t/s range. Two instances run simultaneously (planner and coder).

generation:

30.1 tokens/s

prompt processing (prefill):

1515.8 tokens/s

quant:

Q4_K_M (GGUF)

User benchmarks Qwen3.5-27B Q4_K_M on an RTX 5090 at 30.09 t/s generation and 1515.83 t/s prompt processing with llama-bench (pp1024/tg400, 5 reps). Setup is llama.cpp b10741 with CUDA 13.3 on Windows 11, -ngl 99 and flash attention on, 15.65 GiB GGUF. A Qwen3 MoE 30B-A3B control on the same rig reaches 134.18 t/s tg, ruling out a global Windows/driver overhead. The user notes native Linux with MTP reaches about 76% of roofline and ~1.7x speedup, while the Windows/Ollama path runs 1.5-1.6x slower at every draft depth. PCIe Gen5 x1 vs x4 gave 27.95 vs 30.09 t/s, so decode is not host-link-bound.

generation:

62.0 tokens/s

quant:

Q6_K_XL (GGUF)

User reports 62 t/s generation. Source evidence (verbatim): Source: https://digtvbg.com/blog/llama-server-vulkan-rdna4-vllm-rocm-benchmark/ # Local LLM Inference on AMD RX 9070 XT: Vulkan vs ROCm Benchmarks on RDNA4 Hardware: AMD RX 9070 XT with 16GB VRAM on CachyOS. Model: Qwen3.5-9B-UD-Q6_K_XL (about 6.5 bits per weight). llama-server from llama.cpp built against Vulkan with --n-gpu-layers 999 -c 65536 --batch-size 2048 --ubatch-size 2048 --parallel 1: 62 tokens/second. vLLM on ROCm 7.2: 48 tokens/second. Same prompt, max tokens, temperature 0 and seed 42 for both; 10 measured runs per backend after 2 warmup runs. No speculative decoding is configured.

generation:

77.8 tokens/s

quant:

Q4_K_M (GGUF)

User reports 77.8 t/s generation. Source evidence (verbatim): Source: https://terminalbytes.com/ollama-vs-llama-cpp-vs-mlx-mac-2026/ # Ollama vs llama.cpp vs MLX on a Mac Studio, measured Hardware: Mac Studio with an M3 Ultra and 256GB unified memory. The author ran Qwen3.5 9B through every runtime with the same 34-token prompt, thinking off, one request at a time, two runs each, no speculative decoding. Results, generation speed: llama.cpp llama-bench build 10809 with unsloth Qwen3.5-9B Q4_K_M GGUF, tg128 77.8 tok/s; Ollama 0.32.13 GGUF qwen3.5:9b Q4_K_M 77.7 tok/s; Ollama 0.32.13 MLX NVFP4 90.5 tok/s; rapid-mlx 0.14.1 Qwen3.5-9B 4-bit 105.9 tok/s; mlx-lm 0.31.3 Qwen3.5-9B 4-bit 107.5 tok/s. The author wrote down the numbers from his own runs.

generation:

73.4 tokens/s

prompt processing (prefill):

1808.0 tokens/s

quant:

Q8_0 (GGUF)

kv:

f16

User reports Qwen3.5-35B-A3B at 73.43 tokens/s generation and 1808 tokens/s prefill on an Apple M3 Ultra 96GB. Setup is llama.cpp build 8860 with Q8_0 GGUF weights and f16 KV cache, Metal backend, single machine. The figures are separate synthetic tests (TG128 and PP512), not an end-to-end request. Historical TG128 85.48, turbo3 KV 63.23 and PP8192 1183 belong to other configurations and are kept for comparison only.

generation:

43.5 tokens/s

quant:

Q4_K_M (GGUF)

User reports 43.5 t/s generation. Source evidence (verbatim): Source: https://antekapetanovic.com/blog/qwen3.5-apple-silicon-benchmark/ # Ollama vs. llama.cpp vs. MLX with Qwen3.5 35B on Apple Silicon Apple M4 Max, 128GB unified memory, macOS 26.3. Model Qwen3.5 35B-A3B (MoE). Ollama Q4_K_M, llama.cpp Unsloth Dynamic Q4_K_XL, context 65536 for both, MLX 4-bit group quantization. Engine versions not stated. Generation only, mean of 10 runs, single request, no speculative decoding stated. Engine-reported generation tok/s, thinking off, deterministic params: mlx-py 130.2 +/- 0.8, llama.cpp 71.0 +/- 0.1, Ollama 43.5 +/- 0.4. Client wall-clock: mlx-py 114.9, mlx-http 107.6, llama.cpp 65.8, Ollama 40.5. Prompt processing was not reported.

generation:

29.2 tokens/s

quant:

6-bit (MLX)

User reports 29.2 t/s generation. Source evidence (verbatim): Source: https://alooftwaffle.substack.com/p/96-hours-with-dual-dgx-sparks-and # First-hand Mac Studio M3 Ultra 512GB Qwen3.5-397B-A17B 6-bit mlx-vlm generation measurement Original first-hand report by AlooftWaffle, published March 27, 2026: https://alooftwaffle.substack.com/p/96-hours-with-dual-dgx-sparks-and . Historical source-reported measurements, not a new benchmark or independent reproduction. Selected configuration: one Mac Studio M3 Ultra with 512GB unified memory, running Qwen3.5-397B-A17B at publisher-described 6-bit precision using mlx-vlm. The shortest reported context row gives generation 29.2 tokens/s. The source labels this row 4K context; whether that label denotes actual occupied input or configured capacity is not established. Actual fresh input count, cached prefix, output count and runtime version are unknown. Leave structured context length unknown. Do not replace it with 4096 or infer zero cache. CPU/GPU core counts, sampling, thinking mode, speculative decoding, request concurrency, warmup, repeats, statistical aggregation, exact measurement date, OS, KV precision, Flash Attention, offload, temperatures, power and peak resident memory are unreported. The source's longer Mac rows report generation 28.6 tokens/s at the 8K label and 26.7 tokens/s at 32K; preserve these as separate source rows, not an average. The author also calls values 317/537/476 tokens/s 'prefill throughput (time to first token)'; native timing boundaries and counters are absent, so leave structured prompt throughput and TTFT unknown rather than converting or equating these metrics. Verified upstream model-family reference: https://huggingface.co/Qwen/Qwen3.5-397B-A17B . This is a base/reference repository, not the tested 6-bit artifact. The tested quantization repository, revision, file names, hashes and quantization recipe are unknown; do not assign a community pack or GGUF Q6 code. The source describes nearly 300GB of downloaded model files; that is download/storage volume, not measured resident RAM or a peak-memory measurement. The same article's dual-Spark INT4-AutoRound/vLLM measurements and embedding sentences/s are different hardware, quantization and workloads; none is the selected Mac generation speed or a controlled comparison isolating hardware. Full original source remains the URL above; no author contact or comments were made.

generation:

69.8 tokens/s

quant:

Q4_K_M (GGUF)

User reports 69.8 t/s generation. Source evidence (verbatim): Source: https://www.fitmyllm.com/reports/rtx-3070-8gb # FitMyLLM original RTX3070 single-stream Qwen3.5 9B Q4_K_M measurement FitMyLLM first-party public report preview: measured on one rented NVIDIA GeForce RTX 3070 with 8GB VRAM, run finished 2026-08-29. Selected row Qwen3.5 9B Q4_K_M native decode 69.8 tokens/s, explicitly single stream (concurrent requests 1). llama.cpp build 91f8c9c5fb038c086e13e9cd823c29b33b07ba54, driver 595.71.05. The preview identifies seven successful models and two failed models, but exposes only five decode rates. Actual prompt/context depth, input/output token counts, repetitions, OS/CPU/core counts/system RAM, prefill rate, KVprecision and speculative-decoding settings are not disclosed for this selected row. Do not assign4K/8K/16K/32K depth sweep labels or1–8stream load tests from the paid report contents to this preview row. No exact tested HF repository, file checksum or model revision is linked; do not invent one or substitute a base-model reference. The runtime commit is a llama.cpp build revision, not a model revision. Original public source https://www.fitmyllm.com/reports/rtx-3070-8gb . Public methodology https://www.fitmyllm.com/methodology distinguishes first-party rented-hardware measurements from fitted estimates and describes headline rates as medians of independent runs; the specific repetition count and spread are not exposed in the preview. The methodology says raw JSON is freely accessible, while this report says its JSON requires purchasing the report; no raw API or paid material was accessed. Other public preview rows, separate deployments: Qwen3 4B Instruct2507 Q4_K_M 126; Llama3.1 8B Instruct Q4_K_M 80.9; Qwen3 8B Q4_K_M 78.4; Qwen3 8B Q3_K_M 73.3 tokens/s. These are not repetitions or ranges to average with the selected Qwen3.5 row.

generation:

15.3 tokens/s

quant:

UD-IQ3_XXS (GGUF)

User reports Qwen3.5-397B at 15.3 t/s across two DGX Sparks (128 GB unified each) over llama.cpp RPC, a model that does not fit on one node. Setup is a forked Ollama 0.40.3 with llama.cpp b11351, UD-IQ3_XXS quant (141 GB), 256 output tokens, warm run, two nodes linked by a 200 Gb/s cable. Splitting across nodes costs speed: fast small models lose 30-60% (gemma4:e2b 106.6 to 44.1 t/s), large models lose 15-20%, and load time is 3-5x longer. The default policy keeps a model on one machine and only splits when it does not fit.

prompt processing (prefill):

16459.6 tokens/s

quant:

Q8_0 (GGUF)

User benchmarks a GDN chunked kernel pull request in llama.cpp, reporting prompt-processing throughput on several GPUs and models. On an RTX PRO 6000 Blackwell with Qwen3.5-35B-A3B-Q8_0 at 2048 prompt tokens, the PR reaches 16459.56 tok/s against 13653.58 tok/s on master, a 20.6% gain. The table also covers RTX 5090, RTX 4090, RTX 3090 and Spark GB10 across Qwen3.5-35B-A3B, Qwen3.6-35B-A3B, Qwen3.8-27B and Qwen3.8-Flash-Next at 2048 and 4096 prompt tokens.

generation:

198.4 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

on

User reports 198.4 t/s generation. Source evidence (verbatim): Source: https://rapidmlx.com/compare/mlx-lm # Mac mini M4 Pro48GB Qwen3.5-9B speculative whole-file edit benchmark First-party engine-author measurements October4,2026. Mac mini AppleM4 Pro 48GB unified RAM, macOS26.5.1, ACpower, idle before starts; core counts unknown. Qwen3.5-9B4-bit, Rapid-MLX0.15.6 with MTP plus prompt-lookup versus mlx-lm0.32.0 default nonspeculative server, both MLX0.32.3. Whole-file rename task: 198.4 tok/s decode. Comparison baseline: 46.4 tok/s; median of3fresh-server runs, two discarded warmups, temperature0, thinkingoff. Exact prompt/output counts, contextcapacity, residentRAM and MTPcompanion artifact unknown. Decode=(completiontokens-1)/(last-firststreamedchunk), not end-to-end or prefill. Copy-heavy task benefits from prompt lookup; do not generalize peak. Testedquant repo verified: https://huggingface.co/mlx-community/Qwen3.5-9B-4bit . Sourceversions.json identifies shortened revision8b2b98c; fullSHA unknown and revisiontree read returned generic toolerror. Sources: https://rapidmlx.com/compare/mlx-lm ; https://rapidmlx.com/blog/assets/vs-mlx-2026-10/versions.json ; https://rapidmlx.com/blog/assets/vs-mlx-2026-10/raw/manifest.json . Source author builds tested engine; rawmanifest links individual runs. Selected run is Rapid-MLX with MTP plus prompt-lookup, generation 198.4 tok/s. Baseline mlx-lm measurement is a separate run. Prefill speed unknown.

generation:

22.6 tokens/s

prompt processing (prefill):

247.2 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.5 122B-A10B at 22.6 t/s generation and 247.2 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama-bench build b8171 with the Vulkan backend, UD-Q4_K_XL quant, ngl 99, and a Radeon 8060S Graphics (RADV GFX1151) device. The model is the 122B-A10B artifact at 63.65 GiB; PP512 is 512 benchmark prompt tokens and TG128 is 128 evaluation steps, so the two rates are not a same-request pair.

generation:

31.8 tokens/s

prompt processing (prefill):

747.7 tokens/s

quant:

UD-Q8_K_XL (GGUF)

User reports Qwen3.5 35B-A3B at 31.8 t/s generation and 747.7 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama-bench build b8171 with the Vulkan backend, UD-Q8_K_XL quant, ngl 99, and a Radeon 8060S Graphics (RADV GFX1151) device. The rates are separate synthetic phases: PP512 at 747.72 +/- 44.81 t/s and TG128 at 31.83 +/- 0.03 t/s, not a same-request prompt/output pair.

generation:

53.4 tokens/s

prompt processing (prefill):

865.7 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.5 35B-A3B at 53.4 t/s generation and 865.7 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama-bench build b8171 with the Vulkan backend, UD-Q4_K_XL quant, ngl 99, and a Radeon 8060S Graphics (RADV GFX1151) device. The rates are separate synthetic phases: PP512 at 865.72 +/- 59.59 t/s and TG128 at 53.39 +/- 0.08 t/s, not a same-request prompt/output pair.

generation:

10.7 tokens/s

prompt processing (prefill):

220.4 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.5 27B at 10.7 t/s generation and 220.4 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is the b8171 Vulkan build with the UD-Q4_K_XL GGUF, 26.90 B parameters and 16.40 GiB, ngl 99, on a Radeon 8060S (RADV GFX1151). The native figures are PP512 220.35 +/- 3.36 t/s and TG128 10.66 +/- 0.01 t/s, separate synthetic phases rather than a same-request pair; batch, threads, context depth and KV types are unreported.

generation:

31.1 tokens/s

prompt processing (prefill):

819.2 tokens/s

quant:

UD-Q4_K_XL (GGUF)

User reports Qwen3.5 9B at 31.1 t/s generation and 819.2 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama-bench build b8171 with the Vulkan backend, UD-Q4_K_XL quant, ngl 99, and a Radeon 8060S (RADV GFX1151) device. The rates are separate synthetic phases: PP512 at 819.24 +/- 55.72 t/s and TG128 at 31.09 +/- 0.05 t/s, not a same-request prompt/output pair.