Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen 3 14B at 11.5 tok/s on a Mac Mini M4 with 24 GB unified memory.
Setup is MLX with 4-bit quantization, using 8.4 GB peak memory and leaving 74% of system memory free with no swap.
The study also reports Ollama q4_K_M results: Llama 3.2 3B at 42.0 tok/s, Gemma 2 9B at 15.2 tok/s, Qwen 2.5 14B at 10.8 tok/s, and DeepSeek-R1 32B at 0.28 tok/s. MLX Qwen 3 30B-A3B MoE reaches 44.8 tok/s but uses 16.3 GB and causes 2.74 GB swap; Qwen 3.5 35B-A3B crashes with Metal OOM at 20.4 GB. Tests ran under normal workstation load with ~12-13 GB already committed.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
codingagentic
User benchmarks Qwen3 30B-A3B at 108-110 t/s decode and 1683-4323 t/s prefill on an RTX 5070 Ti 16 GB.
Setup is Ollama 0.35.0 with Q4_K_M at 8192 context, median of 2 runs, measured idle.
The model produced no compilable code (0/2), failed one-word obedience with 899 characters, and returned 0 characters on the reasoning test after burning its 4000-token budget on 15088 characters of thinking. User notes prefill swings 2x with background load and that the fastest model here measures verbosity.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3-30B-A3B 4-bit at 57 tok/s cold generation on an M5 MacBook Air with 32GB.
Setup is MLX with mlx-lm, 4-bit weights, 256 generated tokens, no speculative decoding.
Sustained generation drops to 44 tok/s on battery and 39 tok/s on mains; prompt processing of a 4,096-token prompt is 1,063 tok/s cold, 788 sustained on battery and 725 on mains. The decline finishes at about 150 seconds and the rate then holds flat.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3 4B at 260.52 t/s prompt processing and 12.71 t/s generation on an Intel Arc B580.
Setup is llama.cpp (IPEX-LLM build) with Q4_K_M GGUF, all 99 layers offloaded to the GPU via SYCL.
Compared against stock Ollama on the same model, which reached 14.23 t/s prompt and 6.20 t/s generation, making the Arc GPU about 18.3x faster at prompt processing and 2.05x faster at generation.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3-30B-A3B at 15 t/s on an RTX 4060 Ti 16GB using TensorTune's smart tensor offloading.
Setup is KoboldCpp with Q4_K_M quant, all layers on GPU with tensor offloading.
The baseline with traditional layer offloading (30 layers) was 10 t/s, a 50% improvement.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen 32B at 31.2 t/s generation on 1K context using an M3 Ultra 512GB.
Setup is MLX with Q4 quantization and FP16 KV cache, batch size 1, 256 generation tokens, median of 3 trials.
The post reports a systematic sweep of 5 models, 6 quants, and 7 context lengths (276 runs total), showing generation drops to 8.5 t/s at 128K context and prompt processing falls from ~345 t/s at 1K to ~154 t/s at 128K.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports CLM-v0.1-8B, a frozen Qwen3-8B encoder with fp32 heads, at ~336 tok/s on an M3 Pro.
Setup is MLX with an 8-bit checkpoint of 7.5 GiB, peaking at 9 GB of memory.
Checked against the authors' vLLM server on 778 questions with 99.0% top-1 agreement, within their own run-to-run noise. Unofficial community port, not reviewed by the CLM authors.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3-30B-A3B 4-bit at 127.7 tok/s on an M4 Max 128GB.
Setup is vllm-mlx, a native MLX inference server, with greedy decoding and single-stream decode.
Two other models were also measured on the same machine: Qwen3-0.6B 8-bit at 417.9 tok/s and Llama-3.2-3B-Instruct 4-bit at 205.6 tok/s. Continuous-batching results, KV-cache quantisation and MoE top-k sweeps are documented separately.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
long-context
User reports Qwen3 30B-A3B at 58.57 t/s decode and 920 t/s prefill on an AMD Strix Halo 128GB at 12,001 tokens of context.
Setup is llama.cpp with the Q4_K_M GGUF over Vulkan, 49,152-token context, 96 GB of the 128 GB LPDDR5X allocated to the iGPU.
At 48,001 tokens prefill drops to 254 t/s and decode to 30.06 t/s; at 11 tokens prefill is 122 t/s and decode 88.40 t/s. ROCm gives about 28 percent better prefill while Vulkan gives about 13 percent better decode.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3-1.7B at 33.2 tok/s on an Intel Arc A770 16GB using llama.cpp with the SYCL backend.
Setup is llama.cpp SYCL with Q8_0 quantisation, single-stream (n_parallel=1).
OpenVINO via OVMS reached 65.4 tok/s on the same model. Ten models were tested in total, with OpenVINO faster than SYCL on every model.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3-14B Heretic at 3.56 t/s on an Intel AI Boost NPU37XX in an Acer Predator Helios 16 AI laptop with 32 GB RAM and an RTX 5090 Laptop 24GB.
Setup is OpenVINO GenAI with the machine-made-Fibre INT4_SYM OpenVINO IR export, group_size=-1, all_layers=true, loaded directly via openvino_genai.LLMPipeline on NPU with no conversion. Load took 33.52 s and 96 output tokens took 26.98 s; peak RAM was about 15.6 GB with a 9.7 GB working set.
User also benchmarked Qwen3-8B INT4 SYM on the same NPU at 6.72 t/s with 6.15 s load, and found Qwen3-30B-A3B MoE impractical due to host memory pressure during OpenVINO preparation.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
visioncoding
User reports a VLM is best overall for OCR and detail extraction, correctly reading mixed-script text (Chinese + Latin) and catching fine details other models missed.
Output is verbose at 1.4-2.2k tokens.
The user recommends it as the default for a coding-assistant MCP.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3-235B-A22B at about 7.5 t/s generation and about 75k pp on a single RTX 3090 with 128 GB of system RAM.
Setup is textgen with batch size 1024, 16384 context, autofit layers, and 38 of 48 threads on a dual Xeon DDR4 system.
Generation dips to about 7 t/s at 10k context. The user calls it a daily driver for months for uncensored ERP roleplay, and mentions Step 3.7 Flash as an alternative that is censored and verbose, along with Gemma 4 and MiniMax-M2.5, which were not run.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
summarizationmultilingual
User benchmarks a DSpark PC Tree speculative decoding fork at 159.00 tok/s in its best config, PCTree k3/n16, against 94.27 tok/s for plain decoding.
User also tests Qwen3.8 27B Q4, with worse results.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports 50-54 t/s with a custom CUDA/C++ engine, a 50% improvement over llama.cpp at 33-34 t/s.