Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3-14B-4bit at 67 t/s generation and 2529.8 t/s prompt processing on a Mac Studio M5 Ultra with 256 GB unified memory.
Setup is MLX via mlx-lm 0.31.3 with 4-bit MLX Safetensors, batch 1, no prompt cache, no draft model, and no KV or activation quantization.
The figures are medians across five trials at a native 32768-token input and 128-token generation cell, with peak memory 14.332 GB; the 32768 context is the actual input length, not a supported maximum.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
NetworkChuck campaign measured 2026-09-20, recovered 2026-10-13; historical source evidence, no new/local run. Merged window 2026-09-20T14:41:54.718Z–2026-09-20T16:55:25.045Z; native-cell timestamps unavailable. Credit NetworkChuck, CC BY 4.0 data/docs. Repository: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/ . Native data: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/engine.json . Client summary: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/summary.json . Client requests: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/runs.jsonl . Method: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/docs/method.md .
Headline: native 32768-input/128-generation cell, PP median 570.075, TG median 35.801 tokens/s across five trials; not pooled-duration or client rates. Context 32768 is actual native input length, not supported maximum or configured KV capacity. Batch 1.
Hardware: one Apple Mac Studio M3 Ultra (32-core CPU, 80-core GPU), CPU core composition 24 performance+8 efficiency, 512 GB installed unified memory, macOS 26.5.1, GPU architecture applegpu_g15d. Both review units had 8 TB storage. Installed unified memory is not dedicated VRAM. A stuck background indexing process occupied 100% of one of this M3 Ultra's 32 CPU cores throughout this campaign, until 2026-09-21 10:10 CT. The effect was not measured by rerunning; do not silently correct the rates. This was Chuck's own machine. Power, energy, thermals, clock rates, exact storage part, explicit GPU-offload count and swap state are not reported for these rows.
Model: mlx-community/Qwen3-14B-4bit, 4-bit MLX Safetensors; public identity verified 2026-10-13: https://huggingface.co/mlx-community/Qwen3-14B-4bit . Source calls it dense, nominal 14B total/active, local folder ~/bench-models/Qwen3-14B-4bit. Source disk label 7.8 GB is not runtime memory. Tested HF revision/hash unrecorded; author says weights were not hash-verified. Current HF main, conversion version and download size do not identify the historical snapshot/runtime. Quant group size/mixed-precision exceptions unverified.
Runtime: mlx-lm 0.31.3 and MLX 0.32.2; server fingerprint 0.31.3-0.32.2-macOS-26.5.1-arm64-arm-64bit-applegpu_g15d. Native harness command is mlx_lm.benchmark --model <local-model-folder> -p <prompt-count> -g 128 -n 5, verified in https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/bench/ssh.ts . Model loaded before timing; one discarded warm-up then five trials. Harness median/min/max uses three-decimal printed trials; raw output retained. Warm-loaded, excludes cold loading.
Native method: versioned https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/benchmark.py seeds RNG 0, uses random token IDs, disables EOS stopping and calls stream_generate; no HTTP essay/chat workload. Published command leaves batch 1, prefill step 2048 and delay 0 defaults. https://github.com/ml-explore/mlx-lm/blob/v0.31.3/mlx_lm/generate.py defines prompt_tps=prompt.size/runtime time to first generated token, including that boundary rather than pure matrix-kernel time. Generation timing restarts there; generation_tps=generated count/runtime generation-window time, including final detokenizer handling, final count 128. This differs from client (128−1)/first-to-last text. No prompt_cache passed, so generate_step creates fresh KV. No draft model, KV-quantization or activation-quantization flag supplied. Flash/kernel selection and runtime build commit unrecorded; hardware alone does not establish a toggle.
All native rows, input/output; PP median [min–max]; TG median [min–max], tokens/s; source peak-memory GB; trial count:
512/128; PP 813.779 [812.584–814.936]; TG 69.365 [69.32–69.391]; peak 8.826 GB; n=5
4096/128; PP 814.254 [813.998–815.156]; TG 62.896 [62.85–62.918]; peak 9.701 GB; n=5
16384/128; PP 690.643 [690.481–691.238]; TG 47.455 [47.42–47.465]; peak 11.602 GB; n=5
32768/128; PP 570.075 [569.782–570.199]; TG 35.801 [35.795–35.812]; peak 14.332 GB; n=5
Memory semantics: source peakMemoryGb is the maximum of printed native peak_memory values across five trials. Upstream peak_memory is mx.get_peak_memory()/1e9, an MLX allocator high-water mark in decimal GB; it is not RSS, installed RAM, dedicated VRAM, or a separately reset per-trial allocation. Warm-up and earlier allocation in that benchmark process can contribute to the high-water mark. No model-load timing or energy number is supplied by these rows.
Separate HTTP arm, same model/Mac: serial streaming /v1/chat/completions from LAN controller to mlx_lm.server; --prompt-cache-size 0 and --chat-template-args '{"enable_thinking":false}' , temperature 0, max_tokens 128. Prompt repeats prompts/sample.txt with unique run-ID prefix and summary instruction. One discarded warm-up, five measured requests/cell, 30 s between cells. Single 100K requests, if listed, are not five-repeat cells. Actual counts from usage. All checked raw requests in this group: cachedTokens 0, outputTokens 128, firstWordField content. HTTP cache evidence does not mean cold model loading.
Controller monotonic timing begins before request; first text is first nonempty delta. Reading=input tokens/client first-text seconds, including LAN/HTTP. Writing=(output−1)/(last-text−first-text seconds). Total ends at stream completion. These differ from native metrics. Implementation: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/lane.ts and https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/timing.ts .
All HTTP rows, each metric shown as median [min–max]; nominal length; actual token counts; client reading/writing tokens/s; first-text and total seconds; repetitions:
512 (512 nominal); actual input 507 [507–508], output 128 [128–128]; TTFT-derived reading 681.304886619 [680.15894523–683.748234999]; client writing 61.7620769176 [61.5733692754–61.8337956262]; first text 0.744678 [0.741501–0.745628] s; total 2.801364 [2.797872–2.808342] s; n=5
4K (4096 nominal); actual input 4097 [4095–4097], output 128 [128–128]; TTFT-derived reading 776.541288513 [774.603443497–777.677154077]; client writing 55.7032446921 [55.5113601595–55.8309246173]; first text 5.275959 [5.266477–5.289158] s; total 7.555642 [7.54761–7.577066] s; n=5
16K (16384 nominal); actual input 16280 [16279–16281], output 128 [128–128]; TTFT-derived reading 669.146411504 [668.529108244–669.423496892]; client writing 41.7188837078 [41.4444331315–41.7777862576]; first text 24.330998 [24.320927–24.350473] s; total 27.382021 [27.368085–27.394455] s; n=5
32K (32768 nominal); actual input 32521 [32520–32521], output 128 [128–128]; TTFT-derived reading 550.950577802 [550.6959282–551.18283725]; client writing 31.5317764581 [31.4994611608–31.5856519565]; first text 59.025261 [59.0022–59.054368] s; total 63.05821 [63.031217–63.090287] s; n=5
Independent M3/M5 invocations were merged with shared settings checked. Different OS, M3 indexing load and unverified weight snapshots confound silicon-only comparisons. No independent model-quality evaluation.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3-8B on an RTX PRO 6000 Blackwell 96GB, reporting 86 tok/s single-stream decode in BF16 under vLLM.
Setup is vLLM 0.27.1 with BF16 weights, greedy decoding, concurrency 32 for aggregate figures; SGLang 0.5.9 and llama.cpp CUDA builds were also tested.
Aggregate throughput at concurrency 32 was 1,725 tok/s for vLLM, 1,327 for SGLang and 428 for llama.cpp; single-stream decode sat in an 83 to 96 tok/s band across all stacks. An FP8 checkpoint on vLLM reached 130 tok/s single-stream and 2,597 tok/s aggregate, with p50 latency down from 0.74 s to 0.49 s.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3 30B-A3B at 98.2 t/s on an M5 Pro with 64GB unified memory.
Setup is MLX with 4-bit affine quantization, group size 64, standard 8 routed experts per token, batch 1, 944-token prompt and 512 output tokens.
The figure is the arithmetic mean of six fresh-process runs, ranging 97.97 to 98.37 t/s; an altered 6-expert run reached 107.79 t/s but is not a standard-model result.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3 30B-A3B at 144.4 t/s on an M5 Max with 128GB unified memory.
Setup is MLX with 4-bit affine quantization, group size 64, standard 8 routed experts per token, batch 1 with a 944-token prompt and 512 output tokens.
The figure is the arithmetic mean of six fresh-process runs, ranging 144.10 to 145.01 t/s; an altered 6-expert routing gave 155.61 t/s but is not a standard-model result.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
visionmultilingual
User reports Qwen3-30B MoE running at up to 96 tok/s on a jailbroken PS5 (CFI-1208B, firmware 13.60) using its RDNA2 GPU via the Mesa RADV Vulkan driver.
Setup merges llama.cpp, stable-diffusion.cpp and whisper.cpp into a single homebrew title accessing ~12 GiB of unified Vulkan VRAM.
Also reports Mamba-Codestral 7B at 57 tok/s, 27B/34B models at 16-21 tok/s, Whisper large-v3-turbo plus Qwen 4B and Qwen3-TTS voice agent with first audio in ~2.5 s, Gemma 3 12B and Qwen2.5-VL vision in ~2.7 s, FLUX.2-klein 4B 768x768 images in 17.6 s, and HunyuanVideo and Wan 2.1 video generation.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
agentictool-use
User reports Qwen3-8B at ~10 tok/s decode on a 16GB Mac (M4) using MLX with 4-bit quantization.
Setup runs a persistent local agent with Qwen3-8B on the GPU, Qwen3-4B as 36 int8 layer shards on the Neural Engine, and Qwen2.5-0.5B for routing. The ANE toolchain is custom Rust with no Python.
User notes 16GB is a hard ceiling and the mesh has only run on one machine so far.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3 30B-A3B MoE at 86.1 t/s token generation on a Strix Halo 128GB (Ryzen AI Max+ 395, Radeon 8060S).
Setup is llama.cpp with Q4_K_M GGUF, Vulkan RADV backend, 120W power profile, single host (GMKtec EVO-X2).
Prompt processing measured at 1142 t/s (pp512). The repo also lists MiniMax M2.5 Q3_K_M at 32.8 t/s, Qwen3-Coder-Next 80B-A3B Q4_K_M at 42.7 t/s, GPT-OSS 120B Q4_K_M at 53.4 t/s and Nemotron-3 Nano 30B-A3B MXFP4 at 61.5 t/s on the same hardware.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3-8B Q4_K_M at 61 t/s generation and 851 t/s prompt on an RX 6700 XT 12 GB.
Setup is the bundled llama.cpp (0.85.5) with the AMD Flash Attention kernel, KV cache f16, measured with pp512 / tg128.
The project's headline claim is that the same card goes from 0.6-2.6 t/s under stock llama.cpp to ~61 t/s with ToshLLM. The page also lists Qwen3.6-35B-A3B Q4_K_S at 475 t/s prompt / 29 t/s generation with --n-cpu-moe 24, and community rows including a 122B-A10B MoE across four W6800X GPUs at 226 t/s prompt / 26 t/s generation.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3-0.6B at 531 tok/s decode on an RTX 3090 using a custom CUDA megakernel, 3.9x faster than HuggingFace.
Setup is a batch=1 bf16 cooperative megakernel with static shapes and short context; the project is a learning exercise, not a production engine.
Decode falls to 422 tok/s at position 200 and 158 tok/s at longer contexts, where TensorRT-LLM's 355 tok/s is faster. The kernel is latency-bound by synchronization at 5% memory bandwidth utilization.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3-8B Q4_K_M at 98.8 t/s decode and 4245 t/s prefill (pp512) on an RX 9070 XT with llama.cpp's HIP backend.
Setup is llama.cpp b10290 official HIP build on one RX 9070 XT 16 GB, Q4_K_M weights, all layers on the GPU, Windows 11, no speculative decoding.
The issue reports a Vulkan backend fault on the same card: Qwen3-8B Q4_K_M drops to 21.0 t/s decode and 791 t/s at pp128 there, and models with hidden size 4096 or more run 4.7-6.7x slower than on HIP, while Qwen3-4B is normal at 183.0 t/s. Qwen3-14B Q4_K_M runs 54.3 t/s on HIP against 8.1 on Vulkan.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User reports Qwen3 30B-A3B at 130.628 t/s generation and 5402.434 t/s prompt processing on an Apple M5 Max with 128GB unified memory.
Setup is MLX 0.32.2 with MLX-LM 0.31.3 on macOS 27.0; quantization, context length and KV cache settings are not reported.
Figures come from a third-party hardware review (Tbreak, September 21, 2026), not a reproducible benchmark log; prompt and generated token counts are unknown.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3 0.6B Q4_K_M on an NVIDIA GT 1030 2GB in an old Core2 Quad Q6600 PC with 4GB DDR2 RAM, comparing Vulkan and CUDA backends.
Setup is llama.cpp with Q4_K_M weights, q8_0 KV cache, flash attention on, 99 layers offloaded, and n_batch 1024. Vulkan gives 494.23 t/s pp512 and 64.77 t/s tg128; CUDA gives 834.55 t/s pp512 and 40.72 t/s tg128.
A second Vulkan run without int dot support drops pp512 to 242.19 t/s while tg128 stays at 64.36 t/s. User asks why Vulkan decode is faster and CUDA prefill is faster, and whether a mix of ~800 t/s prefill and ~60 t/s decode is possible.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen3 MoE 30B-A3B at 352.06 t/s on an RTX 5090.
Setup is llama.cpp with Q4_K_M and flash attention, 99 layers offloaded to GPU.
This is the optimized run with kernel fusion and concurrent CUDA streams enabled; baseline was 246.96 t/s. The post also reports gpt-oss 20B MXFP4 at 419.14 t/s on the 5090 and both models on an RTX 4090.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
summarizationtool-use
User benchmarks Qwen3-8B at 25.2 tok/s on an M3 Pro 18GB.
Setup is llama.cpp b11246 with Q4_K_M and 1024-token input, 256 output tokens, single request.
Runs were taken under memory pressure so absolute speeds are pessimistic; MLX-LM reaches 35.5 tok/s at 4 concurrent requests while llama.cpp plateaus at 21.0.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen3-0.6B at 76.2 t/s on an RTX 4050 Laptop 6GB, a 4.19x speedup over uncompressed bf16 (18.2 t/s).
Setup is a custom engine (DanilDrive-W) with lossless dictionary-compressed bf16 weights, decompressed inline in the CUDA kernel; file size drops from 1433.7 MB to 765.5 MB (-46.6%).
User also tested Qwen3-Coder-30B-A3B against Ollama and LM Studio on the same Q4_K_M GGUF, getting 16-18 t/s versus their 24-26 t/s, and notes greedy decoding can diverge after ~200 characters due to float summation order.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports decode speed improvements on Qwen3-30B-A3B with IQ3_XXS quantization on an M5 MacBook Pro 24GB. The decode speed increased from 65.6 to 73.9 tok/s using a llama.cpp PR for Metal optimization.
Setup uses llama.cpp with IQ3_XXS quantization. The user also tested on an M1 Pro 32GB.
The user is looking for testers for Qwen3.8-Flash-Next and provides benchmark commands.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3 8B at 77.70 t/s generation and 1172.08 t/s prompt processing on an Intel Arc Pro B70 32GB external GPU.
Setup is llama.cpp b10852 with Q4_K_M GGUF, FP16 KV cache, flash attention, 512 batch, 10 CPU threads, full GPU offload via SYCL backend on Windows.
SYCL repeat measured 83.51 t/s generation; Vulkan and CUDA backends were also tested. A separate 49B capacity test with Llama 3.3 Nemotron Super 49B Q4_K_M achieved 15.99 t/s generation at depth 0 with all 81 layers offloaded.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen3-30B-A3B-Thinking-2507-Q4_0 on an AMD MI50 using a specialized llama.cpp-gfx906 fork with flash attention and Q8_0 KV cache quantization. Prompt processing reaches 1224.07 t/s at 512 tokens and generation 63.00 t/s at 128 tokens. Without KV quantization, generation is 79.92 t/s but prompt processing drops to 1167.28 t/s at 512 tokens. The fork is optimized for D=128 head dimension models and reports 5-11% prompt processing improvement from flash attention.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen 3 8B at 94.1 tok/s decode on an M4 Max (Mac Studio) under Apple's llm-benchmark protocol.
Setup is Apple Core AI with a 4-bit artifact, 512-token prompt and 1024 generated tokens, greedy, mean of 5 trials. MLX on the same protocol reaches 90.0 tok/s.
The report is a cross-runtime comparison of Apple Core AI against MLX and Core ML on Apple Silicon.