QwQ Qwen 32B Q4_K_M; llama-bench TG128 / PP512
M5 Max 48GB · llama.cpp
- generation:
- 26.3 tokens/s
- prompt processing (prefill):
- 685.4 tokens/s
- quant:
- Q4_K_M (GGUF)
- flash attention:
- on
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
Source-reported historical campaign by emihiggins, saved 2026-09-27T02:04:47Z; recovered 2026-10-13. No local model run or reproduction performed. Primary raw result: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/gguf/qwq-32b_Q4_K_M_ngl99.json . Source publication commit is 34fee99b6df6129fb3c64255c1bfb179a54ca044. The author's upstream submission, including corrections and limitations, is https://github.com/enescingoz/mac-llm-bench/pull/12 . Hardware: one Apple Mac Studio Mac17,14, M5 Max, 18 CPU cores and 40 GPU cores, 48 GB unified memory; macOS 27.0 build 26A428, bare metal, AC power, High Power energy mode. Source corrects its JSON core labels: the 6/12 split denotes 6 Super and 12 Performance cores, not 12 Efficiency cores. iogpu.wired_limit_mb=32768 is a session limit, not installed RAM or dedicated VRAM. PR reports ambient about 20.6 C, external WD_BLACK SN7100 NVMe in OWC Express 1M2 over USB4, idle Ollama with no model loaded and no other heavy apps. Swap was at most 2 MB when checked every 60 s. No macOS thermal/performance warnings were recorded, but per-model thermal state was not captured. Do not import zero-swap/per-cell thermal claims from the author's separate bandwidth harness. Exact model: bartowski/Qwen_QwQ-32B-GGUF, file Qwen_QwQ-32B-Q4_K_M.gguf, GGUF Q4_K_M. Manifest: https://github.com/emihiggins/m5-max-llm-bench/blob/main/data/mac-llm-bench/m5-max_18c-40g_48gb/gguf-revisions.tsv . It records measured repository revision 390cc7b31baedc55a4d094802995e75f40b4a86d and SHA256 87cc1894a68008856cde6ff24bfb9b99488a0d18c2e0a2b1ddeabd43cd0498e0. The exact revision's HF file page was independently read and its displayed SHA256 matched: https://huggingface.co/bartowski/Qwen_QwQ-32B-GGUF/blob/390cc7b31baedc55a4d094802995e75f40b4a86d/Qwen_QwQ-32B-Q4_K_M.gguf . This binds model identity and quantized artifact; today's main is not substituted for the measured revision. Source nominal model size label is 32B. Runtime: llama.cpp Homebrew build 11146, commit 7fe450e19, ggml 0.25.3, Metal. The raw runtime.version accidentally stores a load_backend BLAS log line; the author explicitly corrects it in PR 12 and reports Metal 4 tensor API active. This is not a CPU-only BLAS benchmark. Each model was run in a separate ./bench_gguf.sh --model process because --auto stopped after the first model. Source parameters are n_gpu_layers=99, flash_attention=true, threads=0. The helper omits -t when threads=0; this means runtime default rather than zero running threads. Requested 99 GPU layers is not a measured layer-offload count. Quantization above describes weights, not KV cache. Headline generation is tg128=26.27 tokens/s; prompt processing is pp512=685.41 tokens/s. These are separate native llama-bench microtests from the same campaign, not one 512-input/128-output request, an average across lengths, or server concurrency throughput. All retained native rates in tokens/s: pp128=467.8; pp256=641.72; pp512=685.41; tg128=26.27; tg256=26.15. The pp tests process 128/256/512 synthetic token IDs; the tg tests perform 128/256 one-token decode steps without a preceding 512-token prompt. The context field is intentionally unspecified: these test lengths are not an advertised context window or a long-context setting. Metric method: published mac-llm-bench bench_gguf.sh entrypoint passes -p 128,256,512 -n 128,256 -ngl 99 -fa 1 to llama-bench. It first runs human-readable output, then a separate -o json invocation for saved speed results; those two invocations are not combined. This entrypoint has inline speed/memory logic; the separate lib/run_bench_gguf.sh helper is not the saved-speed entrypoint, so its different display/capture ordering must not be substituted. lib/collect_results.sh maps native n_prompt/n_gen and avg_ts into these speed keys and rounds to 2 decimals. Native llama.cpp code at the reported 7fe450e19 revision times synchronized inference, computes each repetition's token count divided by elapsed nanoseconds, then takes the arithmetic mean of those rates. This is not input divided by TTFT, total wall time, or a pooled count/total-duration calculation. The pinned llama-bench defaults are 5 repetitions, warmup enabled, depth 0, with model memory cleared before each timed repetition; PP uses synthetic random IDs and TG sequential random IDs, without sampling meaningful chat output. This establishes native benchmark semantics. The exact executed shell-harness commit and full raw llama-bench samples, durations, dispersion and command log were not preserved in this result, so the 5-repetition/default-cache interpretation is code/protocol-derived rather than independently counted from raw samples. Current upstream shell code is supporting method evidence, not proof of byte-for-byte historical reproduction. KV precision, actual CPU thread count, realized batch settings, speculation/MTP and quality scores are not separately captured in this JSON; no quality validation or conversational thinking-mode conclusion follows. Memory: file_size_gb=18.49 is the on-disk file size divided by 2^30, despite the GB key. Source peak_rss_gb=18.68 is separately measured process maximum RSS. The published entrypoint obtains RSS with /usr/bin/time -l on a separate llama-bench -p 32 -n 1 invocation, then divides bytes by 2^30. It is not the timed speed test's peak, dedicated VRAM, model-only allocation or installed capacity. The per-record available_ram_gb=14.1 is available system memory at collection, not model usage. Method sources: https://github.com/enescingoz/mac-llm-bench/blob/main/bench_gguf.sh ; https://github.com/enescingoz/mac-llm-bench/blob/main/lib/collect_results.sh ; https://github.com/ggml-org/llama.cpp/blob/7fe450e19/tools/llama-bench/llama-bench.cpp . Source data/documentation license is CC BY 4.0. All values remain source-reported single-machine results; these short synthetic tests do not establish long-context, multi-user or application end-to-end performance. Timing discrepancy: this raw record is timestamped after the PR’s stated GGUF window ending 2026-09-27 01:29 UTC. The exact raw timestamp above is retained; the source does not explain the discrepancy, so the PR end time is not treated as a hard bound.