- generation:
- 64.4 tokens/s
- prompt processing (prefill):
- 491.1 tokens/s
- quant:
- 4-bit (source config) (MLX)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
First-party historical Patrick Whelan benchmark, inspected 2026-10-13. Source snapshot 11f3f8f931dc97a71499e7b9c2583d010a3c160b verified by archive PAX comment and GitHub currentOid; pinned runner/config/hardware/VLM files byte-match archive. This pins the published evidence, not the executable used in April. No inference rerun or model-weight download. Hardware: one participating Apple M2 Max integrated GPU, 30 GPU cores, 12 CPU cores and 64 GB shared unified memory. Raw system data confirms those values; hardware profile explicitly says no discrete GPU and all inference on the M2 Max GPU. Shared 64 GB is system capacity, not dedicated VRAM or resident model use. Raw environment: Darwin 25.3.0 / ARM64_T6020 and Python 3.14.3. Do not borrow stale hardware-profile Sequoia 15.x or ai-gateway/vllm-mlx labels. Power, thermals, clocks, swap, wired-memory limit, GPU utilization and competing workload are unknown. Dates: raw system_info.timestamp=2026-04-03T22:42:14.957326+00:00; this is collected once before a run list, so it is a campaign/system-capture timestamp, not exact model start. Result filename granite-3.2-8b_default_20260403_235419.json uses local completion time with no timezone recorded. README updated 2026-04-04 is a publication label, not every measurement date. Configuration: direct python -m mlx_lm benchmark on Apple MLX/Metal, no HTTP server involved. MLX/MLX-LM/MLX-VLM installed versions and exact runtime commit are not retained in JSON. Requested prompt length 512 and generation length/cap 128, printed batch_size=1; three measured trials retained after discarding Trial 1, plus an internal/explicit warmup. Achieved token-count telemetry is not stored, so actual input/output fields remain unknown. Source context_size=4096, threads=8 and gpu_layers=-1 are explicitly unused for MLX and are not recorded here as active allocation, CPU threads or layer placement. KV/cache precision, prefill chunk size, sampling settings, thinking behavior and memory-cache reuse are unreported for the historical runtime; no drafter is passed by the wrapper. Native rates: PP 491.05 tok/s (sample SD 1.12; retained [491.449, 491.918, 489.785]); TG 64.43 tok/s (sample SD 0.02; retained [64.428, 64.408, 64.441]). Source aggregation rounds the three-trial arithmetic mean/sample SD to two decimals after parsing three-decimal CLI values. All values, means and SD were checked against raw Trial 2-4. CLI Averages includes all four trials and therefore is not the published three-trial metric. No pooling across engines/models. Timing exclusions: raw time_to_first_token_s mean 1.04 is explicitly computed as 512/prompt_tps by runner lines 718-719, not independently measured client or request TTFT. TTFT field is intentionally empty. Overall wall_time_s=724.05 measures the entire subprocess and can include HF downloads, model load and warmup; it is not inference-only latency or end-to-end response timing. As a reference audit only, official MLX-LM tag v0.31.2 benchmark.py uses batch 1 random token IDs, internal warmup, disables EOS, and forwards stream_generate rates. generate.py starts the prompt timer before generation and closes it on the first generated token; PP is prompt.size/prompt_time. TG counts n+1 tokens over elapsed time after that boundary, so its numerator includes the first token while its first-token time is in PP. These are engine-reported native rates, not transport/client rates. The actual installed MLX-LM revision is absent; v0.31.2 is corroborating implementation, not a tested-run version. Artifact: mlx-community/granite-3.2-8b-instruct-preview-4bit. Source-configured HF artifact joined by model key; raw JSON omits repository/digest; historical tested revision unknown. Raw quant is "N/A" because the MLX run dictionary lacks a quant key; quantization identification comes from source config/profile plus HF metadata, not that raw field. No weight digest is logged. Current artifact metadata inspected 2026-10-13 at HF revision c11bf25d3ac421d693a76b6f77f784b2e6ce1e11: MLX 4-bit group 64 (mode not explicitly named in config). This revision is a current metadata reference, not a historical run pin. Model-card conversion-tool versions are not installed benchmark engine versions. Model context capacities from current HF metadata are not active benchmark allocations. Memory: raw retained peak_memory values [5.218, 5.218, 5.218], maximum 5.218 in engine output units. Wrapper overwrites peak_vram_mb with this value times 1024 (5343.2), but leaves a stale unified_memory_system_wide note. It is MLX allocator/process peak reporting, not dedicated VRAM or system-wide resident usage. Reference MLX-LM v0.31.2 reports decimal GB (bytes/1e9), so wrapper times-1024 conversion may be unit-inconsistent; actual installed version is unknown. Dedicated-VRAM and memory-used fields remain empty. Config size_gb is an estimate, not measured disk bytes, and is not substituted for residency. No Mac power measurements. Sources: https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/results/m2-mac-studio/granite-3.2-8b_default_20260403_235419.json https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/config-m2-mac-studio.yaml https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/run_benchmarks.py#L624-L738 https://github.com/patrickwhelan-uk/llm-inference-benchmarks/blob/11f3f8f931dc97a71499e7b9c2583d010a3c160b/hardware/m2-mac-studio.md https://huggingface.co/mlx-community/granite-3.2-8b-instruct-preview-4bit/blob/c11bf25d3ac421d693a76b6f77f784b2e6ce1e11/config.json https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/benchmark.py https://github.com/ml-explore/mlx-lm/blob/v0.31.2/mlx_lm/generate.py#L658-L753 Structured PP/TG fields are rounded to 0.1 tokens/s; the source means, sample deviations and all three retained trial values above preserve the published precision.