Qwen2.5 72B-Instruct / mlx-lm 0.31.2, 4K lorem, local TTFT 24.861s

SOURCE: https://github.com/abhiram304/mlx-dense-bench/blob/b06ff117ab8c2ffc5bf4581cf06e671742563761/results/matrix.jsonl ; configs: https://github.com/abhiram304/mlx-dense-bench/blob/b06ff117ab8c2ffc5bf4581cf06e671742563761/configs/matrix.json . First-party authors Abhi Ram Salammagari and Koushik Salammagari. Original explanation: https://www.abhiramsalammagari.com/writing/dense-70b-mlx-m3-ultra . Historical recorded runs on 2026-05-03, recovered without rerunning benchmarks. HARDWARE/RUNTIME: One Mac Studio, Apple M3 Ultra, source-labelled 512 GB unified memory. Raw environment total_ram_gb=549.755813888 decimal GB (=512 GiB); this is installed capacity, not dedicated GPU VRAM or used memory. CPU/GPU core counts and chip bin were not published. macOS 26.2 arm64, Python 3.12.13, MLX 0.31.1, mlx-lm 0.31.2, memory API mx; raw mlx_lm_git_sha=null. Dependency lock also pins mlx-metal 0.31.1. One physical participating Metal GPU is implementation-supported, not a per-run device listing: exact MLX v0.31.1 commit ce45c52505c8158ea48d2a54e8caae05efd86bfe selects CopyAllDevices()[0] (default-device fallback) and returns one static Metal Device. mlx-lm v0.31.2 dcbf6e33d135a1b7c6767ca0fe7ebbd23df814a7 creates its generation stream on mx.default_device(); MLX defaults to GPU when available. This is a local non-distributed harness with no device override. Core quantity is distinct from GPU quantity. PROTOCOL: One priming generation followed by one timed generation per model/prompt/target-length cell, temperature 0, seed 42, maximum output 128. Lorem is a Moby-Dick summarization prompt; niah is single-needle retrieval. Chat-template-aware inputs have actual counts exceeding nominal targets. Actual outputs below are source generation_tokens counts, including terminal EOS in stopped rows; max_tokens=128 is not assumed achieved. No speculation, request batching or network server is used. No context-capacity claim is made from the maximum tested prompt. TIMING: Custom bench.py starts perf_counter after clear_cache/reset_peak_memory, before stream_generate. There is no explicit pre-start mx.synchronize. TTFT ends when the first GenerationResponse reaches the Python harness; it includes tokenizer/library setup, prompt work, first-token sampling and event delivery. This is measured local time to first library token event, not HTTP/client/display TTFT or model-load time. Both headline rows record first_text='The'. Some other token events may have empty detokenized text; first-EOS Llama rows deliver a terminal event. Load and warmup are outside the timer. Source prefill_time_s equals this TTFT interval and source prefill_tps=P/TTFT is a combined prompt-to-first-event rate, not isolated native PP; the numeric PP field is deliberately unset. Decode is exactly max(N-1,0)/(timer after generator exhaustion minus first-event timer), including final event/finalization overhead. It is not N/time, whole-request throughput, or a guessed output budget. All 24 raw rows pass these arithmetic identities. Library internal prompt_tps also includes the first generated-token boundary, so it is not isolated PP either. Library internal rates are separate cross-checks, not mixed into headline values. SELECTED HEADLINE: qwen72b_lorem_4k, config_hash 7d703b8e092c14de; actual prompt 4132, actual output count 128, timed decode numerator 127, TTFT 24.861239207909s, decode interval 8.861839125166s, TG 14.331110981167 t/s. Timestamp 2026-05-03T07:27:58.017497+00:00. No error/OOM/suspect_swap flag. This is one observation, not a repeated-run mean. FULL MODEL SWEEP (source values rounded here only; raw JSON linked above retains precision): lorem_4k: P=4132, N=128, timed decode numerator=127; TTFT=24.861239208s, decode interval=8.861839125s, TG=14.331110981 t/s. niah_4k: P=4154, N=95, timed decode numerator=94; TTFT=24.751058917s, decode interval=6.551007458s, TG=14.348938023 t/s [early EOS]. lorem_8k: P=8228, N=128, timed decode numerator=127; TTFT=50.718213750s, decode interval=9.398965250s, TG=13.512125710 t/s. niah_8k: P=8250, N=75, timed decode numerator=74; TTFT=50.722647250s, decode interval=5.488790917s, TG=13.482022019 t/s [early EOS]. lorem_16k: P=16420, N=128, timed decode numerator=127; TTFT=108.380474084s, decode interval=10.164079291s, TG=12.494983201 t/s. niah_16k: P=16442, N=56, timed decode numerator=55; TTFT=108.391879959s, decode interval=4.393770958s, TG=12.517721230 t/s [early EOS]. lorem_32k: P=32804, N=128, timed decode numerator=127; TTFT=248.996037833s, decode interval=11.657598750s, TG=10.894181788 t/s. niah_32k: P=32826, N=24, timed decode numerator=23; TTFT=246.696812208s, decode interval=2.121898834s, TG=10.839348056 t/s [early EOS]. lorem_64k: P=65572, N=128, timed decode numerator=127; TTFT=614.634077833s, decode interval=14.946048458s, TG=8.497229241 t/s. niah_64k: P=65594, N=66, timed decode numerator=65; TTFT=614.760152500s, decode interval=7.664965333s, TG=8.480142724 t/s [early EOS]. lorem_128k: P=131108, N=128, timed decode numerator=127; TTFT=1729.998797334s, decode interval=20.970838750s, TG=6.056028636 t/s. niah_128k: P=131130, N=99, timed decode numerator=98; TTFT=1732.439958042s, decode interval=16.178016917s, TG=6.057602764 t/s [early EOS]. MEMORY: Headline MLX allocation high-water mark 43.117035548 decimal GB and active-at-end 40.900624392 GB are process allocator measurements, not weight bytes or physical residency. The harness model cache retains previously loaded models; especially Llama's peak can include the earlier Qwen model. Do not enter these as isolated model residency or VRAM. Swap guard is heuristic, not a measured no-swap proof. ARTIFACT: Source config explicitly names mlx-community/Qwen2.5-72B-Instruct-4bit. Current immutable reference https://huggingface.co/mlx-community/Qwen2.5-72B-Instruct-4bit/tree/36a74b07390031bb18c9f12fb7c06699bc6273c4; config has 4-bit, group_size=64 MLX quantization. Current eight safetensors shards total 40900550929 bytes on disk per HF metadata; hashes and sizes retained in audit manifest. No weight bytes downloaded. Historical tested revision/hash is unpublished, so the pinned snapshot is a same-publisher/model reference, not proof of historical byte identity. Source snapshot is also not a tested runtime/weight hash. CAVEATS: Full historical harness commit, exact installed runtime build hash, CPU thread count, GPU-core bin, actual allocated context capacity and historical model bytes remain unknown. Current harness was inspected at the pinned source snapshot; raw arithmetic confirms the stated timing convention. No new measurements were made. Code proof: https://github.com/ml-explore/mlx/blob/ce45c52505c8158ea48d2a54e8caae05efd86bfe/mlx/backend/metal/device.cpp ; https://github.com/ml-explore/mlx-lm/blob/dcbf6e33d135a1b7c6767ca0fe7ebbd23df814a7/mlx_lm/generate.py ; timer https://github.com/abhiram304/mlx-dense-bench/blob/b06ff117ab8c2ffc5bf4581cf06e671742563761/bench.py . Current pinned Qwen config max_position_embeddings=32768, sliding_window=131072 with use_sliding_window=false; this does not support the blog’s unqualified native-128K claim. Raw 128K output counts are lorem=128 and niah=99; the prose swaps these labels, so raw rows take precedence. LlamaPerf structured generation rate is rounded to 0.1 tokens/s. The measured local first-token-event latency and full timing definitions are retained above; the detailed form has no dedicated TTFT field.