Llama 3.1 Meta-Llama-3.1-70B-Instruct / mlx-lm 0.31.2, 4K lorem, local TTFT 24.093s

SOURCE: https://github.com/abhiram304/mlx-dense-bench/blob/b06ff117ab8c2ffc5bf4581cf06e671742563761/results/matrix.jsonl ; configs: https://github.com/abhiram304/mlx-dense-bench/blob/b06ff117ab8c2ffc5bf4581cf06e671742563761/configs/matrix.json . First-party authors Abhi Ram Salammagari and Koushik Salammagari. Original explanation: https://www.abhiramsalammagari.com/writing/dense-70b-mlx-m3-ultra . Historical recorded runs on 2026-05-03, recovered without rerunning benchmarks. HARDWARE/RUNTIME: One Mac Studio, Apple M3 Ultra, source-labelled 512 GB unified memory. Raw environment total_ram_gb=549.755813888 decimal GB (=512 GiB); this is installed capacity, not dedicated GPU VRAM or used memory. CPU/GPU core counts and chip bin were not published. macOS 26.2 arm64, Python 3.12.13, MLX 0.31.1, mlx-lm 0.31.2, memory API mx; raw mlx_lm_git_sha=null. Dependency lock also pins mlx-metal 0.31.1. One physical participating Metal GPU is implementation-supported, not a per-run device listing: exact MLX v0.31.1 commit ce45c52505c8158ea48d2a54e8caae05efd86bfe selects CopyAllDevices()[0] (default-device fallback) and returns one static Metal Device. mlx-lm v0.31.2 dcbf6e33d135a1b7c6767ca0fe7ebbd23df814a7 creates its generation stream on mx.default_device(); MLX defaults to GPU when available. This is a local non-distributed harness with no device override. Core quantity is distinct from GPU quantity. PROTOCOL: One priming generation followed by one timed generation per model/prompt/target-length cell, temperature 0, seed 42, maximum output 128. Lorem is a Moby-Dick summarization prompt; niah is single-needle retrieval. Chat-template-aware inputs have actual counts exceeding nominal targets. Actual outputs below are source generation_tokens counts, including terminal EOS in stopped rows; max_tokens=128 is not assumed achieved. No speculation, request batching or network server is used. No context-capacity claim is made from the maximum tested prompt. TIMING: Custom bench.py starts perf_counter after clear_cache/reset_peak_memory, before stream_generate. There is no explicit pre-start mx.synchronize. TTFT ends when the first GenerationResponse reaches the Python harness; it includes tokenizer/library setup, prompt work, first-token sampling and event delivery. This is measured local time to first library token event, not HTTP/client/display TTFT or model-load time. Both headline rows record first_text='The'. Some other token events may have empty detokenized text; first-EOS Llama rows deliver a terminal event. Load and warmup are outside the timer. Source prefill_time_s equals this TTFT interval and source prefill_tps=P/TTFT is a combined prompt-to-first-event rate, not isolated native PP; the numeric PP field is deliberately unset. Decode is exactly max(N-1,0)/(timer after generator exhaustion minus first-event timer), including final event/finalization overhead. It is not N/time, whole-request throughput, or a guessed output budget. All 24 raw rows pass these arithmetic identities. Library internal prompt_tps also includes the first generated-token boundary, so it is not isolated PP either. Library internal rates are separate cross-checks, not mixed into headline values. SELECTED HEADLINE: llama70b_lorem_4k, config_hash 9181be8c0a1f86de; actual prompt 4138, actual output count 128, timed decode numerator 127, TTFT 24.093307125149s, decode interval 8.512535624905s, TG 14.919173980129 t/s. Timestamp 2026-05-03T09:22:34.492652+00:00. No error/OOM/suspect_swap flag. This is one observation, not a repeated-run mean. FULL MODEL SWEEP (source values rounded here only; raw JSON linked above retains precision): lorem_4k: P=4138, N=128, timed decode numerator=127; TTFT=24.093307125s, decode interval=8.512535625s, TG=14.919173980 t/s. niah_4k: P=4157, N=14, timed decode numerator=13; TTFT=24.095836417s, decode interval=0.879073167s, TG=14.788302599 t/s [early EOS]. lorem_8k: P=8234, N=128, timed decode numerator=127; TTFT=49.453326292s, decode interval=9.064785625s, TG=14.010259619 t/s. niah_8k: P=8253, N=14, timed decode numerator=13; TTFT=49.469442542s, decode interval=0.935589458s, TG=13.894983415 t/s [early EOS]. lorem_16k: P=16426, N=128, timed decode numerator=127; TTFT=105.872540250s, decode interval=9.808121500s, TG=12.948452973 t/s. niah_16k: P=16445, N=14, timed decode numerator=13; TTFT=105.885152042s, decode interval=1.010276250s, TG=12.867767602 t/s [early EOS]. lorem_32k: P=32810, N=128, timed decode numerator=127; TTFT=241.514821250s, decode interval=11.329528791s, TG=11.209645374 t/s. niah_32k: P=32829, N=14, timed decode numerator=13; TTFT=241.538686208s, decode interval=1.166795584s, TG=11.141625985 t/s [early EOS]. lorem_64k: P=65578, N=128, timed decode numerator=127; TTFT=604.677038417s, decode interval=14.617659750s, TG=8.688121229 t/s. niah_64k: P=65597, N=128, timed decode numerator=127; TTFT=604.581685083s, decode interval=14.611410375s, TG=8.691837183 t/s. lorem_128k: P=131114, N=1, timed decode numerator=0; TTFT=1809.205743792s, decode interval=0.008948083s, TG=0.000000000 t/s [first EOS; decode invalid]. niah_128k: P=131133, N=1, timed decode numerator=0; TTFT=1806.923823916s, decode interval=0.009040625s, TG=0.000000000 t/s [first EOS; decode invalid]. MEMORY: Headline MLX allocation high-water mark 82.790731804 decimal GB and active-at-end 80.58895156 GB are process allocator measurements, not weight bytes or physical residency. The harness model cache retains previously loaded models; especially Llama's peak can include the earlier Qwen model. Do not enter these as isolated model residency or VRAM. Swap guard is heuristic, not a measured no-swap proof. ARTIFACT: Source config explicitly names mlx-community/Meta-Llama-3.1-70B-Instruct-4bit. Current immutable reference https://huggingface.co/mlx-community/Meta-Llama-3.1-70B-Instruct-4bit/tree/7772c93cf077b642f5503dd8d763a4176d7d406c; config has 4-bit, group_size=64 MLX quantization. Current eight safetensors shards total 39688567605 bytes on disk per HF metadata; hashes and sizes retained in audit manifest. No weight bytes downloaded. Historical tested revision/hash is unpublished, so the pinned snapshot is a same-publisher/model reference, not proof of historical byte identity. Source snapshot is also not a tested runtime/weight hash. CAVEATS: Full historical harness commit, exact installed runtime build hash, CPU thread count, GPU-core bin, actual allocated context capacity and historical model bytes remain unknown. Current harness was inspected at the pinned source snapshot; raw arithmetic confirms the stated timing convention. No new measurements were made. Code proof: https://github.com/ml-explore/mlx/blob/ce45c52505c8158ea48d2a54e8caae05efd86bfe/mlx/backend/metal/device.cpp ; https://github.com/ml-explore/mlx-lm/blob/dcbf6e33d135a1b7c6767ca0fe7ebbd23df814a7/mlx_lm/generate.py ; timer https://github.com/abhiram304/mlx-dense-bench/blob/b06ff117ab8c2ffc5bf4581cf06e671742563761/bench.py . Both 128K Llama inputs (131114/131133 actual tokens) exceed the current reference config limit 131072 and immediately emit EOS. Their source-reported first-event latency is retained only as an EOS-only diagnostic, not real first-output TTFT; zero decode throughput is invalid, not a usable TG result. LlamaPerf structured generation rate is rounded to 0.1 tokens/s. The measured local first-token-event latency and full timing definitions are retained above; the detailed form has no dedicated TTFT field.