Llama 2 7B Chat / IQ1_S / b2296 Metal / native PP512 TG128
M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal
- generation:
- 53.0 tokens/s
- prompt processing (prefill):
- 499.1 tokens/s
- quant:
- IQ1_S (GGUF)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 499.09 +/- 0.56 tokens/s and TG128 53.01 +/- 0.33 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 1.42 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: IQ1_S. The source native model descriptor is llama 7B IQ1_S - 1.5625 bpw. The author used importance-matrix calibration from 20k_random_data.txt (https://github.com/ggerganov/llama.cpp/files/13970111/20k_random_data.txt) for I-quants; the generated calibration matrix and quantized weight hashes are not published. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.