Llama 2 reports

28 reports. These are individual setups, not a controlled benchmark. Llama 2 speeds by hardware →

Llama 2 7B Chat / IQ1_S / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 53.0 tokens/s
prompt processing (prefill):: 499.1 tokens/s
quant:: IQ1_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 499.09 +/- 0.56 tokens/s and TG128 53.01 +/- 0.33 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 1.42 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: IQ1_S. The source native model descriptor is llama 7B IQ1_S - 1.5625 bpw. The author used importance-matrix calibration from 20k_random_data.txt (https://github.com/ggerganov/llama.cpp/files/13970111/20k_random_data.txt) for I-quants; the generated calibration matrix and quantized weight hashes are not published. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / IQ2_XXS / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 50.5 tokens/s
prompt processing (prefill):: 462.1 tokens/s
quant:: IQ2_XXS (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 462.08 +/- 0.30 tokens/s and TG128 50.45 +/- 0.03 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 1.73 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: IQ2_XXS. The source native model descriptor is llama 7B IQ2_XXS - 2.0625 bpw. The author used importance-matrix calibration from 20k_random_data.txt (https://github.com/ggerganov/llama.cpp/files/13970111/20k_random_data.txt) for I-quants; the generated calibration matrix and quantized weight hashes are not published. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / IQ2_XS / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 48.4 tokens/s
prompt processing (prefill):: 472.2 tokens/s
quant:: IQ2_XS (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 472.16 +/- 0.41 tokens/s and TG128 48.42 +/- 0.03 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 1.89 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: IQ2_XS. The source native model descriptor is llama 7B IQ2_XS - 2.3125 bpw. The author used importance-matrix calibration from 20k_random_data.txt (https://github.com/ggerganov/llama.cpp/files/13970111/20k_random_data.txt) for I-quants; the generated calibration matrix and quantized weight hashes are not published. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / IQ2_S / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 47.3 tokens/s
prompt processing (prefill):: 469.5 tokens/s
quant:: IQ2_S (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 469.49 +/- 1.99 tokens/s and TG128 47.32 +/- 0.26 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 2.05 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: IQ2_S. The source native model descriptor is llama 7B IQ2_S - 2.5 bpw. The author used importance-matrix calibration from 20k_random_data.txt (https://github.com/ggerganov/llama.cpp/files/13970111/20k_random_data.txt) for I-quants; the generated calibration matrix and quantized weight hashes are not published. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / Q2_K_S / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 62.5 tokens/s
prompt processing (prefill):: 512.2 tokens/s
quant:: Q2_K_S (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 512.21 +/- 0.57 tokens/s and TG128 62.49 +/- 0.20 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 2.16 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: Q2_K_S. The source native model descriptor is llama 7B Q2_K - Small. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / IQ2_M / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 38.4 tokens/s
prompt processing (prefill):: 462.4 tokens/s
quant:: IQ2_M (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 462.36 +/- 0.45 tokens/s and TG128 38.38 +/- 0.03 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 2.20 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: IQ2_M. The source native model descriptor is llama 7B IQ2_M - 2.7 bpw. The author used importance-matrix calibration from 20k_random_data.txt (https://github.com/ggerganov/llama.cpp/files/13970111/20k_random_data.txt) for I-quants; the generated calibration matrix and quantized weight hashes are not published. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / Q2_K / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 51.6 tokens/s
prompt processing (prefill):: 493.4 tokens/s
quant:: Q2_K (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 493.38 +/- 0.42 tokens/s and TG128 51.57 +/- 0.11 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 2.36 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: Q2_K. The source native model descriptor is llama 7B Q2_K - Medium. Exact-build llama.cpp prints Q2_K - Medium for LLAMA_FTYPE_MOSTLY_Q2_K; this is Q2_K, not an invented Q2_K_M label. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / IQ3_XXS / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 46.4 tokens/s
prompt processing (prefill):: 470.0 tokens/s
quant:: IQ3_XXS (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 470.03 +/- 0.38 tokens/s and TG128 46.44 +/- 0.07 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 2.41 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: IQ3_XXS. The source native model descriptor is llama 7B IQ3_XXS - 3.0625 bpw. The author used importance-matrix calibration from 20k_random_data.txt (https://github.com/ggerganov/llama.cpp/files/13970111/20k_random_data.txt) for I-quants; the generated calibration matrix and quantized weight hashes are not published. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / IQ3_XS / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 47.9 tokens/s
prompt processing (prefill):: 464.9 tokens/s
quant:: IQ3_XS (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 464.88 +/- 0.42 tokens/s and TG128 47.87 +/- 0.05 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 2.60 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: IQ3_XS. The source native model descriptor is llama 7B IQ3_XS - 3.3 bpw. The author used importance-matrix calibration from 20k_random_data.txt (https://github.com/ggerganov/llama.cpp/files/13970111/20k_random_data.txt) for I-quants; the generated calibration matrix and quantized weight hashes are not published. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / Q3_K_S / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 42.6 tokens/s
prompt processing (prefill):: 466.8 tokens/s
quant:: Q3_K_S (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 466.80 +/- 0.27 tokens/s and TG128 42.60 +/- 0.05 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 2.75 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: Q3_K_S. The source native model descriptor is llama 7B Q3_K - Small. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / IQ3_S / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 47.7 tokens/s
prompt processing (prefill):: 456.1 tokens/s
quant:: IQ3_S (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 456.12 +/- 0.32 tokens/s and TG128 47.74 +/- 0.04 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 2.75 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: IQ3_S. The source native model descriptor is llama 7B IQ3_S - 3.4375 bpw. The author used importance-matrix calibration from 20k_random_data.txt (https://github.com/ggerganov/llama.cpp/files/13970111/20k_random_data.txt) for I-quants; the generated calibration matrix and quantized weight hashes are not published. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / IQ3_M / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 47.7 tokens/s
prompt processing (prefill):: 459.5 tokens/s
quant:: IQ3_M (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 459.51 +/- 0.40 tokens/s and TG128 47.68 +/- 0.04 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 2.90 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: IQ3_M. The source native model descriptor is llama 7B IQ3_S mix - 3.66 bpw. Exact-build llama.cpp maps the printed IQ3_S mix descriptor to LLAMA_FTYPE_MOSTLY_IQ3_M; it is distinct from IQ3_S. The author used importance-matrix calibration from 20k_random_data.txt (https://github.com/ggerganov/llama.cpp/files/13970111/20k_random_data.txt) for I-quants; the generated calibration matrix and quantized weight hashes are not published. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / Q3_K_M / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 44.2 tokens/s
prompt processing (prefill):: 469.6 tokens/s
quant:: Q3_K_M (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 469.64 +/- 0.61 tokens/s and TG128 44.19 +/- 0.01 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 3.07 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: Q3_K_M. The source native model descriptor is llama 7B Q3_K - Medium. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / Q3_K_L / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 40.9 tokens/s
prompt processing (prefill):: 449.5 tokens/s
quant:: Q3_K_L (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 449.53 +/- 0.64 tokens/s and TG128 40.93 +/- 0.02 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 3.35 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: Q3_K_L. The source native model descriptor is llama 7B Q3_K - Large. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / IQ4_XS / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 51.9 tokens/s
prompt processing (prefill):: 491.7 tokens/s
quant:: IQ4_XS (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 491.69 +/- 0.44 tokens/s and TG128 51.86 +/- 0.06 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 3.37 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: IQ4_XS. The source native model descriptor is llama 7B IQ4_XS - 4.25 bpw. The author used importance-matrix calibration from 20k_random_data.txt (https://github.com/ggerganov/llama.cpp/files/13970111/20k_random_data.txt) for I-quants; the generated calibration matrix and quantized weight hashes are not published. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / IQ4_NL / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 53.2 tokens/s
prompt processing (prefill):: 520.6 tokens/s
quant:: IQ4_NL (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 520.62 +/- 0.52 tokens/s and TG128 53.15 +/- 0.05 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 3.56 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: IQ4_NL. The source native model descriptor is llama 7B IQ4_NL - 4.5 bpw. The author used importance-matrix calibration from 20k_random_data.txt (https://github.com/ggerganov/llama.cpp/files/13970111/20k_random_data.txt) for I-quants; the generated calibration matrix and quantized weight hashes are not published. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / Q4_0 / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 61.4 tokens/s
prompt processing (prefill):: 536.4 tokens/s
quant:: Q4_0 (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 536.36 +/- 2.01 tokens/s and TG128 61.39 +/- 0.59 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 3.57 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: Q4_0. The source native model descriptor is llama 7B Q4_0. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / Q4_K_S / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 52.0 tokens/s
prompt processing (prefill):: 474.3 tokens/s
quant:: Q4_K_S (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 474.25 +/- 0.59 tokens/s and TG128 52.02 +/- 0.50 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 3.59 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: Q4_K_S. The source native model descriptor is llama 7B Q4_K - Small. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / Q4_K_M / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 49.2 tokens/s
prompt processing (prefill):: 469.9 tokens/s
quant:: Q4_K_M (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 469.91 +/- 2.67 tokens/s and TG128 49.24 +/- 0.30 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 3.80 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: Q4_K_M. The source native model descriptor is llama 7B Q4_K - Medium. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / Q4_1 / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 57.2 tokens/s
prompt processing (prefill):: 534.1 tokens/s
quant:: Q4_1 (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 534.08 +/- 1.73 tokens/s and TG128 57.19 +/- 0.51 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 3.95 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: Q4_1. The source native model descriptor is llama 7B Q4_1. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.