Llama 2 reports

28 reports. These are individual setups, not a controlled benchmark. Llama 2 speeds by hardware →

Llama 2 7B Chat / Q5_K_S / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 40.3 tokens/s
prompt processing (prefill):: 422.9 tokens/s
quant:: Q5_K_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 422.92 +/- 1.60 tokens/s and TG128 40.34 +/- 0.25 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 4.33 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: Q5_K_S. The source native model descriptor is llama 7B Q5_K - Small. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / Q5_K_M / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 39.7 tokens/s
prompt processing (prefill):: 428.6 tokens/s
quant:: Q5_K_M (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 428.63 +/- 0.80 tokens/s and TG128 39.67 +/- 0.05 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 4.45 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: Q5_K_M. The source native model descriptor is llama 7B Q5_K - Medium. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / Q5_1 / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 41.9 tokens/s
prompt processing (prefill):: 455.7 tokens/s
quant:: Q5_1 (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 455.69 +/- 3.64 tokens/s and TG128 41.92 +/- 0.18 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 4.72 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: Q5_1. The source native model descriptor is llama 7B Q5_1. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / Q6_K / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 39.0 tokens/s
prompt processing (prefill):: 444.3 tokens/s
quant:: Q6_K (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 444.28 +/- 2.46 tokens/s and TG128 39.03 +/- 0.20 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 5.15 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: Q6_K. The source native model descriptor is llama 7B Q6_K. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / Q8_0 / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 40.4 tokens/s
prompt processing (prefill):: 547.3 tokens/s
quant:: Q8_0 (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 547.28 +/- 0.46 tokens/s and TG128 40.38 +/- 0.01 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 6.67 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: Q8_0. The source native model descriptor is llama 7B Q8_0. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / F16 / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 23.2 tokens/s
prompt processing (prefill):: 603.1 tokens/s
quant:: F16 (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 603.07 +/- 4.25 tokens/s and TG128 23.18 +/- 0.10 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 12.55 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: F16. The source native model descriptor is llama 7B F16. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / F32 / b2296 Metal / native PP512 TG128

M1 Max 64GB · llama.cpp b2296 (d5ab2975) / Metal

generation:: 12.6 tokens/s
prompt processing (prefill):: 549.3 tokens/s
quant:: F32 (GGUF)

First-party measurement: https://beebopkim.github.io/2024/03/09/Benchmarks-for-lots-of-quantization-types-in-llama-cpp/ . Beebopkim reports Llama 2 7B Chat on an M1 Max Mac Studio, native PP512 549.31 +/- 1.32 tokens/s and TG128 12.60 +/- 0.08 tokens/s, Metal backend, ngl 99. The article was posted March 9, 2024 and updated March 10; an exact measurement timestamp is not supplied. These are separate synthetic llama-bench phases, not one 512-input/128-output chat request. TTFT was not measured and is unknown. Hardware and memory: the author explicitly identifies Mac Studio M1 Max, 32 GPU cores and 64 GB system unified memory. One local Metal GPU participates, established by the exact-runtime execution path below; 32 is a core count, not a GPU quantity. CPU core count 10 is based on Apple's fixed M1 Max Mac Studio specification, rather than run telemetry: https://support.apple.com/en-za/111900 . The source's generic statement that up to 48 GB is usable by the GPU is not a measured allocation or dedicated VRAM. Dedicated VRAM and measured memory use are not established. The native table's 25.10 GiB is llama-bench model tensor size, not installed RAM, a disk-file byte count, or measured resident memory. OS revision, clocks, thermals, power, swap, background workload and storage are unreported. Model and artifact: Quantization label: F32. The source native model descriptor is llama 7B all F32. The source identifies https://huggingface.co/meta-llama/Llama-2-7b-chat-hf as the starting model. That link is a BASE/REFERENCE MODEL ONLY, not the tested quantized artifact. These are the author's locally converted/quantized weights. Exact output filenames, public artifact URLs, tested weight hashes, conversion command, calibration-matrix hash and base-model revision are unknown. Format GGUF is supported by this successful run's exact b2296 loader, which uses gguf_init_from_file and rejects failed GGUF reads (https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2511-L2526). GGUF version is unknown. The table rounds parameter count to 6.74B; do not treat that as an exact tensor count. Runtime and settings: source declares llama.cpp b2296, d5ab29757ebc59a30f03e408294ec20628a6374e. Binary hash, compiler, build flags and any local changes are unknown. Metal and positive ngl 99 are printed for this measurement. Full command, CPU threads, batch, KV precision, repetition count and raw repetitions are not supplied. The upstream benchmark defaults include batch 512, F16 K/V and 5 repetitions, but defaults are not independent run telemetry and are not asserted as measured settings. No explicit speculative-decoding setup is reported. Native metric and context audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/examples/llama-bench/llama-bench.cpp#L1103-L1124 uses BOS token IDs for its synthetic work: PP evaluates 512 tokens in batches; TG makes 128 successive one-token evaluations, rather than sampling response text. The timing loop at lines 1207-1229 performs a short warm-up, clears KV before each repetition, and times phase execution only. Model loading and initialization are outside that interval. The high-resolution clock at lines 26-30 returns nanoseconds; lines 663-675 compute arithmetic mean and sample standard deviation from token-count/elapsed-time rates. Metal waits for command-buffer completion at https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L2421-L2433. The benchmark constructs separate PP512/TG0 and PP0/TG128 cases, and requests n_ctx = n_prompt + n_gen (lines 514-584). Thus phase contexts are 512 and 128, starting at depth 0. Application configured context is unknown. The sum 640 and the base model's 4096 maximum are not benchmark context settings; neither prompt duration nor inverse throughput is a TTFT measurement. Exact participating GPU proof: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L11832-L11843 initializes one Metal backend when n_gpu_layers is positive. https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/ggml-metal.m#L255-L275 selects one system-default MTLDevice and builds its execution queue; lines 2779-2795 wrap this context in one Metal backend. The device inventory loop merely lists devices and does not execute on all of them. Source-bound successful Metal/ngl99 output together with this pinned single-device path establishes exactly one participating GPU, without inferring quantity from the 32 GPU cores. A separate startup inventory and utilization trace were not published. Quantization-label audit: https://github.com/ggml-org/llama.cpp/blob/d5ab29757ebc59a30f03e408294ec20628a6374e/llama.cpp#L2912-L2949 maps the printed native descriptors to the runtime's file types. Results are preserved as the author's benchmarks and have not been independently rerun. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

Llama 2 7B Chat / Q4_0 GGJT v3 / M1 Ultra 64GB / b944 CPU-Accelerate PP and Metal TG

M1 Ultra 64GB · llama.cpp b944 (8183159) / CPU-Accelerate prefill; Metal decode · 512 ctx

generation:: 72.4 tokens/s
prompt processing (prefill):: 37.8 tokens/s
quant:: Q4_0

Michael Tedder's August 4, 2023 first-party demonstration: https://speakerdeck.com/falken/llama-dot-cpp-plus-metal-on-m1-ultra . Slides 3, 9, 10 identify a 2022 Mac Studio M1 Ultra, 64 GB unified RAM, macOS Ventura 13.4.1, and Llama 2 7B Chat Q4_0. GPU core variant unknown. Runtime uses 16 CPU threads of 20 available logical threads. Model filename llama-2-7b-chat.ggmlv3.q4_0.bin, legacy GGML/GGJT v3, 3,791,725,184 bytes. The file size is not RAM. Historical download origin, checksum and revision unknown. https://huggingface.co/meta-llama/Llama-2-7b-chat-hf is a verified base/reference-only link, not the tested quantized artifact. Native llama.cpp build 944, commit 8183159cf3def112f6d1fe94815fce70e1bffa12. Source PP 37.75 tokens/s from 19 tokens in 503.32 ms. Source TG 72.39 tokens/s from 27 evaluation runs in 372.99 ms. There are 28 sampling runs, not a verified 28-token output count. Context 512, batch 512, ngl 32, n_predict=-1, n_keep 0, temperature 0.1, seed 1691074574. Load 4943.32 ms, sampling 20.34 ms and total 899.05 ms are separately reported. Prompt asks Mount Fuji's height; displayed response ends normally. No measured TTFT. Only one displayed run; repetition count and hardware frequencies, thermals, power and swap are unknown. Exactly one participating GPU is established by the positive-offload command, built-in unified-memory Metal startup, and the same build's implementation. https://github.com/ggml-org/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/llama.cpp#L3321-L3324 invokes one Metal initializer when n_gpu_layers>0; https://github.com/ggml-org/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/ggml-metal.m#L95-L104 selects one MTLCreateSystemDefaultDevice and creates its command queue. GPU core quantity and GPU device quantity are separate. The printed 49152.00 MB recommended working-set limit is not installed RAM or measured memory use. Important phase distinction: https://github.com/ggml-org/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/llama.cpp#L1832-L1864 dispatches single-token evaluation to Metal, while multi-token prompt evaluation falls back to the CPU/Accelerate graph implementation. These are native phases from that hybrid 2023 runtime. The PP number must not be described as GPU-prefill throughput. Metal completion is awaited before returning: https://github.com/ggml-org/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/ggml-metal.m#L1112-L1120 . Timing uses ggml_time_us, backed by clock_gettime(CLOCK_MONOTONIC) and converted to microseconds on this platform: https://github.com/ggml-org/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/ggml.c#L544-L555 . llama_eval_internal starts its timer before graph construction and accumulates after execution and logits/embedding extraction, counting N>1 tokens as prompt evaluation and N=1 calls as evaluation runs: https://github.com/ggml-org/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/llama.cpp#L1785-L1787 and #L1889-L1918. Sampling and user-interface delivery have different boundaries. The pinned main implementation performs a one-BOS-token warmup and resets timings before generation: https://github.com/ggml-org/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/examples/main/main.cpp#L409-L414 . That is code-derived behavior, not proof of a cold application start. The unmodified context default is F16 KV; the source excerpt does not independently print KV precision. Local patches and executable hash are unknown. Later slides' aggregate 7B/13B durations are not promoted to PP, TG or TTFT. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.