Local model performance on your hardware

generation:

52.7 tokens/s

prompt processing (prefill):

683.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical John Rocky benchmark on Apple M4 Max, 128 GiB unified RAM, macOS 27.0 build 26A5416b. This report selects Falcon-H1 7B-Instruct, Q4_K_M, in the source's September 5 campaign. The actual JSON timestamps are 2026-09-04T21:13:54Z for PP and 2026-09-04T21:13:57Z for TG. Preserve both dates: the source local/campaign label is September 5, while the raw clock is September 4 UTC. No timing or date is inferred from webpage freshness. Hardware identity: the repository identifies the author's m4max host as Mac Studio Mac16,9, with chassis verification recorded in the August 15 ENV and the shared renderer's DEVICE_ALIASES. That earlier hardware inventory states a 40-core GPU and 128 GB RAM. This run directly records Apple M4 Max and the campaign declares 128 GiB; it does not independently log a chassis identifier or core count. The September 4 prose still says lid open, which conflicts with the later repository chassis correction. The headline uses M4 Max and 128 GiB; no physical core-count inference is made from 12 threads. Dedicated VRAM is inapplicable/unknown as a separate pool; shared RAM capacity is not model memory. One participating GPU is run-backed. Each native record lists gpu_info exactly Apple M4 Max, backends BLAS, MTL, devices auto, 99 GPU layers, main_gpu 0 and no CPU-MoE override. At the logged 15f786e65 revision, llama-bench get_gpu_info enumerates every GPU/IGPU description without deduplicating names. The accompanying stderr initializes only MTL0; BLAS and CPU are separate backend types. This establishes one enumerated and selected inference GPU, rather than converting physical computer count to GPU count. Runtime: Homebrew llama.cpp build 8680, commit 15f786e65, 12 CPU threads. Other local build changes and executable hash are unknown. Config: n_batch 2048, n_ubatch 512, K/V f16, Flash Attention on, mmap on, direct IO off, layer split, no_kv_offload false, no_host false, no tensor buffer overrides, no speculative drafter. The source script and raw parameters agree. Native workload: standalone synthetic PP512 and TG256, depth 0, single sequence. These are separate tests, not one 512-input/256-output chat and not a 256-token generation after a 512-token prompt. The logged build requests context as prompt + generation + depth separately (512 for PP, 256 for TG), but actual allocated/rounded context is not logged and the top-level context field stays unknown. Three measured repetitions per test, after warmup. KV/recurrent state is cleared before each repetition. PP times 512 synthetic input IDs including synchronization; TG times 256 serial single-token decode calls with synchronization. The numerator is native processed tokens, not display chunks or a chat tokenizer estimate. Load and initial warmup are outside the measured phase. Mean and sample SD are calculated from each trial's tokens divided by nanoseconds; both were independently checked. No TTFT, HTTP or user-facing first-token latency was measured. Headline native JSON results: PP 682.958202 ± 4.356899 tokens/s; TG 52.696786 ± 0.186655 tokens/s. Raw trial nanoseconds are PP [754395416, 749867291, 744837833] and TG [4848657667, 4847487792, 4877920416]. Three separate invocations are available: this JSON run, a text-table run (PP512 686.96 ± 1.16; TG256 52.91 ± 0.4), and an RSS run (PP512 683.65 ± 2.1; TG256 53.03 ± 0.93). They are retained as companion observations of this configuration rather than inflated into new reports. The source Markdown uses the text run, whereas its structured JSON and README use the JSON run; differences are separate executions, not rounding errors. Do not pair timing from one with memory from another silently. Model identity: https://huggingface.co/tiiuae/Falcon-H1-7B-Instruct-GGUF/blob/058c8c8f08e57da131ba5f070f9ff1280d141c39/Falcon-H1-7B-Instruct-Q4_K_M.gguf Exact source-recorded HF snapshot and GGUF filename; the pinned artifact returned HTTP 200 to HEAD without downloading weights. No measured local-file hash is published. Raw llama_model_size tensor storage is 4592622104 bytes, and native tensor parameter count is 7585654880 (7.58565488 B). Those are engine metadata, not an independent nominal model-card parameter verification. The publisher's advertised size label is retained. HTTP disk-object size is 4598344960 bytes where available; neither quantity is resident RAM. The separate RSS invocation reports maximum process RSS 4889559040 bytes, including mapped weights and runtime; it is not model-only memory and is not used as the selected JSON run's memory field. No allocation-based or measured power number is supplied. The synthetic benchmark performs no language-quality evaluation. Separate completion checks in the source described this llama.cpp arm as coherent, but that is a distinct prompt and call. Source MLX controls have some known degraded outputs and different quantization and prompt shapes; their rates are not merged into this arm. Sources: https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/raw-2026-09-05/falcon-h1-7b-q4km.llama-bench.json https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/raw-2026-09-05/falcon-h1-7b-q4km.llama-bench.stderr https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/hybrid-apple-silicon-bench-2.md https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/hybrid-apple-silicon-bench-2.json https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/raw-2026-09-05/bench_llama.sh https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/scripts/bench_common.py https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/raw/2026-08-15-muse-glimmer-30b-3way/ENV.md https://github.com/ggml-org/llama.cpp/blob/15f786e65/tools/llama-bench/llama-bench.cpp Structured rate fields are rounded to the form’s 0.1 token/s precision; exact source means, standard deviations and separate workloads are retained above.

generation:

86.4 tokens/s

prompt processing (prefill):

1297.7 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical John Rocky benchmark on Apple M4 Max, 128 GiB unified RAM, macOS 27.0 build 26A5416b. This report selects Falcon-H1 3B-Instruct, Q4_K_M, in the source's September 5 campaign. The actual JSON timestamps are 2026-09-04T21:42:41Z for PP and 2026-09-04T21:42:42Z for TG. Preserve both dates: the source local/campaign label is September 5, while the raw clock is September 4 UTC. No timing or date is inferred from webpage freshness. Hardware identity: the repository identifies the author's m4max host as Mac Studio Mac16,9, with chassis verification recorded in the August 15 ENV and the shared renderer's DEVICE_ALIASES. That earlier hardware inventory states a 40-core GPU and 128 GB RAM. This run directly records Apple M4 Max and the campaign declares 128 GiB; it does not independently log a chassis identifier or core count. The September 4 prose still says lid open, which conflicts with the later repository chassis correction. The headline uses M4 Max and 128 GiB; no physical core-count inference is made from 12 threads. Dedicated VRAM is inapplicable/unknown as a separate pool; shared RAM capacity is not model memory. One participating GPU is run-backed. Each native record lists gpu_info exactly Apple M4 Max, backends BLAS, MTL, devices auto, 99 GPU layers, main_gpu 0 and no CPU-MoE override. At the logged 15f786e65 revision, llama-bench get_gpu_info enumerates every GPU/IGPU description without deduplicating names. The accompanying stderr initializes only MTL0; BLAS and CPU are separate backend types. This establishes one enumerated and selected inference GPU, rather than converting physical computer count to GPU count. Runtime: Homebrew llama.cpp build 8680, commit 15f786e65, 12 CPU threads. Other local build changes and executable hash are unknown. Config: n_batch 2048, n_ubatch 512, K/V f16, Flash Attention on, mmap on, direct IO off, layer split, no_kv_offload false, no_host false, no tensor buffer overrides, no speculative drafter. The source script and raw parameters agree. Native workload: standalone synthetic PP512 and TG256, depth 0, single sequence. These are separate tests, not one 512-input/256-output chat and not a 256-token generation after a 512-token prompt. The logged build requests context as prompt + generation + depth separately (512 for PP, 256 for TG), but actual allocated/rounded context is not logged and the top-level context field stays unknown. Three measured repetitions per test, after warmup. KV/recurrent state is cleared before each repetition. PP times 512 synthetic input IDs including synchronization; TG times 256 serial single-token decode calls with synchronization. The numerator is native processed tokens, not display chunks or a chat tokenizer estimate. Load and initial warmup are outside the measured phase. Mean and sample SD are calculated from each trial's tokens divided by nanoseconds; both were independently checked. No TTFT, HTTP or user-facing first-token latency was measured. Headline native JSON results: PP 1297.664754 ± 4.914553 tokens/s; TG 86.354246 ± 0.010372 tokens/s. Raw trial nanoseconds are PP [392993291, 395967792, 394714958] and TG [2964199125, 2964491833, 2964907708]. Three separate invocations are available: this JSON run, a text-table run (PP512 1300.65 ± 5.63; TG256 86.11 ± 0.4), and an RSS run (PP512 1297.7 ± 5.3; TG256 86.45 ± 0.04). They are retained as companion observations of this configuration rather than inflated into new reports. The source Markdown uses the text run, whereas its structured JSON and README use the JSON run; differences are separate executions, not rounding errors. Do not pair timing from one with memory from another silently. Model identity: https://huggingface.co/tiiuae/Falcon-H1-3B-Instruct/tree/01087ec4c132d7f186908716b3530ea187f062a1 Verified historical upstream bf16 base only, not a downloadable copy of the measured GGUF. Reporter converted this base to F16 with Homebrew llama.cpp 8680 convert_hf_to_gguf.py, then llama-quantize Q4_K_M. Measured local GGUF hash and reproducibility of the exact resulting bytes are unknown. Raw llama_model_size tensor storage is 1885939156 bytes, and native tensor parameter count is 3149392384 (3.149392384 B). Those are engine metadata, not an independent nominal model-card parameter verification. The publisher's advertised size label is retained. The measured self-converted GGUF disk-object size is not published; tensor-storage bytes are not resident RAM. The separate RSS invocation reports maximum process RSS 2147205120 bytes, including mapped weights and runtime; it is not model-only memory and is not used as the selected JSON run's memory field. No allocation-based or measured power number is supplied. The synthetic benchmark performs no language-quality evaluation. Separate completion checks in the source described this llama.cpp arm as coherent, but that is a distinct prompt and call. Source MLX controls have some known degraded outputs and different quantization and prompt shapes; their rates are not merged into this arm. Sources: https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/raw-2026-09-05/falcon-h1-3b-q4km-selfconv.llama-bench.json https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/raw-2026-09-05/falcon-h1-3b-q4km-selfconv.llama-bench.stderr https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/hybrid-apple-silicon-bench-2.md https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/hybrid-apple-silicon-bench-2.json https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/raw-2026-09-05/bench_llama.sh https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/scripts/bench_common.py https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/raw/2026-08-15-muse-glimmer-30b-3way/ENV.md https://github.com/ggml-org/llama.cpp/blob/15f786e65/tools/llama-bench/llama-bench.cpp Structured rate fields are rounded to the form’s 0.1 token/s precision; exact source means, standard deviations and separate workloads are retained above.

generation:

147.5 tokens/s

prompt processing (prefill):

2535.2 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party historical John Rocky benchmark on Apple M4 Max, 128 GiB unified RAM, macOS 27.0 build 26A5416b. This report selects Falcon-H1 1.5B-Instruct, Q4_K_M, in the source's September 5 campaign. The actual JSON timestamps are 2026-09-04T21:06:13Z for PP and 2026-09-04T21:06:14Z for TG. Preserve both dates: the source local/campaign label is September 5, while the raw clock is September 4 UTC. No timing or date is inferred from webpage freshness. Hardware identity: the repository identifies the author's m4max host as Mac Studio Mac16,9, with chassis verification recorded in the August 15 ENV and the shared renderer's DEVICE_ALIASES. That earlier hardware inventory states a 40-core GPU and 128 GB RAM. This run directly records Apple M4 Max and the campaign declares 128 GiB; it does not independently log a chassis identifier or core count. The September 4 prose still says lid open, which conflicts with the later repository chassis correction. The headline uses M4 Max and 128 GiB; no physical core-count inference is made from 12 threads. Dedicated VRAM is inapplicable/unknown as a separate pool; shared RAM capacity is not model memory. One participating GPU is run-backed. Each native record lists gpu_info exactly Apple M4 Max, backends BLAS, MTL, devices auto, 99 GPU layers, main_gpu 0 and no CPU-MoE override. At the logged 15f786e65 revision, llama-bench get_gpu_info enumerates every GPU/IGPU description without deduplicating names. The accompanying stderr initializes only MTL0; BLAS and CPU are separate backend types. This establishes one enumerated and selected inference GPU, rather than converting physical computer count to GPU count. Runtime: Homebrew llama.cpp build 8680, commit 15f786e65, 12 CPU threads. Other local build changes and executable hash are unknown. Config: n_batch 2048, n_ubatch 512, K/V f16, Flash Attention on, mmap on, direct IO off, layer split, no_kv_offload false, no_host false, no tensor buffer overrides, no speculative drafter. The source script and raw parameters agree. Native workload: standalone synthetic PP512 and TG256, depth 0, single sequence. These are separate tests, not one 512-input/256-output chat and not a 256-token generation after a 512-token prompt. The logged build requests context as prompt + generation + depth separately (512 for PP, 256 for TG), but actual allocated/rounded context is not logged and the top-level context field stays unknown. Three measured repetitions per test, after warmup. KV/recurrent state is cleared before each repetition. PP times 512 synthetic input IDs including synchronization; TG times 256 serial single-token decode calls with synchronization. The numerator is native processed tokens, not display chunks or a chat tokenizer estimate. Load and initial warmup are outside the measured phase. Mean and sample SD are calculated from each trial's tokens divided by nanoseconds; both were independently checked. No TTFT, HTTP or user-facing first-token latency was measured. Headline native JSON results: PP 2535.20394 ± 4.882498 tokens/s; TG 147.525027 ± 0.873067 tokens/s. Raw trial nanoseconds are PP [202386125, 201627250, 201856542] and TG [1729113958, 1747234292, 1729670083]. Three separate invocations are available: this JSON run, a text-table run (PP512 2550.53 ± 4.11; TG256 148.58 ± 0.33), and an RSS run (PP512 2554.44 ± 3.18; TG256 147.97 ± 0.17). They are retained as companion observations of this configuration rather than inflated into new reports. The source Markdown uses the text run, whereas its structured JSON and README use the JSON run; differences are separate executions, not rounding errors. Do not pair timing from one with memory from another silently. Model identity: https://huggingface.co/tiiuae/Falcon-H1-1.5B-Instruct-GGUF/blob/0d3a6cfe25fb4eeab0153fb8623aac5b69d6bd0a/Falcon-H1-1.5B-Instruct-Q4_K_M.gguf Exact source-recorded HF snapshot and GGUF filename; the pinned artifact returned HTTP 200 to HEAD without downloading weights. No measured local-file hash is published. Raw llama_model_size tensor storage is 942121232 bytes, and native tensor parameter count is 1554863488 (1.554863488 B). Those are engine metadata, not an independent nominal model-card parameter verification. The publisher's advertised size label is retained. HTTP disk-object size is 944786656 bytes where available; neither quantity is resident RAM. The separate RSS invocation reports maximum process RSS 1130528768 bytes, including mapped weights and runtime; it is not model-only memory and is not used as the selected JSON run's memory field. No allocation-based or measured power number is supplied. The synthetic benchmark performs no language-quality evaluation. Separate completion checks in the source described this llama.cpp arm as coherent, but that is a distinct prompt and call. Source MLX controls have some known degraded outputs and different quantization and prompt shapes; their rates are not merged into this arm. Sources: https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/raw-2026-09-05/falcon-h1-1.5b-q4km.llama-bench.json https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/raw-2026-09-05/falcon-h1-1.5b-q4km.llama-bench.stderr https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/hybrid-apple-silicon-bench-2.md https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/hybrid-apple-silicon-bench-2.json https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/hybrid/raw-2026-09-05/bench_llama.sh https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/scripts/bench_common.py https://github.com/john-rocky/apple-silicon-llm-bench/blob/6e171c3a22fb7469c41e40d229df9e634a5e4be0/results/raw/2026-08-15-muse-glimmer-30b-3way/ENV.md https://github.com/ggml-org/llama.cpp/blob/15f786e65/tools/llama-bench/llama-bench.cpp Structured rate fields are rounded to the form’s 0.1 token/s precision; exact source means, standard deviations and separate workloads are retained above.