Llama 3.3

generation:

5.1 tokens/s

prompt processing (prefill):

78.0 tokens/s

quant:

Q4_K_XL

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

First-party benchmark by Valérian de Thézan de Gaussan, published November 25, 2025: https://valerian.dtdg.fr/blog/2025/amd-strix-halo-ai-395-llm-benchmark/ . The author identifies the measured machine as a Framework Desktop with AMD AI Max 395+ and 128 GB LPDDR5, and the benchmark section explicitly says llama.cpp Vulkan on the Radeon 8060S iGPU, with no dedicated GPU. One participating integrated GPU is attributed to that bounded local test setup. There is no numeric GPU inventory or startup-device log; GPU core/CU variant details, CPU core count, BIOS and OS/driver versions are not provided. The supplied CPU-thread count is 6, not the physical CPU core count. Source model Llama 3.3 70B; quantization Q4_K_XL. Exact tested conversion and any fine-tune are not specified. Tested filename, model publisher, revision, checksum and exact artifact are unknown. https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct is a verified base/model-family reference only and must not be treated as the measured quantized artifact. Source Size column is 39.73 GB; this is model storage size, not installed, loaded or peak RAM. Physical unified RAM is independently stated as 128 GB; VRAM carve-out and measured memory use are unknown. Native llama-bench protocol: llama-bench -p 512 -n 128 -t 6 -ngl 999. Source explicitly defines pp as prompt/context ingestion and tg as decode, and reports PP 78 tokens/s and TG 5.05 tokens/s for this row. PP512 and TG128 are separate synthetic phase tests; do not combine them into an end-to-end rate. The source command requests maximum GPU offload; actual layer placement is not logged. Vulkan KHR cooperative matrices are stated. Runtime build/commit, executable checksum and patches are unknown. No defaults from another llama.cpp revision are assumed. The publication provides rounded rates without raw phase durations, per-repeat samples, statistical uncertainty or repetition count. The 512-token prompt and 128-token generation lengths are the declared benchmark protocol, not raw token-counter observations. Timer implementation and exact warmup/cache handling cannot be independently pinned to an unknown build. Context depth/capacity, batch/ubatch, flash-attention mode, KV precision, power limits, hardware clocks, thermals, background loads and swap are unreported. Measurement date is unreported; publication date is not silently treated as run date. No measured TTFT or first-token clock is supplied. No quality/correctness test is reported. These are source-reported native throughput measurements, not fresh measurements or estimates. All eleven source model rows share this setup; each model/quant pair is kept separate. Structured PP/TG are rounded to the form’s 0.1 tokens/s step; exact source rates remain above.

generation:

6.8 tokens/s

quant:

Q8 (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

NetworkChuck source-reported historical M3-only benchmark, 2026-09-19T17:03:53.712Z–17:51:07.654Z (raw timestamps), recovered2026-10-13, not locally rerun. Credit NetworkChuck; measurements/docs CC BY4.0. Primary summary: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/dropped.json . Original requests: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/data/language-models/baseline-m3-ultra/runs.jsonl . Method: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/docs/method.md . Source dropped Llama from the comparison on Sept20 as an older model, not due to failed measurements. M5 never ran it. CLIENT-ONLY headline: write=6.763430949 tokens/s median of five measured requests at nominal32K; actual prompt median32396 (range32396–32398), output128 each. promptTps deliberately blank: no native PP phase was measured for this model. The input/TTFT reading rate127.05078209328913 is retained only below as client-derived throughput. Actual median prompt count is32396, distinct from nominal32768. Configured context/KV capacity is unreported; capacity field is blank. Decode is measured from first-to-last streamed text, not native generation_tps, not total-request throughput and not cross-context maximum. Hardware: one Mac Studio M3 Ultra, 32-core CPU(24P+8E), 80-core GPU, 512GB unified memory, macOS26.5.1, applegpu_g15d, 8TB storage, Chuck's own unit. Unified RAM is not dedicated VRAM. A stuck indexer occupied one CPU core; no impact-adjustment rerun exists. Allocator peak, RSS, model-load time, power, thermals, clocks, swap and GPU-offload count are unknown for these Llama rows. Artifact: mlx-community/Llama-3.3-70B-Instruct-8bit, MLX Safetensors8-bit. Exact HF identity/precision verified2026-10-13 at https://huggingface.co/mlx-community/Llama-3.3-70B-Instruct-8bit . The card identifies a conversion from meta-llama/Llama-3.3-70B-Instruct using mlx-lm0.20.1; conversion runtime is not measurement runtime. Historical tested HF revision/weight hashes and exact group/mixed-precision details unrecorded; author did not hash-verify weights. Source nominal70B dense label is not an asserted exact architecture parameter count. Measurement runtime is mlx-lm0.31.3 with MLX0.31.2, NOT main-comparison MLX0.32.2. Each raw request fingerprint: 0.31.3-0.31.2-macOS-26.5.1-arm64-arm-64bit-Mach-O-applegpu_g15d. Earlier baseline manifest is dated Sept20 and lists the later lineup, so it must not be used to claim Llama ran then. dropped.json and raw Llama timestamps are the appropriate campaign evidence. Workload: serial streaming HTTP chat requests to mlx_lm.server over LAN from a separate controller; prompts/sample.txt repeated to nominal length after unique run-ID prefix and summary instruction. Source settings: temperature0, thinking disabled, output cap128; one discarded warm-up then five measured requests for each of512/4K/16K/32K. Raw check finds24 requests,4 warmups,20 measured, all cachedTokens0/outputTokens128/firstWordField=content, no errors or invalid flags. This confirms uncached input and warm-up protocol, not cold model-load timings. Exact historical server launch flags and intercell rest are not independently logged for this older Llama session, so do not silently copy the later manifest's30-second rest into it. Runtime build/source revision, explicit flash-attention, KV quantization and speculative settings are not established. Client clock is monotonic in controller, starts immediately before writing request. First text is first nonempty text delta, excluding empty opening frames. read=usage.prompt_tokens/firstWordSec; write=(usage.completion_tokens−1)/(lastWordSec−firstWordSec); total through stream completion. Network/HTTP overhead remains. These are not native PP/TG. Published timing implementation: https://github.com/theNetworkChuck/mac-studio-m5-ultra-local-ai/blob/main/harness/src/lane.ts . Summary medians/ranges were independently recomputed from raw rows and match dropped.json; this model had no cells.json entry and no native engine result. All rows: nominal; actual in/out; client read/write tokens/s; first/total seconds; values median[min,max], n measured. Rates/times shown to10 significant digits; full precision in raw/source packet: 512: in=510[509,510]; out=128[128,128]; read=154.0893621[153.4980397,154.2952624]; write=8.724754202[8.722782087,8.737563764]; first=3.305909[3.303278,3.322518]; total=17.863348[17.83836,17.87311]; n=5 4K: in=4095[4093,4095]; out=128[128,128]; read=162.6170391[162.49048,162.6614415]; write=8.337744569[8.329403101,8.339250152]; first=25.181125[25.174989,25.189168]; total=40.418484[40.40705,40.423623]; n=5 16K: in=16223[16222,16224]; out=128[128,128]; read=146.2999658[144.8342954,146.3908233]; write=7.331800698[7.329965859,7.332916608]; first=110.885968[110.826619,112.003859]; total=128.207912[128.153024,129.323906]; n=5 32K: in=32396[32396,32398]; out=128[128,128]; read=127.0507821[127.0372148,127.0934549]; write=6.763430949[6.757327657,6.768535795]; first=255.000398[254.906911,255.011888]; total=273.764986[273.678006,273.812481]; n=5 The 32K first-text median is255.000398s and full-request median273.764986s. Neither includes a separately measured model-load phase. No independent quality claim follows from these throughput results.