Local model performance on your hardware

generation:

23.7 tokens/s

prompt processing (prefill):

192.8 tokens/s

quant:

UD-Q8_K_XL (GGUF)

kv:

Q8

flash attention:

on

mtp (multi-token prediction):

off

User reports GLM-4.7 at 23.7 t/s generation and 192.8 t/s prompt processing on a single Radeon 8060S gfx1151 device. Setup is llama.cpp build b8118 with UD-Q8_K_XL, q8_0 KV cache, -ngl 999, -fa 1, -ub 2048 -b 2048, and --mmap 0 -dio 1 on the ROCm/HIP arm. The headline pair comes from the same ROCm/HIP run, with exact rates 23.66 ± 1.80 t/s TG128 and 192.79 ± 3.02 PP512 at depth 8000; the Vulkan arm measured 21.14 ± 1.39 and 163.58 ± 1.83.

generation:

26.5 tokens/s

prompt processing (prefill):

234.0 tokens/s

quant:

UD-Q6_K_XL (GGUF)

kv:

Q8

flash attention:

on

mtp (multi-token prediction):

off

User reports GLM-4.7 at 26.5 t/s generation and 234 t/s prompt processing on a single Radeon 8060S gfx1151 device. Setup is llama.cpp b8118 with UD-Q6_K_XL, q8_0 KV cache, -ngl 999, -fa 1, -ub 2048 -b 2048, and ROCm/HIP with BLAS. The headline pair comes from the same ROCm arm, measured as PP512 233.98 ± 2.41 and TG128 26.49 ± 2.30 tok/s at depth 8000, with no cross-pairing.

generation:

36.0 tokens/s

prompt processing (prefill):

267.3 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

Q8

flash attention:

on

mtp (multi-token prediction):

off

User reports GLM-4.7 at 36 t/s generation and 267.3 t/s prompt processing on a single Radeon 8060S gfx1151 device. Setup is llama.cpp b8118 with UD-Q4_K_XL and q8_0 KV cache, 16 CPU threads, -ngl 999, -fa 1, -ub 2048 -b 2048, no speculative draft. The headline figures are the ROCm/HIP arm, measured as PP512 267.25 ± 1.56 and TG128 35.95 ± 3.53 tok/s at depth 8000; the 8000-token prefix is excluded from timing.

generation:

52.6 tokens/s

prompt processing (prefill):

413.7 tokens/s

quant:

4-bit (source config) (MLX)

User reports GLM-4.7 at 52.6 t/s generation and 413.7 t/s prompt processing on a single Apple M2 Max with 64 GB unified memory. Setup is MLX with a 4-bit artifact, batch size 1, requested 512-token prompt and 128-token generation; KV cache precision and prefill chunk size are unreported. The figures are three-trial means after discarding Trial 1, with sample SD 0.04 for TG and 0.38 for PP; peak memory was 17.497 in engine output units.

generation:

43.2 tokens/s

User reports glm-4.7-flash at 43.16 tok/s on a Framework host with Core Ultra CPU and RTX 3060 12GB. Setup is Ollama with a 192-token generation cap; the model scored 5/5 on the repo's compact coding, tool-use and agent-orchestration quality suite. It is the fastest perfect-quality model in the capture, ahead of qwen3.5:35b-a3b at 39.90 tok/s and qwen3-coder-next at 32.55 tok/s. A separate llama-server comparison on a T5500 host lists omnicoder:9b-q4_k_m at 1.48 tok/s, qwen3.5:35b-a3b-q2_k_l at 3.28 tok/s, lfm2:24b at 4.36 tok/s and granite4:7b-a1b-h at 6.48 tok/s.

generation:

168.0 tokens/s

User reports GLM-4.7-Flash at 168 tok/s on a single NVIDIA H100 80GB, up from a 120 tok/s baseline with EAGLE3 speculative decoding. Setup is SGLang v0.5.6 with the FlashInfer backend, a 277 MB EAGLE3 draft head, 6 draft tokens proposed per step, and a 40% acceptance rate averaging 2.4 accepted tokens per verification step. At batch size 32 the same setup reaches 440 tok/s versus 259 tok/s baseline (1.70x), with per-request latency improving 1.30x. The user also reproduced the EAGLE3 paper's Llama-3.1-8B result at 1.25x versus the published 1.32x, and notes a SGLang token-counting bug that deflated measured throughput by about 35%.

generation:

13.1 tokens/s

prompt processing (prefill):

160.3 tokens/s

quant:

Q4_K_XL (GGUF)

User benchmarks GLM-4.7 Flash (30B-A3B MoE) on an AMD Radeon 680M iGPU with 64GB DDR5 RAM, comparing MXFP4, Q4_K_M, and Q4_K_XL quants. The Q4_K_XL quant achieves the fastest generation at 13.13 t/s, while MXFP4 leads prompt processing at 258.36 t/s. Setup uses llama.cpp with Vulkan backend and Flash Attention enabled; models are offloaded to system RAM due to iGPU memory limits.

generation:

11.7 tokens/s

prompt processing (prefill):

258.4 tokens/s

quant:

MXFP4 (GGUF)

User benchmarks GLM-4.7 Flash quants on a Radeon 680M iGPU. MXFP4 gives the fastest prompt processing at 258.36 t/s and Q4_K_XL the fastest generation at 13.13 t/s.

generation:

40.0 tokens/s

User reports running 3 local sessions of GLM 4.7 at about 40 tok/s each on an AMD Strix Halo mini PC with 128 GB unified RAM. The model family is inferred as GLM-4.7 from the title and text. The user is positive about the setup for budget local AI.