Local model performance on your hardware

generation:

52.6 tokens/s

prompt processing (prefill):

413.7 tokens/s

quant:

4-bit (source config) (MLX)

User reports GLM-4.7 at 52.6 t/s generation and 413.7 t/s prompt processing on a single Apple M2 Max with 64 GB unified memory. Setup is MLX with a 4-bit artifact, batch size 1, requested 512-token prompt and 128-token generation; KV cache precision and prefill chunk size are unreported. The figures are three-trial means after discarding Trial 1, with sample SD 0.04 for TG and 0.38 for PP; peak memory was 17.497 in engine output units.

generation:

43.2 tokens/s

User reports glm-4.7-flash at 43.16 tok/s on a Framework host with Core Ultra CPU and RTX 3060 12GB. Setup is Ollama with a 192-token generation cap; the model scored 5/5 on the repo's compact coding, tool-use and agent-orchestration quality suite. It is the fastest perfect-quality model in the capture, ahead of qwen3.5:35b-a3b at 39.90 tok/s and qwen3-coder-next at 32.55 tok/s. A separate llama-server comparison on a T5500 host lists omnicoder:9b-q4_k_m at 1.48 tok/s, qwen3.5:35b-a3b-q2_k_l at 3.28 tok/s, lfm2:24b at 4.36 tok/s and granite4:7b-a1b-h at 6.48 tok/s.

generation:

168.0 tokens/s

User reports GLM-4.7-Flash at 168 tok/s on a single NVIDIA H100 80GB, up from a 120 tok/s baseline with EAGLE3 speculative decoding. Setup is SGLang v0.5.6 with the FlashInfer backend, a 277 MB EAGLE3 draft head, 6 draft tokens proposed per step, and a 40% acceptance rate averaging 2.4 accepted tokens per verification step. At batch size 32 the same setup reaches 440 tok/s versus 259 tok/s baseline (1.70x), with per-request latency improving 1.30x. The user also reproduced the EAGLE3 paper's Llama-3.1-8B result at 1.25x versus the published 1.32x, and notes a SGLang token-counting bug that deflated measured throughput by about 35%.

generation:

40.0 tokens/s

User reports running 3 local sessions of GLM 4.7 at about 40 tok/s each on an AMD Strix Halo mini PC with 128 GB unified RAM. The model family is inferred as GLM-4.7 from the title and text. The user is positive about the setup for budget local AI.