Local model performance on your hardware

generation:

80-90 tokens/s

prompt processing (prefill):

1600.0 tokens/s

quant:

oQ4e

User reports GLM-5.3 Flash at 80-90 t/s generation and 1.6k t/s prefill on a Mac Studio M5 Ultra 256GB (64-core), short of the 140-150 t/s and ~2k pp seen in oMLX benchmark recipes. Setup is oMLX 0.7.3rc1 (build 3517) with oQ4e quants, both Jundot's GLM-5.3-Flash-oQ4e-mtp and a self-quantized oQ4e from FP8. MTP was used, and DFlash2, patch-size changes and other tweaks were tried without reaching the target speeds.

generation:

35.0 tokens/s

User reports running GLM-5.3 Flash on an RTX PRO 6000 Max-Q with DDR4 system memory at roughly 35 tok/s with 512K context. No engine, quant or KV cache format is stated; the DDR4 mention implies weights or KV cache are partly served from system RAM. The user asks whether upgrading to DDR5, at a cost of 6k EUR, would be worth it if it reaches about 50 tok/s, and asks for others' experience.

generation:

7.0 tokens/s

quant:

Q4 (GGUF)

User reports GLM-5.3-Flash 320B at about 7 tok/s on an RTX 2080 Ti 11GB. Setup is llama.cpp with Q4 weights (200GB) and MoE experts kept in system RAM via -ncmoe, on a box with 256GB of DDR4-3600 and an i9-10940X. User calls it slow for hard questions but workable, and plans to benchmark more models on real reverse-engineering tasks.

generation:

18.9 tokens/s

prompt processing (prefill):

740.0 tokens/s

User reports GLM-5.3 Flash at 18.9 tok/s plain decode on a Strix Halo 128GB box (Ryzen AI Max+ 395, 8060S gfx1151). Setup is a fork of gufo that splits prefill between the GPU and the XDNA2 NPU and stacks speculative decoding on top. With DFlash2 speculation on code the model reaches 29.7 tok/s. Prefill peaks at 740 tok/s at 8k, 17.5% over GPU only. The same fork also runs Flint, Qwen3.6-35B-A3B, Qwen3.8 Flash-Next and Qwen3.8-27B, with best-case decode of 73, 159, 118 and 62 tok/s respectively.

generation:

61.7 tokens/s

prompt processing (prefill):

1587.5 tokens/s

quant:

mixed-4_8bit (MLX)

User benchmarks GLM-5.3-Flash-MLX-mixed-4_8bit on an M5 Ultra 256GB with oMLX, reporting 61.7 t/s generation and 1587.5 t/s prompt processing at 1024-token prompt. Setup uses MLX mixed 4/8-bit weights with peak MLX memory of 174.17 GB; a 131k context prefill would need ~208.28 GB, exceeding the 205.80 GB prefill limit. At longer prompts generation stays 58.1-59.6 t/s and prefill 2323.9-2373.0 t/s; continuous batching reaches 67.3 t/s at 2x and 99.3 t/s at 4x.

generation:

80-100 tokens/s

quant:

oQ4e

User reports GLM 5.3 Flash on an M5 Ultra 256GB with oMLX 0.7.1, gaining about 20-30 tok/s on long running tasks at single concurrency versus 0.7.0, which was around 60-80 tok/s. Setup is oMLX 0.7.1 with an oQ4e quant on a single machine. With Lightning MTP enabled the user gets anywhere from 80-100 tok/s.

generation:

64.8 tokens/s

prompt processing (prefill):

1973.0 tokens/s

quant:

2.05bpw EXL3 (EXL3)

kv:

8-bit

User reports 64.8 t/s generation and 1973 t/s prompt processing. Source evidence (verbatim): Source: https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090 # GLM-5.3-Flash (321B MoE, 2.05bpw EXL3) on 2x RTX 5090 + 64 GB DDR5, ExLlamaV3 + TabbyAPI Hardware: two NVIDIA RTX 5090 (32 GB each, PCIe 5.0 x8), AMD Ryzen 7 9800X3D, 60 GB usable of 2x32 GB DDR5-6000. Engine: ExLlamaV3 dev 5783a93 served through TabbyAPI. Model: GLM-5.3-Flash by zai-org, 321B parameters with about 18B active, turboderp 2.05 bpw EXL3 quant (76 GB), with experts offloaded to system RAM (96 and 102 CPU-resident experts per MoE layer out of 288) and 8-bit KV cache at 262144 tokens. Single-stream decode with MTP draft enabled (DRAFT=1, DRAFT_N=1): 64.8 tokens/s median of ten requests, 1,024 forced output tokens at temperature 0, median of three boots. Cold-prompt prefill measured at 1,973 tokens/s at 32,738 prompt tokens. Without the draft, decode with 2, 3 and 4 concurrent streams was 74.4, 92.0 and 98.0 tokens/s in total.

generation:

26.3 tokens/s

prompt processing (prefill):

2165.0 tokens/s

quant:

EXL3 3.05 bpw (EXL3)

User reports 26.3 t/s generation and 2165 t/s prompt processing. Source evidence (verbatim): Source: https://dev.to/seppegadeyne/run-glm-53-flash-321b-locally-with-project-maya-and-exl3-on-one-rtx-5090-24pj # Run GLM-5.3-Flash (321B) locally with Project Maya and EXL3 on one RTX 5090 Hardware: one RTX 5090 32 GB (also driving the desktop), Ryzen 9 9950X3D, 128 GB DDR5 at 5,600 MT/s, Arch Linux, NVIDIA driver 615.78, CUDA 13.4. Engine: Project Maya 1.1.11 (a Strata fork) compiling llama.cpp at a pinned commit. Model: GLM-5.3-Flash 321B MoE with about 18B active, turboderp EXL3 3.05 bpw (125.3 GB), with the expert cache and KV streaming offloading to system RAM. Benchmark: streaming client, 8 runs, greedy decoding, 512 output tokens, median reported. At 128K context with the default reserve: decode 26.3 tokens/s (30.4 on Dutch prose, 21.9 on code) and prefill 2,165 tokens/s at 126K. At 262K context: decode 24.1 tokens/s and prefill 2,039 tokens/s at 254K.

generation:

15.0 tokens/s

quant:

Q2

User reports about 15 t/s running GLM-5.3 Flash at Q2 on an RTX 3090 with DDR4 RAM and SSD offload, under an engine called Project Maya. The setup streams part of the model from system memory and disk, since a Q2 GLM-5.3 Flash does not fit in the card's 24GB alone. User's main complaint is behavioral rather than speed: directed to start a new project, the model continued on an old project with a similar name and made little to no improvement, a mistake no other model or engine made in the same environment.

generation:

120.0 tokens/s

quant:

NVFP4 (NVFP4)

User asks whether upgrading from GLM-5.3 to Muse Spark 816B-A42B-NVFP is worth it, and reports getting around 120 tok/s with GLM-5.3 in NVFP4 on Blackwell. No engine, context length or KV cache quant is stated. The user says the only real gains they can see are Muse's vision encoder and better reasoning and long-horizon performance, and asks how Muse's speed compares. They add they are getting tired of chasing every new model release.

generation:

76.0 tokens/s

User reports GLM 5.3 Flash at 76 tok/s on a 256GB M5 Ultra Mac Studio, run locally with oMLX. Setup is oMLX on the Mac Studio; the user also runs Qwen 3.8 27B locally for other tasks. The user says the biggest weakness of local LLMs is being able to run only one instance of GLM, and that coding is a no-go locally, though images and video are fine. They also compare against Ollama Cloud's $20 plan and use Kimi K3 and DeepSeek 4.1 alongside GLM.

generation:

30.0 tokens/s

prompt processing (prefill):

1200.0 tokens/s

User reports GLM 5.3 Flash at 30 t/s decode and 1200 t/s prefill, compared with Qwen Flash Next at 125 t/s decode and 3000 t/s prefill. No hardware or engine details are given. User says GLM 5.3 Flash is slower but better at thinking, and that it produced the same front-end issues and bugs as Qwen Flash Next and a 27B model on a coding task.

generation:

16.0 tokens/s

quant:

IQ2_XXS (GGUF)

User pruned GLM-5.3-Flash (321B MoE) from 288 to 144 routed experts per layer, selecting experts on agentic coding sessions, then quantized with an imatrix built on the same sessions. The IQ2_XXS build (45.3 GB) runs on an RX 7900 XTX (24 GB) plus 32 GB DDR4-2133 with an i7-7700K over PCIe 3.0, at roughly 16 tok/s generation and roughly 950 tok/s prompt processing. Setup is llama.cpp with IQ2_XXS GGUF weights and 384K context. The 2-bit version is experimental: it solved 39/48 attempts versus 48/48 for Q3_K and above, and when it fails it loops or keeps exploring until the turn limit. An anti-loop guard did not help (27/37 vs 30/37). Evaluation used 26 Python SWE tasks from ansible, openlibrary and qutebrowser, 2 attempts each, 200 turns, in a sealed clean room with no network or git history. The code-scored expert mask needed x1.21 turns versus the full model, against x2.21 for a frequency mask and x2.29 for the public multi-domain REAP 50% mask. IQ2_S wants 24 + 48 GB, Q3_K 24 + 64 GB, Q4_K_M 24 + 96 GB or 128 GB unified memory.

generation:

36.8 tokens/s

quant:

4-bit (MLX)

User reports GLM-5.3-Flash 4-bit with MTP at 36.8 tok/s decode on a disaggregated setup pairing an M5 Ultra 256GB (prefill) with an M3 Ultra 512GB (decode). Setup is MLX with the JACCL RDMA backend over two Thunderbolt 5 cables, mlx-vlm 0.7.6, a 17.5k-token prompt and 256 output tokens. Prefill runs on the M5 at 988 tok/s (17.8 s), the 358 MB cache is handed off over TB5 in 38 ms, and first token arrives on the M3 about 18.5 s after the request instead of about 40 s. Single-node figures: M3 Ultra 512GB prefill 438 tok/s at 2k chunks and 454 tok/s at 8k chunks, decode 32.9 tok/s; M5 Ultra 256GB prefill 1,046 tok/s at 2k chunks and 1,188 tok/s at 8k chunks, decode 47.2 tok/s. The user notes the M5 alone decodes faster for a single request, and that the prefill node needs the full model so this only works for models that fit in 256 GB.

generation:

9.8 tokens/s

prompt processing (prefill):

97.9 tokens/s

quant:

Q4_K (GGUF)

mtp (multi-token prediction):

off

User reports GLM-5.3 at 9.8 t/s generation and 97.9 t/s prompt processing on two AMD Strix Halo nodes with 128GB each. Setup is the staged GLM path over OdinLink USB4/TB5 RDMA with Q4_K, TP=2 across two Radeon 8060S iGPUs, fixed 4,096 prompt tokens and 300 generated tokens at prefill batch 256. The source gives exact rates of 97.89 t/s prefill and 9.77 t/s decode from a single diagnostic run, with no TTFT reported and no single-node scaling claimed.

generation:

10.1 tokens/s

prompt processing (prefill):

103.4 tokens/s

quant:

Q4_K (GGUF)

mtp (multi-token prediction):

off

User reports GLM-5.3 Flash Q4_K at 10.1 t/s generation and 103.4 t/s prompt processing on two AMD Strix Halo nodes, each with 128 GB installed memory. Setup is a custom DS4 fork's staged GLM TP=2 ROCm path with a 4,608-token context allocation, 4,096-token prompt and 300-token decode at prefill batch 256, no speculative draft or MTP. The 4K point uses a cross-disciplinary prompt and is one observation; the source waived repeated timing, so no spread or confidence interval is established.

generation:

30.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

flash attention:

on

mtp (multi-token prediction):

on

User reports GLM-5.3 at 30 t/s on a two-node cluster of two Strix Halo APUs and two Radeon AI Pro R9700 cards, one 32 GB card per node. Setup is the sixvolts/llama-halo-hybrid HIP/ROCm llama.cpp fork with RPC, UD-Q4_K_XL GGUF at 131072 context, batch 32768, ubatch 1024, -fa on, and draft-MTP with up to two drafts on the head-node R9700. The 30 t/s is a reported decode rate at a prompt labeled 25.8K, alongside 92 ms/step and MTP acceptance 0.90; the timing primitive and aggregation formula are unknown.

generation:

43.8 tokens/s

quant:

UD-Q2_K_XL (GGUF)

flash attention:

on

mtp (multi-token prediction):

on

User reports GLM-5.3-Flash at 43.8 t/s on one Strix Halo APU/iGPU plus one Radeon AI Pro R9700 with 32 GB VRAM. Setup is the llama-halo-hybrid HIP/ROCm llama.cpp fork with UD-Q2_K_XL, 131072-token context, one slot, -fa on, -ngl 999 and draft-mtp with at most two draft tokens. The 43.8 t/s is the single-stream median of 40 real category prompts, not a token-weighted aggregate; prose-category speed is 32.8 t/s and draft acceptance 0.73.

generation:

39.1 tokens/s

quant:

UD-Q2_K_XL (GGUF)

flash attention:

on

mtp (multi-token prediction):

on

User reports GLM-5.3-Flash at 39.1 t/s on one Strix Halo APU/iGPU plus one Radeon Pro V620 with 32 GB VRAM. Setup is the llama-halo-hybrid HIP/ROCm fork with UD-Q2_K_XL, 131072 tokens of context, one slot, -fa on, -ngl 999, and draft-mtp with at most two draft tokens. The 39.1 t/s is the single-stream median of 40 real category prompts with prefix caching disabled, not a token-weighted aggregate; prose-category speed is 30.9 t/s and draft acceptance is 0.74.

generation:

38.6 tokens/s

quant:

UD-Q2_K_XL (GGUF)

flash attention:

on

mtp (multi-token prediction):

on

User reports GLM-5.3-Flash at 38.6 t/s on one Strix Halo APU/iGPU plus one Radeon RX 7800 XT with 16 GB VRAM. Setup is the llama-halo-hybrid HIP/ROCm llama.cpp fork with UD-Q2_K_XL, 131072 tokens of context, one slot, -fa on, -ngl 999, and draft-mtp with at most two draft tokens. The 38.6 t/s is the single-stream median of 40 real category prompts, not a token-weighted aggregate; prose-category speed is 30.5 t/s and draft acceptance is 0.74.