Local model performance on your hardware

generation:

80-90 tokens/s

prompt processing (prefill):

1600.0 tokens/s

quant:

oQ4e

User reports GLM-5.3 Flash at 80-90 t/s generation and 1.6k t/s prefill on a Mac Studio M5 Ultra 256GB (64-core), short of the 140-150 t/s and ~2k pp seen in oMLX benchmark recipes. Setup is oMLX 0.7.3rc1 (build 3517) with oQ4e quants, both Jundot's GLM-5.3-Flash-oQ4e-mtp and a self-quantized oQ4e from FP8. MTP was used, and DFlash2, patch-size changes and other tweaks were tried without reaching the target speeds.

generation:

35.0 tokens/s

User reports running GLM-5.3 Flash on an RTX PRO 6000 Max-Q with DDR4 system memory at roughly 35 tok/s with 512K context. No engine, quant or KV cache format is stated; the DDR4 mention implies weights or KV cache are partly served from system RAM. The user asks whether upgrading to DDR5, at a cost of 6k EUR, would be worth it if it reaches about 50 tok/s, and asks for others' experience.

generation:

7.0 tokens/s

quant:

Q4 (GGUF)

User reports GLM-5.3-Flash 320B at about 7 tok/s on an RTX 2080 Ti 11GB. Setup is llama.cpp with Q4 weights (200GB) and MoE experts kept in system RAM via -ncmoe, on a box with 256GB of DDR4-3600 and an i9-10940X. User calls it slow for hard questions but workable, and plans to benchmark more models on real reverse-engineering tasks.

generation:

18.9 tokens/s

prompt processing (prefill):

740.0 tokens/s

User reports GLM-5.3 Flash at 18.9 tok/s plain decode on a Strix Halo 128GB box (Ryzen AI Max+ 395, 8060S gfx1151). Setup is a fork of gufo that splits prefill between the GPU and the XDNA2 NPU and stacks speculative decoding on top. With DFlash2 speculation on code the model reaches 29.7 tok/s. Prefill peaks at 740 tok/s at 8k, 17.5% over GPU only. The same fork also runs Flint, Qwen3.6-35B-A3B, Qwen3.8 Flash-Next and Qwen3.8-27B, with best-case decode of 73, 159, 118 and 62 tok/s respectively.

generation:

61.7 tokens/s

prompt processing (prefill):

1587.5 tokens/s

quant:

mixed-4_8bit (MLX)

User benchmarks GLM-5.3-Flash-MLX-mixed-4_8bit on an M5 Ultra 256GB with oMLX, reporting 61.7 t/s generation and 1587.5 t/s prompt processing at 1024-token prompt. Setup uses MLX mixed 4/8-bit weights with peak MLX memory of 174.17 GB; a 131k context prefill would need ~208.28 GB, exceeding the 205.80 GB prefill limit. At longer prompts generation stays 58.1-59.6 t/s and prefill 2323.9-2373.0 t/s; continuous batching reaches 67.3 t/s at 2x and 99.3 t/s at 4x.

generation:

64.8 tokens/s

prompt processing (prefill):

1973.0 tokens/s

quant:

2.05bpw EXL3 (EXL3)

kv:

8-bit

User reports 64.8 t/s generation and 1973 t/s prompt processing. Source evidence (verbatim): Source: https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090 # GLM-5.3-Flash (321B MoE, 2.05bpw EXL3) on 2x RTX 5090 + 64 GB DDR5, ExLlamaV3 + TabbyAPI Hardware: two NVIDIA RTX 5090 (32 GB each, PCIe 5.0 x8), AMD Ryzen 7 9800X3D, 60 GB usable of 2x32 GB DDR5-6000. Engine: ExLlamaV3 dev 5783a93 served through TabbyAPI. Model: GLM-5.3-Flash by zai-org, 321B parameters with about 18B active, turboderp 2.05 bpw EXL3 quant (76 GB), with experts offloaded to system RAM (96 and 102 CPU-resident experts per MoE layer out of 288) and 8-bit KV cache at 262144 tokens. Single-stream decode with MTP draft enabled (DRAFT=1, DRAFT_N=1): 64.8 tokens/s median of ten requests, 1,024 forced output tokens at temperature 0, median of three boots. Cold-prompt prefill measured at 1,973 tokens/s at 32,738 prompt tokens. Without the draft, decode with 2, 3 and 4 concurrent streams was 74.4, 92.0 and 98.0 tokens/s in total.

generation:

26.3 tokens/s

prompt processing (prefill):

2165.0 tokens/s

quant:

EXL3 3.05 bpw (EXL3)

User reports 26.3 t/s generation and 2165 t/s prompt processing. Source evidence (verbatim): Source: https://dev.to/seppegadeyne/run-glm-53-flash-321b-locally-with-project-maya-and-exl3-on-one-rtx-5090-24pj # Run GLM-5.3-Flash (321B) locally with Project Maya and EXL3 on one RTX 5090 Hardware: one RTX 5090 32 GB (also driving the desktop), Ryzen 9 9950X3D, 128 GB DDR5 at 5,600 MT/s, Arch Linux, NVIDIA driver 615.78, CUDA 13.4. Engine: Project Maya 1.1.11 (a Strata fork) compiling llama.cpp at a pinned commit. Model: GLM-5.3-Flash 321B MoE with about 18B active, turboderp EXL3 3.05 bpw (125.3 GB), with the expert cache and KV streaming offloading to system RAM. Benchmark: streaming client, 8 runs, greedy decoding, 512 output tokens, median reported. At 128K context with the default reserve: decode 26.3 tokens/s (30.4 on Dutch prose, 21.9 on code) and prefill 2,165 tokens/s at 126K. At 262K context: decode 24.1 tokens/s and prefill 2,039 tokens/s at 254K.

generation:

15.0 tokens/s

quant:

Q2

User reports about 15 t/s running GLM-5.3 Flash at Q2 on an RTX 3090 with DDR4 RAM and SSD offload, under an engine called Project Maya. The setup streams part of the model from system memory and disk, since a Q2 GLM-5.3 Flash does not fit in the card's 24GB alone. User's main complaint is behavioral rather than speed: directed to start a new project, the model continued on an old project with a similar name and made little to no improvement, a mistake no other model or engine made in the same environment.

generation:

76.0 tokens/s

User reports GLM 5.3 Flash at 76 tok/s on a 256GB M5 Ultra Mac Studio, run locally with oMLX. Setup is oMLX on the Mac Studio; the user also runs Qwen 3.8 27B locally for other tasks. The user says the biggest weakness of local LLMs is being able to run only one instance of GLM, and that coding is a no-go locally, though images and video are fine. They also compare against Ollama Cloud's $20 plan and use Kimi K3 and DeepSeek 4.1 alongside GLM.

generation:

16.0 tokens/s

quant:

IQ2_XXS (GGUF)

User pruned GLM-5.3-Flash (321B MoE) from 288 to 144 routed experts per layer, selecting experts on agentic coding sessions, then quantized with an imatrix built on the same sessions. The IQ2_XXS build (45.3 GB) runs on an RX 7900 XTX (24 GB) plus 32 GB DDR4-2133 with an i7-7700K over PCIe 3.0, at roughly 16 tok/s generation and roughly 950 tok/s prompt processing. Setup is llama.cpp with IQ2_XXS GGUF weights and 384K context. The 2-bit version is experimental: it solved 39/48 attempts versus 48/48 for Q3_K and above, and when it fails it loops or keeps exploring until the turn limit. An anti-loop guard did not help (27/37 vs 30/37). Evaluation used 26 Python SWE tasks from ansible, openlibrary and qutebrowser, 2 attempts each, 200 turns, in a sealed clean room with no network or git history. The code-scored expert mask needed x1.21 turns versus the full model, against x2.21 for a frequency mask and x2.29 for the public multi-domain REAP 50% mask. IQ2_S wants 24 + 48 GB, Q3_K 24 + 64 GB, Q4_K_M 24 + 96 GB or 128 GB unified memory.

generation:

36.8 tokens/s

quant:

4-bit (MLX)

User reports GLM-5.3-Flash 4-bit with MTP at 36.8 tok/s decode on a disaggregated setup pairing an M5 Ultra 256GB (prefill) with an M3 Ultra 512GB (decode). Setup is MLX with the JACCL RDMA backend over two Thunderbolt 5 cables, mlx-vlm 0.7.6, a 17.5k-token prompt and 256 output tokens. Prefill runs on the M5 at 988 tok/s (17.8 s), the 358 MB cache is handed off over TB5 in 38 ms, and first token arrives on the M3 about 18.5 s after the request instead of about 40 s. Single-node figures: M3 Ultra 512GB prefill 438 tok/s at 2k chunks and 454 tok/s at 8k chunks, decode 32.9 tok/s; M5 Ultra 256GB prefill 1,046 tok/s at 2k chunks and 1,188 tok/s at 8k chunks, decode 47.2 tok/s. The user notes the M5 alone decodes faster for a single request, and that the prefill node needs the full model so this only works for models that fit in 256 GB.

generation:

9.8 tokens/s

prompt processing (prefill):

97.9 tokens/s

quant:

Q4_K (GGUF)

mtp (multi-token prediction):

off

User reports GLM-5.3 at 9.8 t/s generation and 97.9 t/s prompt processing on two AMD Strix Halo nodes with 128GB each. Setup is the staged GLM path over OdinLink USB4/TB5 RDMA with Q4_K, TP=2 across two Radeon 8060S iGPUs, fixed 4,096 prompt tokens and 300 generated tokens at prefill batch 256. The source gives exact rates of 97.89 t/s prefill and 9.77 t/s decode from a single diagnostic run, with no TTFT reported and no single-node scaling claimed.

generation:

10.1 tokens/s

prompt processing (prefill):

103.4 tokens/s

quant:

Q4_K (GGUF)

mtp (multi-token prediction):

off

User reports GLM-5.3 Flash Q4_K at 10.1 t/s generation and 103.4 t/s prompt processing on two AMD Strix Halo nodes, each with 128 GB installed memory. Setup is a custom DS4 fork's staged GLM TP=2 ROCm path with a 4,608-token context allocation, 4,096-token prompt and 300-token decode at prefill batch 256, no speculative draft or MTP. The 4K point uses a cross-disciplinary prompt and is one observation; the source waived repeated timing, so no spread or confidence interval is established.

generation:

164.8 tokens/s

quant:

W4A16-NVFP4-K32-Experts-FP8-WO (safetensors)

kv:

FP8 E4M3

User reports GLM-5.3-Flash at 164.8 tok/s single-stream decode (388.2 tok/s aggregate at four concurrent requests) on two RTX PRO 6000 Blackwell GPUs. Setup is SGLang v0.4.3 SM120 with the ormandj W4A16-NVFP4-K32-Experts-FP8-WO checkpoint, FP8 E4M3 KV cache, 524,288-token context, adaptive EAGLE 5/1/6 speculative decoding and tensor parallel size 2. Cold prompt processing reaches 5,174 tok/s at 2K and 5,961 tok/s at 32K. Four simultaneous requests is the tuned operating point; at eight, aggregate falls 7.7% and median time to first token rises to 1,428 ms. The 4- and 8-GPU capacity figures are arithmetic projections, not measurements.

generation:

48.8 tokens/s

quant:

4bit (MLX)

User reports GLM-5.3-Flash Abliterated at 48.8 tok/s decode on a Mac Studio M3 Ultra 256 GB. Setup is mlx-serve 26.10.1 with MLX 4-bit weights and 131072 context, MTP on with draft rerank, resident size about 172–181 GB. Real sessions at 50k–85k tokens decode about 45–63 tok/s with prefill 150–300 tok/s on cache hits. A short probe at 8192 context measured 49.9 tok/s. TensorFold 1.0.5 on the same Mac left the MTP head on disk and one short reply measured 15.7 tok/s.

generation:

18.3 tokens/s

prompt processing (prefill):

320.0 tokens/s

quant:

Maya-S-v2-IQ2_XXS (GGUF)

User reports GLM-5.3 Flash (Maya-S-v2 IQ2_XXS) at 18.3 t/s generation and 320 t/s prompt processing on an AMD Strix Halo 128GB. Setup is Project Maya 1.0.30 on HIP with the Maya-S-v2-IQ2_XXS mixed quant, one request, 262K configured context, the whole model in unified memory; whether MTP was on is not stated. Source: https://github.com/mw00/project-maya/issues/102 (LIN-LAB-AI, 2026-10-11; third-party measurement, not our run). Headline metrics are separate built-in --bench tests: 320 tok/s prefill for 8,192 input tokens; generation 18.3 tok/s reported mean, three 256-token answers (18.3/18.4/18.3), greedy, thinking off. Short/warm decode is not generation following that 8K prefill. Also: 267 tok/s prefill for 2,048 input tokens. Prefill repetition count unspecified. Configured context 262,144, not headline prompt length. Hardware/runtime: one Radeon 8060S (gfx1151), Ryzen AI Max+ 395, 128GB unified; 96GiB BIOS carve-out. Windows reports 31.65GiB host RAM; HIP reports 107.87GB GPU memory. These are overlapping unified-memory views, not additive capacities. Native Windows 11 build 26100; HIP SDK 7.2/clang21; Maya 1.0.30 c189f7a; --backend hip --gpu 0; NVMe model storage; --prefill auto, PREFILL_SUB=1024, PREFILL_MB=6144. Separate cold technical-prose sweep asserts reused=0 and uses engine prompt_ms/decode_ms: at 8,632–9,070 input tokens median 315.1 pp / 15.90 tg; at 261,748–261,771 input median 274.5 pp / 14.20 tg. Answers usually ~65 tokens, so not comparable to 256-token headline. No downloadable per-request sweep logs linked. KV precision, Flash Attention, MTP status, clocks/power and exact device manufacturer unspecified. Verified model identity: peasantsmith/GLM-5.3-Flash-Maya-GGUF, folder Maya-S-v2-IQ2_XXS, three GLM-5.3-Flash-Maya-S-v2-IQ2_XXS-0000{1,2,3}-of-00003.gguf shards. Source reports SHA256 verification against its launcher. Tested-version manifest: https://github.com/mw00/project-maya/blob/v1.0.30/maya.py . It accepts two first-shard headers and uses revision main; exact tested Hugging Face commit is unknown. Quant is mixed: IQ2_XXS gate/up experts, IQ2_S/IQ3_XXS down, Q6_K attention, including an MTP block (presence does not establish use). Model card/file verification: https://huggingface.co/peasantsmith/GLM-5.3-Flash-Maya-GGUF/tree/main/Maya-S-v2-IQ2_XXS and https://huggingface.co/peasantsmith/GLM-5.3-Flash-Maya-GGUF . Card identifies 321B total/~18B active and 96.5GB text weights. Source 90.90GiB on-disk includes vision files; not peak RAM. Vision was untested. No quality benchmark reported.

generation:

61.0 tokens/s

User reports GLM-5.3-Flash (320B) at ~61 tok/s generation on an M5 Ultra Mac Studio with 256GB unified memory. Setup is llama.cpp with GLM's built-in MTP head (probabilistic drafting, 3 draft tokens); without MTP generation is 39.7 tok/s. The full 262K context fits with ~15 GiB to spare. Prompt reading is ~765 tok/s at 4K tokens and ~630 tok/s at 59K; a 29K-token prompt takes 45 s. Long-context recall was correct at 58.8K tokens. Images and MTP cannot be enabled simultaneously, and deep context on Metal needs an unmerged fix.

generation:

19.1 tokens/s

prompt processing (prefill):

41.0 tokens/s

quant:

GSQ-RCO 3.5-bit (GGUF)

User reports GLM-5.3-Flash at 19.1 t/s decode and 41 t/s prefill on an RTX 3090 with a 47-token prompt. Setup is Strata with GSQ-RCO 3.5-bit GGUF at 32,768 tokens of context, most experts in RAM (121 GiB of 177 GiB system RAM used). Prefill rises with prompt length to 974 t/s at 26,641 tokens; decode stays 18.7-19.0 t/s across contexts. On 9 GPUs (8x RTX 5060 Ti 16 GB plus the 3090) decode is about 28 t/s and 8K prompts run 1150-1210 t/s. Calibration kept the engine's own settings at 21.6 t/s.

generation:

28.6 tokens/s

quant:

exl3-3.05bpw (EXL3)

User reports GLM-5.3-Flash at 28.55 tok/s decode on an RTX 3090. Setup is ExLlamaV3 with exl3-3.05bpw and 131k context, all-RAM fast mode with CPU offload requiring ~238 GB free RAM. Lab suite passed 6/6 with 28.6 tok/s decode and 989 tok/s prefill; at concurrency 4 the aggregate was 28.62 tok/s (28.45 per stream) because fast mode decodes one stream at a time.

generation:

17.8 tokens/s

User reports GLM-5.3-Flash at 17.8 tok/s decode and 996 tok/s prefill on an RTX 3090 with the 55 GB NVMe offload launch. Setup is the glm53-flash-offload v4.4-nvme image, with the model's weights streamed from NVMe (55 GB) plus 16 GB resident; the 16 GB launch measured 13.3 tok/s decode and 986 tok/s prefill and failed the 15 tok/s speed gate. Previous runs were 17.3 tok/s for 55 GB and 13.1 tok/s for 16 GB. The 55 GB launch passed all gates and was accepted; the 16 GB launch was reported. Host settings outside the launch hash were --gpus device=0 and --cpuset-cpus 2-39 with no read cap.