GLM-4.5-Air

generation:

138.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-4.5-Air at about 138 tok/s on an RTX PRO 6000 96GB. Setup is llama.cpp with Q4_K_M, measured at empty context. The figure comes from a site of 941 measured rows, including 540 llama-bench rows the user ran on rented GPUs from an RTX 3060 12GB through an RTX 5090 and an RTX PRO 6000 96GB.

generation:

133.0 tokens/s

quant:

FP8 (FP8)

kv:

fp8_e5m2

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports GLM-4.5-Air FP8 at 133 tokens/sec with SGLang on an RTX 6000 Pro Blackwell, compared with about 78 tokens/sec in vLLM. Setup is SGLang with FP8 weights, fp8_e5m2 KV cache, 128000 context length and tensor parallelism across 4 GPUs. The user asks how to reach the same 133 tokens/sec in vLLM with Triton, and whether the Cutlass design is slower than Triton.

generation:

8.3 tokens/s

prompt processing (prefill):

444.2 tokens/s

quant:

UD-Q3_K_XL (GGUF)

kv:

q8_0

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User benchmarks GLM-4.5-Air with Unsloth UD-Q3_K_XL quant on an RTX PRO 6000 96GB with CPU offload for some experts, using ik_llama.cpp at 16k context with q8_0 KV cache. Setup is ik_llama.cpp with UD-Q3_K_XL and q8_0 KV cache, offloading some expert layers to a Ryzen 9 7950X with 128GB DDR5-3600. At 12288 tokens of context, prompt processing reaches 444.23 t/s and generation 8.25 t/s. The user also compares mainline llama.cpp and an Ubergarm IQ3_KT quant, noting a batch warm-up quirk in the forked llama-sweep-bench.

mtp (multi-token prediction):

on

User recommends enabling MTP in llama.cpp for GLM-4.5-Air, a 106B MoE with 12B active params, on 3090s. User notes MTP also works for full GLM-4.5. User mentions creative writing and RP finetunes and Intellect 3.x.