Local model performance on your hardware

generation:

7.3 tokens/s

prompt processing (prefill):

234.1 tokens/s

quant:

UD-Q8_K_XL (GGUF)

flash attention:

on

User reports Muse-Glimmer-30B at 7.3 t/s generation and 234.1 t/s prompt processing on a single Radeon 8060S iGPU in an AMD Strix Halo system with 128 GB unified memory. Setup is llama.cpp with the Unsloth UD-Q8_K_XL quant, full offload, batch 2048 and flash attention, run as separate llama-bench pp512 and tg128 microbenchmarks rather than one end-to-end request. The main table figure is rounded; the source narrative gives approximately PP234 and TG7.25. A separate DFlash server test reports 13.53 t/s, which is not tg128 and is not part of this row.

generation:

28.5 tokens/s

prompt processing (prefill):

253.9 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Muse-Glimmer-30B at 28.5 t/s generation and 253.9 t/s prompt processing on a Mac Studio M4 Max with 128 GB unified memory. Setup is llama.cpp b10794 with UD-Q4_K_XL and F16 K/V cache, flash attention, 99 GPU layers requested, single sequence. Figures are separate 512-token prefill and 128-token generation microbenchmarks at prefix depth 0; at depth 4096 generation is 27.82 t/s and at 16384 it is 27.07 t/s.

generation:

13.2 tokens/s

prompt processing (prefill):

370.9 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

User reports Muse Glimmer 30B at 13.2 t/s generation and 370.9 t/s prompt processing on an AMD Strix Halo 128GB machine. Setup is llama.cpp build 11146 with Q4_K_M, F16 K/V cache, ngl 999, FA1, batch 2048 and ubatch 512, one Vulkan device. The headline pair is pass 1 pp512/tg128 at prepared depth 0, r10; tg128 at prepared depth 8192 measured 13.0 t/s.

generation:

8.6 tokens/s

User reports Muse at 8.6 t/s on an Apple Mac mini M6 with 32 GB unified memory. Setup is LM Studio with an unknown quantization and a loaded context of 32,768 tokens on one machine. The 8.6 t/s is the median of three English-prose runs, client-observed streaming decode throughput rather than a pooled or peak rate; the JSON task passed 0 of 3 automatic checks.

generation:

63.0 tokens/s

prompt processing (prefill):

3241.8 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

User reports Muse Glimmer 30B at 63 t/s generation and 3241.8 t/s prompt processing on two RTX PRO 5000 Blackwell cards, 72 GB and 48 GB. Setup is llama.cpp CUDA b10941 with Q8_0, f16 K/V cache, flash attention on, batch 2048 and microbatch 512, tensor split 1/1 across the two GPUs. The 63 t/s is TG128 and the 3241.8 t/s is PP512 from separate synthetic tests, not a combined request; the author reports TG reproducibility around 1% and PP drift up to about 10% between sessions.

generation:

38.5 tokens/s

prompt processing (prefill):

2677.5 tokens/s

quant:

Q8_0 (GGUF)

kv:

F16

flash attention:

on

User reports Muse Glimmer 30B at 38.5 t/s generation and 2677.5 t/s prompt processing on one NVIDIA RTX PRO 5000 Blackwell 72 GB. Setup is llama.cpp CUDA b10941 with Q8_0, f16 K and V cache, flash attention on, batch 2048 and microbatch 512, whole model on GPU0. The 38.5 t/s is TG128 and the 2677.5 t/s is PP512, each the mean of five repetitions; the author reports generation reproducibility around 1% and prompt-processing drift up to about 10% across sessions.

generation:

26.7 tokens/s

prompt processing (prefill):

269.0 tokens/s

quant:

int4hu block32 symmetric

User reports Muse at 26.7 t/s generation and 269 t/s prompt processing on an M4 Max 128GB. Setup is Apple's coreai-pipelined GPU engine with int4hu block32 symmetric quantization, text decoder only, 16.35 GB bundle size. The source table pairs 269.0 tok/s prompt with 26.69 tok/s decode under llm-benchmark at 512 prompt / 1024 generation, 3 trials; aggregation, cache state and batch configuration for that row are not stated.

generation:

42.0 tokens/s

prompt processing (prefill):

684.9 tokens/s

quant:

IQ4_XS (GGUF)

kv:

Q8_0

User reports Glimmer 30B at 41.97 t/s generation and 684.86 t/s prompt processing on an RTX 3060 12GB plus RTX 3070 8GB. Setup is llama.cpp with IQ4_XS weights, Q8_0 KV cache, 65,536-token context, Flash Attention on, and a DFlash draft model (Q4_0) on CUDA1 with a 45/55 layer split. The benchmark prompt was 994 tokens, not a filled 65K context; generation ranged 41.71–42.45 t/s over 128 output tokens, with 4.5 s average request time and 9.7 s model load.

generation:

30.0 tokens/s

quant:

Q4_K_M (GGUF)

User reports Muse Glimmer 30B at 29.96 t/s median completion throughput on an RTX 3090. Setup is llama.cpp (build 10437, CUDA) with GGUF Q4_K_M at 8,192 tokens of context, one concurrent request and no speculative decoding. The model used about 18.0 GB of VRAM on the 24 GB card, which ran at a fixed 280 W power limit. Median latency was 6.94 seconds per question; prompt processing speed was not reported separately.

generation:

35.4 tokens/s

prompt processing (prefill):

355.4 tokens/s

quant:

Q6_K_XL (GGUF)

kv:

f16

User benchmarks Muse Glimmer 30B on AMD v620 GPUs, with Q6 on one GPU at 355.38 t/s prefill and 35.38 t/s generation. Setup is a tensor split across two GPUs: Q6 gives 472.27 t/s prefill and 36.32 t/s generation, Q8 gives 550.04 t/s prefill and 26.55 t/s generation. Speculative decoding uses a dflash draft model.

generation:

173.0 tokens/s

prompt processing (prefill):

2654.0 tokens/s

quant:

UD-Q6_K_XL

User reports speculative decoding with DFlash at a peak of 287 t/s and an average of 173 t/s. Prompt processing runs 1000-2100 t/s on short prompts and 2654 t/s on long ones. Draft acceptance is 63.3-97.5%, with a mean accepted draft length of up to 14 tokens, and KV cache reuse is 99.9%. The user compares it to Qwen 3.6 27B and reports no Chinese language-mixing bugs, no overthinking loops, and concise responses.

generation:

20.0 tokens/s

prompt processing (prefill):

308.0 tokens/s

quant:

UD-Q2-K-XL

User reports Muse Glimmer 30B at about 308 t/s prompt processing and about 20 t/s generation on an RX 7600 XT 16GB. Setup is llama.cpp with ROCm, the UD-Q2-K-XL quant and DFlash speculative decoding. The run completed a coding task.

generation:

84.6 tokens/s

quant:

Q5_K_M

User reports speculative decoding with DFlash reaching 84.64 t/s at 38.67% acceptance on a best tuned 256K run, against 17.98 t/s for regular decoding. An NVFP4 hybrid reached 94.36 t/s but with worse perplexity, so Q5_K_M was chosen as best. A context fill of 262,116 tokens completed without OOM using 22,920/24,467 MiB VRAM, with decode at 21.56 t/s.

generation:

30.0 tokens/s

quant:

EXL3-SC 3.00bpw H4 (EXL3)

kv:

Q8

User reports Qwen 3.8 27B at SC2.20bpw H3. User prefers Unsloth UD_Q4_K_XL for coding.

generation:

94.0 tokens/s

prompt processing (prefill):

1400.0 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

F16

User reports Muse Glimmer fits on a single RTX 3090 at Q4_K_XL with full context, DFlash and mmproj, using about 22-23 GB of VRAM. Generation speed ranges from 64 to 124 t/s depending on prose versus code, with an average of 94 t/s. Prompt processing runs at about 1400 t/s. The user notes Qwen3.6-27B and Gemma-4-31B are borderline unusable on the same GPU.

generation:

18.0 tokens/s

quant:

Q4_K_XL (GGUF)

kv:

q4_0

User reports a model fits on a single RTX 5060 Ti 16GB at 131k context with a Q4 KV cache. Setup uses GGUF weights only, with no dflash or mmproj loaded. A Q8 KV cache allows about 90k context.

generation:

233.4 tokens/s

User reports a benchmark on an RTX 5090 with 256k context reachable on 24 GB. The model name is Glimmer, which is not in the canonical list and is extracted as raw.

generation:

253.0 tokens/s

quant:

UD-Q5_K_M

User reports Muse Glimmer 30B at 253 t/s on an RTX 5090 at 262k context. Setup is llama-server with PR #26842 and DFlash, using a dflash draft model. The run used a code patch workload. The user notes Meta's published figure of 233 t/s.

generation:

90-160 tokens/s

quant:

Q5_K_XL

User reports Muse-Glimmer with dflash on an RTX 5090 at roughly 90-160 tok/s depending on the task. The user observes unusual reasoning traces that are disorganized, repetitive, and self-referential, using "we" and mentioning policy. The user compares the reasoning to Gemini models and contrasts it with Qwen 3.6, questions whether reasoning helps or hinders, and also tests a coding prompt for a Flappy Bird HTML game.

generation:

57.0 tokens/s

quant:

BF16

User reports Muse Glimmer 30B at ~57 t/s with speculative decoding using DFlash, against ~25 t/s without speculation, a 2.3x speedup. Setup is vLLM with six patches applied to the image. Mean acceptance length is ~2.5 tokens per verification step, with overall draft acceptance ~10%. Per-position acceptance is ~73% at position 0, ~40% at 1, ~15% at 2, and near zero past 5. The recipe's 3.1x figure was greedy decoding with K-quant on llama.cpp under different conditions. The predicted ceiling is ~65 t/s at 2.6 mean acceptance.