Local model performance on your hardware

generation:

113.0 tokens/s

prompt processing (prefill):

2800.0 tokens/s

quant:

MXFP4 (MXFP4)

kv:

BF16

User reports MiniMax-M3 at ~113 tok/s single-stream decode on 4x RTX PRO 6000 Blackwell (96 GB each). Setup is vLLM with the MXFP4 checkpoint (~256 GB, ~60 GB per card), BF16 KV cache, tensor parallelism 4, and 250K context; cold prefill runs ~2,800 tok/s flat from 8K to 131K. Four concurrent requests reach ~184 tok/s aggregate. A second NVFP4/SGLang configuration on the same host decodes at ~87 tok/s single-stream with a ~643K-token KV pool and image/video support.

generation:

88.5 tokens/s

quant:

NVFP4 (safetensors)

kv:

BF16

User reports MiniMax-M3-uncensored-NVFP4 at about 88.5 tokens/s for a warmed-up single-request decode on 4x NVIDIA RTX PRO 6000 Blackwell 96 GB. Setup is SGLang with ModelOpt NVFP4 routed-expert weights, BF16 KV cache, tensor parallel size 4, 32,768-token configured context, CUDA graph enabled, and two concurrent requests allowed. The figure is a measured short-request decode rate after warmup; the checkpoint was compressed from 854.2 GB BF16 to 260.4 GB. Long-context probe recalled a sentinel from a 20,212-token prompt, and vLLM was not accepted for this deployment.

generation:

17.3 tokens/s

quant:

4bit (MLX)

User reports a bug in oMLX 0.6.4 distributed clustering where the coordinator fails to release RAM and GPU after a crash. The model is mlx-community/MiniMax-M3-4bit at 236 GB, run across rank 0 on a Mac Studio M3 Ultra 256GB and rank 1 on a Mac Studio M2 Ultra 192GB. The first completion produced 17 tokens from a 7,693-token prompt at 17.3 tok/s. The crash was triggered by overlapping requests with pipeline_prefill_overlap and coalesced batching. After the crash, roughly 116 GB of wired memory has no owning process and the GPU stays pinned at 100%, requiring a reboot. The M2 Ultra worker released memory cleanly. The user also reports that MiniMax-M3 crashes the cluster after the first prompt.