Local model performance on your hardware

generation:

49.6 tokens/s

prompt processing (prefill):

2370.2 tokens/s

quant:

2.20 bpw (EXL3)

User reports MiMo-V2.6-Flash-RL at 49.57 tok/s decode (p50, single stream) on an NVIDIA RTX 6000 Ada 96GB. Setup is ExLlamaV3 (vcruz305 fork) with a 2.20 bpw EXL3 pack (86.94 GB) at 65,536-token context and a 65,536-token KV pool; prefill measured 2,370.2 tok/s p50. With the DFlash drafter attached, decode rises to 184.11 tok/s p50 and prefill is 2,257.7 tok/s. Aggregate throughput under load reaches 236.2 tok/s at C=8 without the drafter and 330.6 tok/s with it; the eight-stream per-stream decode p50 is 34.95 and 54.46 tok/s respectively. The 2.50 bpw pack (98.48 GB) does not fit a 96 GB card. On a DGX Spark / GB10 unified-memory host the 2.50 bpw pack runs at 34.9 tok/s decode (code) and 23.7 tok/s (prose) with draft_accept 0.78.

generation:

260.0 tokens/s

quant:

Q4_K_M (GGUF)

kv:

q8_0

User reports Ornith-Agents-A1-3.6-35B-A3B (a dare_ties merge on Qwen3.6-35B-A3B with a grafted MTP block) at up to ~260 tok/s on an RTX 6000 Ada with 96GB VRAM. Setup is llama.cpp with Q4_K_M GGUF, q8_0 KV cache, 131072 context, --parallel 4, and native MTP speculative decoding (--spec-type draft-mtp, --spec-draft-n-max 4). Model used ~25 GB of the 95.6 GB VRAM. MTP draft acceptance stayed in the 42-48% range across generation lengths; measured throughput converged to ~245-260 tok/s. Q8_0 with MTP gave 190 tok/s, MTP disabled gave 150-200 tok/s, and vLLM measured 130 tok/s. spec-draft-n-max 3 gave 270 tok/s and spec-draft-n-max 2 gave 277 tok/s.