Qwen3.6 35B (3B active)
NVIDIA H100 80GB · Strata · 4,096 ctx
- generation:
- 290.0 tokens/s
- prompt processing (prefill):
- 11486.0 tokens/s
- quant:
- Q8_0 (GGUF)
User benchmarks Qwen3.6-35B-A3B at Q8_0 on an H100 80GB, reporting 290 t/s decode and 11,486 t/s prefill on a 4K prompt. Setup is the RustingStrata engine with Q8_0 weights (37.8 GB) fully in VRAM, CUDA kernels, no speculation for the reported figure; MTP with the model's own draft head reaches 508 t/s decode and costs 4-11% prefill. The post is a multi-GPU roundup: H100 NVL BF16 230 t/s decode, A100 80GB Q8_0 193 t/s, A100 80GB BF16 171 t/s, L40S 48GB Q8_0 165 t/s, all with MTP figures higher. Q8_0 on a 24 GB A30 with 63% of experts resident decodes at 130 t/s (200 with MTP), while BF16 at 31% resident reaches about 12.5 t/s. H100 prefill on Vulkan is 6,883 t/s against 11,486 on CUDA. No perplexity or KLD run exists on these Q8_0/BF16 files; the last check was UD-Q3_K_M on a 3060 at perplexity 9.0714 vs 9.0587 for llama.cpp, mean KLD 0.0055, same top token 96.3%.