255.0 tokens/s prompt processing (prefill)Prompt processing measures input prefill speed; it does not tell you the time to first token.
Quant
W8A8
System RAM
1024 GB
VRAM reported
11 GB
Summary
User reports custom Turing CUDA kernels for W8A8 INT8 matmul, heterogeneous inference with 4x 11/22GB VRAM and 1TB system RAM, and computation-communication overlap for MoE routing.
The work is open-sourced on GitHub.