- generation:
- 187.0 tokens/s
- quant:
- FP8 (FP8)
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Metis-1.6 Logos decoding FP8 at 187 tok/s single-stream on an RTX 5070 Ti with the author's Dromos engine, against 118 tok/s for vLLM. Dromos speculates from context for about 1.3x on chat in direct mode; the model is a looped Mamba-2 plus latent-MoE design with 10.23B total and 1.18B active parameters and 131K context. Across 32 streams Dromos reaches 1,331 tok/s versus vLLM's 1,143. The post is a release write-up: the author flags that AIME, HMMT and other olympiad benchmarks are not yet in the decontamination manifest, and that thinking runs a median of about 25k tokens.