Summary
User reports PXA, a fork of ik_llama.cpp with a vLLM plugin, at 65.7 t/s plain decode with TP4 on an 8x V100 SXM2 NVLink system.
Setup is PXA as shipped with tensor parallel across four cards and no speculative decoding, measured on 16 GSM8K questions with 192 greedy tokens.
With speculative decoding at k=7 the same TP4 setup reaches 159.6 t/s, and prefill at 3k is 2273 t/s with a TTFT of 1.4 s. 1Cat vLLM on the same system reads 61.0 t/s plain decode at TP4.