Qwen3.5 9B
on 8× NVIDIA T4 16GB · llama.cpp · 87,000 ctx
Use cases
agenticsummarizationtool-use
Summary
User asks how to serve Qwen3.5-9B to 10-15 concurrent users on an Asus ESC4000 G3 with 8x T4 16GB and 128 GB DDR4 RAM. Currently runs 8 separate llama.cpp instances serving Qwen3.5-9B-Q5_K_XL at 87k context per slot with 44 slots, each slot generating 25-40 t/s and prefill at 1400 t/s. User is concerned about latency and asks whether vLLM or another engine would be faster, noting the T4s lack FP8 support and that vLLM with AWQ-INT8 was fast on their MI50 32GB.