Qwen3.5 9B
on 8× NVIDIA T4 16GB · llama.cpp · 87,000 ctx
Summary
User reports serving Qwen3.5-9B at 35-40 t/s per single user on a server with 8x T4 GPUs and 128 GB DDR4 RDIMM.
Setup is llama.cpp with Q5_K_XL at 87k context per slot, one llama.cpp instance per T4, 44 slots total behind an nginx round-robin load balancer. TTFT is about 3-5 seconds depending on context size.
The user is asking for advice on fine-tuning the model with QLoRA on roughly 500 GB of raw data, and lists other hardware (2x V620, 2x MI50 32GB, 2x RTX 3060 12GB) that has boot problems in several motherboards.