Local model performance on your hardware

generation:

35-40 tokens/s

quant:

Q5_K_XL (GGUF)

User reports serving Qwen3.5-9B at 35-40 t/s per single user on a server with 8x T4 GPUs and 128 GB DDR4 RDIMM. Setup is llama.cpp with Q5_K_XL at 87k context per slot, one llama.cpp instance per T4, 44 slots total behind an nginx round-robin load balancer. TTFT is about 3-5 seconds depending on context size. The user is asking for advice on fine-tuning the model with QLoRA on roughly 500 GB of raw data, and lists other hardware (2x V620, 2x MI50 32GB, 2x RTX 3060 12GB) that has boot problems in several motherboards.

generation:

55-57 tokens/s

quant:

EXL3 4.0bpw (EXL3)

User benchmarks Qwen3.8-27B on two free Kaggle T4s (no P2P, 70W cap per card) with EXL3 4.0bpw under exllamav3, tensor parallel over PCIe, and a DFlash2 block drafter of 7 tokens. Overall throughput is 55-57 tok/s with drafting versus ~18-19 without; HumanEval reaches 154 tok/s and prose 35-37 tok/s. Prose acceptance is the limit at about 2.5 accepted tokens per step versus ~7 on code. On the same cards llama.cpp best was 16.5 tok/s and vLLM int4 with its own MTP drafter did 40.8 tok/s.

generation:

25-40 tokens/s

quant:

Q5_K_XL (GGUF)

User asks how to serve Qwen3.5-9B to 10-15 concurrent users on an Asus ESC4000 G3 with 8x T4 16GB and 128 GB DDR4 RAM. Currently runs 8 separate llama.cpp instances serving Qwen3.5-9B-Q5_K_XL at 87k context per slot with 44 slots, each slot generating 25-40 t/s and prefill at 1400 t/s. User is concerned about latency and asks whether vLLM or another engine would be faster, noting the T4s lack FP8 support and that vLLM with AWQ-INT8 was fast on their MI50 32GB.

generation:

58.0 tokens/s

quant:

AWQ (AWQ)

User benchmarks Qwen2.5-7B-Instruct-AWQ on two Kaggle T4 GPUs, comparing single GPU, data parallel and tensor parallel serving with vLLM 0.30.0. Tensor parallel (TP=2) reached 58 tok/s for one user against 32 tok/s for single GPU and data parallel, with 276 ms time to first token. At 32 concurrent requests TP reached 437 tok/s aggregate output throughput, against 382 tok/s for data parallel and 215 tok/s for single GPU. Benchmarks used 512-token prompts and 128-token replies, run inside Kaggle through a load-balancing proxy. The two T4s are connected over PCIe PHB with no NVLink, and the user notes the result may differ on a host with more than 4 vCPUs.

generation:

1-2 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

turbo3_tcq

User profiles Kimi K2.6 on a Tesla T4 16GB with an Intel Xeon Gold 5120 and 740 GB RAM, reporting it works but is unusable at 1-2 tokens/s with a 50k context. Setup is llama.cpp with the Unsloth UD-Q4_K_XL GGUF (~544 GiB, 14 shards merged), turbo3_tcq KV cache, 49920 context, --n-cpu-moe 61 so all 61 layers and 384 experts run on CPU. The Q8_K_XL quant is not viable on the T4, and the notes also profile hypothetical RTX 4090 and RTX 5090 hosts where all layers still run on CPU.

quant:

Q4_K_M (GGUF)

User measures full next-token distributions before and after quantization on Qwen2.5-0.5B, using 500 probe positions and a float32 CPU reference under transformers 4.57.6 on a T4. GGUF Q4_K_M moves the distribution by 0.0951 mean Hellinger distance and raises perplexity by 5.02%, recovering 67% of the distributional damage and 89% of the perplexity gap relative to an uncalibrated 4-bit group-128 baseline. User notes the 0.5B model size limits conclusions about AWQ and GPTQ, and that 128 calibration samples at 512 tokens is modest.

generation:

17.6 tokens/s

prompt processing (prefill):

159.6 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

Q8

User reports Qwen3.8-Flash-Next, 180B total with 6B active across 512 experts, generating 17.6 t/s on short prompts and 16.1 t/s at 12.5K context. Non-expert weights run on a T4 using 4606 MiB, with the experts held in host RAM. Prompt processing reaches 159.6 t/s on a cold 12.5K prompt. User reports solid refactor and coding performance, and finds it less verbose than Opus.