Local model performance on your hardware

generation:

14.6 tokens/s

quant:

Q4_K_M (GGUF)

User reports Qwen3.8-27B at 14.6 tok/s on a Colab L4 GPU. Setup is llama.cpp with Q4_K_M weights, one pass at the preset's defaults. The same model measured 47.2 tok/s on an A100 40GB and 71.2 tok/s on a G4 (RTX PRO 6000). The post also lists EXL3 4.0 bpw runs (16.3 / 47.5 / 65.3 tok/s) and Qwen3.8-Flash-Next IQ2_XS under Strata (28.4 / 42.2 / 87.9 tok/s), plus compute-unit costs of 1.54 CU/h for the L4, 5.30 for the A100 and 8.90 for the G4.

generation:

17.6 tokens/s

quant:

bf16 (safetensors)

User benchmarks vLLM against SGLang on one NVIDIA L4 with Qwen2.5-7B-Instruct in bf16, 128 output tokens per request with ignore_eos. Setup pins the model, GPU class, dtype, context length and memory fraction across both engines, driven through /v1/completions by a Go load generator that times TTFT at the client. Prefix caching is on except where noted. At one request each engine decodes about 17.5 t/s; at 32 concurrent short prompts vLLM reaches 459.3 t/s and SGLang 449.0 t/s. On long prompts at 32 concurrent requests vLLM gets 252.7 t/s and SGLang 310.2 t/s; turning prefix caching off drops vLLM to 147.1 t/s and raises p99 TTFT from 6.7 s to 18.6 s. SGLang reaches first token sooner under load (258 ms vs 639 ms at 32 concurrent short prompts) while vLLM is quicker at a single request (72 ms vs 120 ms). Cost per million output tokens falls from $12.60 at one request to $0.48 at 32 on the same card. Cold start was 122 s for SGLang and 149 s for vLLM. The author notes one run per cell and that throughput moved by more than the gap between engines on a re-measurement, so small differences are noise.

generation:

33.0 tokens/s

prompt processing (prefill):

579.0 tokens/s

quant:

UD-Q4_K_XL (GGUF)

kv:

q4_0

User reports Qwen3.8-27B at 32.96 tok/s average decode (24.18 stock) on an NVIDIA L4 24 GB, with 39.4 tok/s on prose and 37.4 tok/s across seven workloads. Setup is llama.cpp (PR #27342 plus a MMVQ crossover patch) with UD-Q4_K_XL GGUF, q4_0 KV cache, 226,048-token context, DFlash 2 Q4_K_M drafter at n-max 7, and GGML_MMVQ_MAX=2; 23.0 of 24.0 GiB resident, ECC off, 72 W cap. Prefill is 579 tok/s at a 7k prompt, 514 at 33k and 387 at 79k; mean accepted length is 3.04. The user notes the card cannot reach 100 tok/s because the 15.75 GiB weight read would need 6.7 accepted tokens per pass, and that a 3090-class card would land near 101 tok/s.