Qwen3.8 27B
on NVIDIA L4 · llama.cpp
Summary
User reports Qwen3.8-27B at 14.6 tok/s on a Colab L4 GPU.
Setup is llama.cpp with Q4_K_M weights, one pass at the preset's defaults.
The same model measured 47.2 tok/s on an A100 40GB and 71.2 tok/s on a G4 (RTX PRO 6000). The post also lists EXL3 4.0 bpw runs (16.3 / 47.5 / 65.3 tok/s) and Qwen3.8-Flash-Next IQ2_XS under Strata (28.4 / 42.2 / 87.9 tok/s), plus compute-unit costs of 1.54 CU/h for the L4, 5.30 for the A100 and 8.90 for the G4.