Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks Qwen2.5-7B-Instruct-AWQ on two Kaggle T4 GPUs, comparing single GPU, data parallel and tensor parallel serving with vLLM 0.30.0.
Tensor parallel (TP=2) reached 58 tok/s for one user against 32 tok/s for single GPU and data parallel, with 276 ms time to first token. At 32 concurrent requests TP reached 437 tok/s aggregate output throughput, against 382 tok/s for data parallel and 215 tok/s for single GPU.
Benchmarks used 512-token prompts and 128-token replies, run inside Kaggle through a load-balancing proxy. The two T4s are connected over PCIe PHB with no NVLink, and the user notes the result may differ on a host with more than 4 vCPUs.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User compares GGUF and MLX inference speeds on a Mac Studio M2 Ultra with 192GB, using LM Studio 0.3.10 with MLX 0.6.
For Qwen2.5 72B 8-bit, GGUF reaches 7.68 tokens/sec while MLX drops to 0.42 tokens/sec. Smaller models show the opposite: Qwen2.5 0.5B 4-bit runs at 79 tokens/sec in GGUF versus 317 tokens/sec in MLX, and Mistral Small 22B 8-bit is roughly equal at 25.01 versus 25.96 tokens/sec.
User reports MLX is 4-5x faster for very small models but about 10x slower for large ones, with the inflection point around 22B parameters, and notes MLX does not use much GPU for larger models while LM Studio lacks a layer-offload option for MLX.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
coding
User benchmarks Qwen2.5-Coder-7B-Instruct Q4_K_M on an RTX 4090, comparing llama.cpp at 158.90 tok/s decode against apr 0.64.0 at 103.26 tok/s.
Setup is llama.cpp build 39173bcac and apr 0.64.0 CUDA with the same GGUF, profile medium, 15s warmup, 3 x 30s runs, 10s cooldown, concurrency 1, streaming.
Decode is 0.650x of llama.cpp, with ITL p50 6.29 ms vs 9.68 ms and throughput 1.24 vs 0.77 req/s. The user notes apr's figure is 2.6x the ~40 tok/s 7B reference and that an earlier 11.8 tok/s report was an end-to-end rate including ~9.7s model load.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen2.5-0.5B Q4_K_M at 69.3 t/s on CPU and 5.2 t/s on the OpenCL GPU path on an ASUS ROG Phone 7 (Snapdragon 8 Gen 2 / Adreno 740).
Setup is llama.cpp via a llama.rn fork with an experimental OpenCL backend, off by default; the lm_head K-quants are routed back to CPU to avoid an Adreno bug, and Flash Attention q8_0 is rejected on the A7X driver so attention falls back to CPU.
User notes the OpenCL path produced garbage tokens on this driver, and that Gemma 2 9B Q4_K_M was still about 2.1x slower on GPU. GPU is recommended only for 7B-9B+ models on drivers that do not miscompile K-quant GEMV.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
User benchmarks Qwen 2.5-Coder 14B Instruct Q4_K_M at 33.51 t/s generation and 1210.29 t/s prompt processing on an AMD RX 9060 XT 16GB via Vulkan.
Setup is llama.cpp build c4ae9a88f8 with -ngl 99 and -fa 1; the same model on ROCm reaches 30.79 t/s generation and 1211.32 t/s prompt processing.
A Qwen 3.5 9B Q4_K_M run reaches 50.15 t/s generation and 1904.16 t/s prompt processing on Vulkan, and a 14B quant sweep gives about 32, 28 and 20 tok/s for Q4_K_M, Q5_K_M and Q8_0.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
codinglong-context
User benchmarks Qwen2.5-32B-Instruct-AWQ at 71.37 tok/s median decode on 2x RTX PRO 4000 Blackwell.
Setup is SGLang 0.5.19 with AWQ weights, NGRAM speculative decoding (K=5), tensor parallelism 2 over PCIe Gen4, and FP8 KV cache.
NGRAM speculative decoding gives a 1.28x median speedup over the 55.60 tok/s baseline, with high inter-prompt variance (std dev 19.18 tok/s). Per-domain decode ranges are 65-107 tok/s for JSON, 60-226 tok/s for code, and 58-82 tok/s for prose. FP8 KV cache alone measured 54.77 tok/s, a 1.5% regression.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports Qwen2.5 7B Instruct at 49.0 tok/s on an RTX 3060 12GB, and 6.4 tok/s CPU-only on the same machine.
Setup is Ollama 0.11.4 with Q4_K_M weights, single-stream, warm speed as the median of three runs with 200-token answers. About 1.2 GB of the card was already in use by the desktop and a Stable Diffusion web UI.
The same model measured 48.9 tok/s on the GPU in a separate CPU-vs-GPU comparison, with a cold first answer of 5.6 s on GPU against 31.3 s CPU-only, a 7.6x speed-up. The user notes the CPU is a 2012 Xeon E5-2665 without AVX2, and that a modern CPU with AVX2 or AVX-512 will beat these CPU numbers.
User reports a crystallization experiment on Qwen2.5-7B-Instruct using an activation-steering engine called AkbasCore 3.0 / SEASC on 1× NVIDIA A100-SXM4-40GB.
Setup uses LoRA gradient updates to transfer part of the induced layerwise representation geometry into the model weights, then removes the runtime steering motor and adapter. Training ran 1.549 min (~93 seconds) and consumed 2.6329 Wh of measured GPU-board energy at a time-weighted mean board power of 102.33 W.
The locked final test showed +15.7881% progress with the motor off, best state at update 40, and automatic overshoot detection at update 70. An unresolved merge diagnostic reports 1.762% relative hidden-state drift.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User reports a 7B Qwen2.5 model at 28.87 tok/s on an RTX 5050 Laptop 8GB in an MSI laptop with an Intel i7 240H and 32 GB DRAM, with 0.53 s time to first token.
With RAG retrieval in the loop, generation drops to 25.18 tok/s and TTFT rises to 2.35 s including the knowledge search.
A legacy LG Gram with an Intel i7-6500U, 8 GB DRAM and integrated graphics is used for comparison, running 0.5B to 1.5B models. Quantisation format and context length are not stated.
User benchmarks quantization accuracy damage on Qwen2.5-1.5B and 0.5B across the Q8 to Q2 ladder, running 24,600 generations on an 8GB M2 Air with llama.cpp and MLX.
On the 1.5B model, JSON schema conformance stays at 100% at every rung including Q2, GSM8K arithmetic falls from 56.5% to 19.0% with the first significant drop at Q3, and TriviaQA recall falls from 19% to 9%, breaking only at Q2.
Q4 significantly beat Q8 on the 1.5B model in both runtimes (p=0.044 llama.cpp, p=0.015 MLX), while on the 0.5B model MLX 4-bit collapses from 45% to 27.5%. Q2_K is strictly dominated on 0.5B: Q3 gives 40.5% at 122 tok/s versus Q2 at 13.5% and 119 tok/s.
User measures full next-token distributions before and after quantization on Qwen2.5-0.5B, using 500 probe positions and a float32 CPU reference under transformers 4.57.6 on a T4.
GGUF Q4_K_M moves the distribution by 0.0951 mean Hellinger distance and raises perplexity by 5.02%, recovering 67% of the distributional damage and 89% of the perplexity gap relative to an uncalibrated 4-bit group-128 baseline.
User notes the 0.5B model size limits conclusions about AWQ and GPTQ, and that 128 calibration samples at 512 tokens is modest.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.
User benchmarks abliteration tools Apostate, Huihui and Heretic on Qwen 2.5 7B on an RTX 5090 32GB.
Setup is vLLM 0.19.0 in bf16, evaluated with lm-evaluation-harness.
The run reports MMLU, GSM8K, HellaSwag, ARC Challenge, WinoGrande, TruthfulQA MC2, PiQA, LAMBADA ppl, HarmBench ASR and KL divergence. No tokens/sec is reported.
Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.
coding
User reports Qwen2.5-Coder-32B at about 28 t/s generation and around 450 t/s prompt processing on a single RTX 3090.
Setup is llama.cpp build b4400 with a Q4_K_M quant at 32k context and a Q8 KV cache.
User calls it solid for autocomplete, but says it occasionally hallucinates imports in multi-file refactors.