Qwen2.5 reports

generation:

0.6 tokens/s

quant:

Q4_0 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen2.5-0.5B-Instruct at 0.63 tok/s on a 2006 Dell Latitude D620 laptop with no GPU. Setup is llama.cpp on CPU with Q4_0 GGUF, 512 context, 2 threads, 128 tokens generated; the machine has an Intel Core Duo T2300 and 2 GB DDR2 RAM. A single run took 203.2 s with 41.2 s to first token, and 1024 context crashed with OOM. The user does not recommend it and calls the results not generalizable.

generation:

37.0 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen2.5 32B at 37 tokens per second on an RTX 5090. No engine, quant, context length or KV cache settings are stated. The user is an amateur and has not collected full benchmarks yet, and may post more results later.

generation:

7-8 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User runs Qwen2.5-Coder 7B and Qwen3 8B with Ollama in an LXC container on Proxmox, entirely on the CPU (Intel i5-12400, 32GB DDR4, no dedicated GPU), getting around 7-8 tokens per second. Setup is Ollama with no GPU, 18GB RAM allocated to the container, used for programming via the Continue extension in VS Code. User asks which GPU to buy for this machine; the 7-8 t/s is a range and no GPU was used for the reported run.

generation:

253.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen2.5-1.5B-MoE-A0.9B at 253 t/s text generation on an RTX 4060 8 GB, up 32% over the dense Qwen2.5-1.5B at 191 t/s. Setup is llama.cpp with Q4_K_M GGUF on both models; CPU generation was 77.5 t/s for the MoE versus 54.9 t/s for the dense model, a 41% gain. Perplexity is about 40% higher for the MoE (WikiText-2 14.63 vs 10.41, C4 22.88 vs 15.76) and zero-shot mean acc_norm drops from 0.565 to 0.462, with the largest losses on ARC. The user calls this a proof of concept, noting quality is roughly in Qwen2.5-0.5B range and that the GGUF is larger (1.1 GB vs 0.9 GB) because llama.cpp's MoE architecture needs an untied output head.

generation:

14.3 tokens/s

quant:

4-bit (MLX)

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen2.5-72B-Instruct-4bit at 14.3 t/s on a Mac Studio with an M3 Ultra and 512 GB of unified memory. Setup is MLX 0.31.1 with mlx-lm 0.31.2, 4-bit group_size=64 quantization, no speculative decoding or batching, at a 4132-token prompt with 128 output tokens. The figure is one observation, not a repeated-run mean; the source's more precise value is 14.331110981167 t/s, and the same sweep drops to 6.056028636 t/s at 131108 prompt tokens.

generation:

47.4 tokens/s

prompt processing (prefill):

556.9 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen2.5 3B at 47.4 t/s generation and 556.9 t/s prompt processing on a base Apple M4 Mac mini with 16 GB unified memory. Setup is llama.cpp with Q4_K_M, all layers on the GPU via Metal, measured with llama-bench PP512 and TG128 passes. The figures are means of three repetitions, with 10.42 GiB baseline system RAM in use during the run, so it is not an idle-machine result.

generation:

47.7 tokens/s

prompt processing (prefill):

1512.4 tokens/s

quant:

Q4_K_M (GGUF)

kv:

F16

flash attention:

on

mtp (multi-token prediction):

off

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen2.5-7B-Instruct Q4_K_M at 47.7 t/s generation and 1512.4 t/s prompt processing on a Framework Desktop with an AMD Ryzen AI Max+ 395 and Radeon 8060S. Setup is llama.cpp build baf6360be with f16 K/V cache, flash attention, n_batch and n_ubatch of 512, and one Vulkan device. The figures are separate depth-0 arms: PP512 at 1512.379830 t/s and TG32 at 47.677208 t/s, not one 512-input/32-output run. At depth 32768 the same arm drops to 662.571662 t/s PP512 and 33.575848 t/s TG32.

generation:

76.8 tokens/s

prompt processing (prefill):

602.8 tokens/s

quant:

4-bit (source config) (MLX)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen2.5 7B at 76.8 t/s generation and 602.8 t/s prompt processing on an Apple M2 Max with 64GB unified memory. Setup is MLX via python -m mlx_lm benchmark with the mlx-community/Qwen2.5-7B-Instruct-4bit artifact, batch size 1, 512-token prompt and 128-token generation cap. The 76.8 t/s and 602.8 t/s are three-trial means after discarding Trial 1; retained generation values were 76.773, 76.747 and 76.821 t/s.

generation:

77.0 tokens/s

quant:

Q4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks llama.cpp, LM Studio, and Ollama on an RTX 5060 Ti 16GB and an Apple M3 Max 36GB, serving Qwen2.5-Coder-7B Q4 GGUF. On the RTX 5060 Ti, llama.cpp reaches 77.0 tok/s, LM Studio 76.8 tok/s (0.3% behind), and Ollama 69.1 tok/s (10.3% slower). On the M3 Max, llama.cpp reaches 53.5 tok/s, Ollama 46.2 tok/s (14% slower), and LM Studio 38.2 tok/s (29% slower). The metric is wall-clock tokens per second over a 12-prompt suite with 256 tokens generated per prompt and greedy decoding. The author attributes the gap to wrapper overhead rather than the shared llama.cpp engine.

generation:

93.4 tokens/s

prompt processing (prefill):

4842.2 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen2.5 Coder at 93.4 t/s generation and 4842.2 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at 16384 tokens of configured context, on one 16GB card. The recorded prompt was 1368 tokens, not a 16384-token prefill, and the figures are medians of 3 fresh-prefill runs at temperature 0 with a 256-token target.

generation:

171.4 tokens/s

prompt processing (prefill):

9825.5 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen2.5 Coder-7B at 171.4 t/s generation on one NVIDIA RTX 5080 16GB. Setup is Ollama with Q4_K_M at 16384 tokens of configured context, on a Ryzen 7 7800X3D under Windows 11. The figure is the median of 3 fresh-prefill runs with a 1368-token prompt, not a 16384-token prefill; prompt processing was 9825.5 t/s.

generation:

176.6 tokens/s

prompt processing (prefill):

9984.4 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen2.5 7B at 176.6 t/s generation and 9984.4 t/s prompt processing on an NVIDIA RTX 5080. Setup is Ollama with Q4_K_M at a configured 16384-token context, on one 16GB card. The recorded prompt was 1674 tokens, not a full 16384-token prefill, and the figures are medians of 3 fresh-prefill runs.

generation:

94.6 tokens/s

prompt processing (prefill):

4864.7 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing.

User reports Qwen2.5 14B at 94.6 t/s generation and 4864.7 t/s prompt processing on one NVIDIA RTX 5080 16GB. Setup is Ollama with Q4_K_M at a configured 16384-token context; the recorded prompt was 1674 tokens, not a full 16384-token prefill. The figures are medians of 3 fresh-prefill runs, with 256 target output tokens and a unique nonce per repetition to avoid prompt-cache hits.

generation:

32.1 tokens/s

quant:

FP16 (safetensors)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen2.5-1.5B-Instruct at 32.14 tok/s single-stream on an AMD Instinct MI50 32GB with vLLM. Setup is vLLM with unquantized FP16 weights on one MI50, one request, no speculative decoding, enforce-eager, ROCm flash attention, max model length 8192, in an unprivileged Proxmox LXC container. Under continuous batching aggregate throughput rises to 63.8 tok/s at 2 streams, 238.9 at 8 and 422.8 at 32. llama.cpp with 4-bit GGUF weights on the same card gave 31.25 tok/s single-stream and stayed near 31 at every concurrency. Qwen2.5-0.5B reached 648.2 tok/s aggregate.

generation:

11.0 tokens/s

quant:

Q4_K_M (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen2.5-0.5B-Instruct at ~11.01 tok/s on a Samsung Galaxy A50s (Exynos 9611) phone, CPU only. Setup is llama.cpp in Termux with Q4_K_M GGUF, ARM64 NEON, 4 threads; model uses ~542 MB RSS and ~463 MB on disk, TTFT ~2858 ms. The run drives an offline Android AI Bridge workflow via llama-server on localhost:8080.

generation:

95.6 tokens/s

quant:

Q4 (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks Qwen2.5-0.5B Q4 at 95.6 t/s decode with 18 ms TTFT on an Apple M1 16GB, averaged across defaults and normalized modes on the chat_turn workload. Setup is llama.cpp (llama-server) with Q4 GGUF weights, Metal backend, 1 warmup plus 4 measured reps per cell. LlamaStash leads raw llama-server (91.9 t/s / 20 ms), LM Studio (88.4 / 68) and Ollama 0.24.0 (79.6 / 102); in defaults mode LlamaStash reaches 99.0 t/s / 15 ms vs raw llama-server's 92.3 / 19, and normalized mode collapses the gap to under 1% (92.2 vs 91.5).

generation:

17.6 tokens/s

quant:

bf16 (safetensors)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User benchmarks vLLM against SGLang on one NVIDIA L4 with Qwen2.5-7B-Instruct in bf16, 128 output tokens per request with ignore_eos. Setup pins the model, GPU class, dtype, context length and memory fraction across both engines, driven through /v1/completions by a Go load generator that times TTFT at the client. Prefix caching is on except where noted. At one request each engine decodes about 17.5 t/s; at 32 concurrent short prompts vLLM reaches 459.3 t/s and SGLang 449.0 t/s. On long prompts at 32 concurrent requests vLLM gets 252.7 t/s and SGLang 310.2 t/s; turning prefix caching off drops vLLM to 147.1 t/s and raises p99 TTFT from 6.7 s to 18.6 s. SGLang reaches first token sooner under load (258 ms vs 639 ms at 32 concurrent short prompts) while vLLM is quicker at a single request (72 ms vs 120 ms). Cost per million output tokens falls from $12.60 at one request to $0.48 at 32 on the same card. Cold start was 122 s for SGLang and 149 s for vLLM. The author notes one run per cell and that throughput moved by more than the gap between engines on a re-measurement, so small differences are noise.

generation:

34.2 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports qwen2.5-7b at 34.2 tok/s on an RX 580 2048SP, part of a six-model OpenAI-compatible stack running on Mesa RADV Vulkan. Setup is llama.cpp with a model-swapping router; the 30B-A3B MoE ran 18.6 GB from system RAM at 21.3 tok/s, and forcing -ngl 99 dropped it to 8.1 tok/s versus 22.3 tok/s with auto-fit layers. Other models measured: qwen2.5-coder-1.5b 99.2 tok/s, qwen2.5-vl-3b 66.4 tok/s, ornith-1.5-9b 28.0 tok/s. Model swaps take 7.9-38.5 s; SD 1.5 renders in 27.5 s and SD 3.5 Medium in 53.3 s per 512x512.

generation:

99.2 tokens/s

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports qwen2.5-coder-1.5b at 99.2 tok/s on an RX 580 8GB. Setup is llama.cpp with Vulkan (Mesa RADV) backend, temp 0, max_tokens 300, on a Ryzen 5 5600G with 32 GB RAM. Other models measured: qwen2.5-vl-3b 66.4 tok/s, qwen2.5-7b 34.2 tok/s, ornith-1.5-9b 28.0 tok/s, qwen3-30b-a3b 21.3 tok/s (18.6 GB, from RAM). Forcing -ngl 99 on the MoE cost 8.1 vs 22.3 tok/s with auto-fit layers. Model swap takes 7.9-38.5 s. Agent first message ~161 s on a 20k-token prompt, ~25 s after caching.

generation:

115.0 tokens/s

quant:

GPTQ-Int4 (GPTQ)

kv:

fp16

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen2.5-7B-Instruct-GPTQ-Int4 at 115 tok/s single-stream decode on a Radeon RX 7900 XT (gfx1100) under native Windows with AMD ROCm. Setup is vLLM v0.19.1 with GPTQ Int4 weights, fp16 KV cache, native exllama W4A16 GEMM and hipGraph decode capture (FULL_DECODE_ONLY), gpu_memory_utilization=0.9, verified spill-free. Aggregate throughput scales with concurrency: ~73 tok/s at batch 4, ~232 at batch 16, ~358 at batch 32. KVarN KV-cache quantization (K 4-bit / V 2-bit) gives 4.74x KV capacity (999,296 vs 210,784 tokens) at 74.7 tok/s, about 35% slower. A community-validated RX 7800 XT (gfx1101) reached 81.9 tok/s on the same model.