Qwen2.5

reported speed:

382.0 tokens/s generation

quant:

QLoRA

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports a Qwen2.5-3B QLoRA fine-tune at around 382 tok/s on an RTX 4060 Laptop GPU with 8GB VRAM. Setup is the MOLT tool on Windows, training on a one million token dataset with temperature monitoring and thermal pacing. The user asks what GPUs, model sizes, and workflows others use for local fine-tuning.

reported speed:

63.0 tokens/s generation

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

reported speed:

70.0 tokens/s generation · 1850.0 tokens/s prompt processing

quant:

Q6_K_XL (gguf)

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports reliable code generation and codebase ingestion with multi-token prediction enabled. Setup uses 96 GB of total VRAM.

User benchmarks abliteration tools Apostate, Huihui and Heretic on Qwen 2.5 7B on an RTX 5090 32GB. Setup is vLLM 0.19.0 in bf16, evaluated with lm-evaluation-harness. The run reports MMLU, GSM8K, HellaSwag, ARC Challenge, WinoGrande, TruthfulQA MC2, PiQA, LAMBADA ppl, HarmBench ASR and KL divergence. No tokens/sec is reported.

reported speed:

28.0 tokens/s generation · 450.0 tokens/s prompt processing

quant:

Q4_K_M (gguf)

kv:

Q8

Reported by the source; GPU count, offloading and concurrent requests can change this figure. Check the full setup before comparing.

User reports Qwen3.8 27B at 25.6 t/s on an RTX 3090 at 131,072 context. Setup is llama.cpp build b4400 with the Huihui abliterated Q4_K_S GGUF and a Q4 KV cache, the whole 24 GB card in use. The user finds it solid for autocomplete but notes it occasionally hallucinates imports in multi-file refactors.