NVIDIA RTX 3070 reports

generation:

97.0 tokens/s

quant:

q4

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports 97 t/s generation. Source evidence (verbatim): Source: https://dev.to/fr4ncis/the-fastest-llama-uncovering-the-speed-of-llms-5ap8 # Original RTX3070 Phi3 mini4k instruct Q4 LMStudio measurement Francesco Mattia original experiments, September1,2024, edited September10. Selected single-prompt result: NVIDIA RTX3070 8GB, LMStudio, Phi3 mini4k instruct q4, approximately 97 tokens/s generation and 0.01s TTFT. Prompt: Who is the president of the US? Temperature0. Exact quantization subtype, tested repository/revision, runtime version, token counts, context depth, resident memory, speculation and concurrency unknown. Separate local PC CPU-only setup lists Ryzen5500 six cores and64GB RAM; whether those exact CPU/RAM specs apply to the selected GPU run is not established. The article also mixes Ollama benchmark-script and LMStudio tests; do not combine the earlier Phi3 input/output table with this specific q4 result. HF artifact link unavailable; do not guess. Original source: https://dev.to/fr4ncis/the-fastest-llama-uncovering-the-speed-of-llms-5ap8 . Selected structured generation measurement: approximately 97 tokens/s. Prompt processing speed unknown; do not derive it from TTFT. CPU/RAM belong to a separate CPU-only setup and must stay unknown for this selected GPU run.

generation:

5.0 tokens/s

quant:

IQ4_NL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Xyntetik-Kvist-14B at ~5 tok/s on an RTX 3070 with 50 of 52 layers offloaded. Setup is Xyntetik Runner with an IQ4_NL mix (7.6 GB) that does not fit an 8 GB card whole. The model is a width-pruned and distilled version of Muse-Glimmer-30B, scoring 57 of 60 held-out tool tasks against the parent's 60, with KLD 0.762 and 84.0% top-1 agreement. It also runs at Q8_0 (15.4 GB) or Q5_0 mix (10.3 GB) on a 24 GB card.

generation:

30.0 tokens/s

quant:

Q5_K_S (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports Qwen3.6-35B-A3B at about 30 t/s with Q5_K_S at 128k context on an RTX 3070 8GB with 64GB DDR4. Setup is llama.cpp. Q4_K_XL reached 32 t/s, and IQ4_XS gave 25-30 t/s at 32k context. The user found that larger quants (Q4_K_XL, Q5_K_S) gave better speed than the smaller Q4 (IQ4_XS).