NVIDIA A100 40GB

NVIDIA · 40GB · 2 reports

Engines people use on it: llama.cpp 1 (reports)

See reported performance Estimate what fits

How to compare these reports →Filter reports by setup →Best local LLMs for 48 GB of VRAM →

Run models on your NVIDIA A100 40GB? Add your numbers.

Your coding agent can run a speed test on your machine and send it here with a free account. Or paste a result you've already posted.

Benchmark with your agent Paste a result

Reported performance on the NVIDIA A100 40GB

These reports mix GPU counts and memory setups, so a model listed here may have needed extra cards or CPU offloading. The calculator estimates which models fit in 40 GB of VRAM.

This page is thin (2 of 3 reports needed for indexing). Help fill it in.

Date ↓Date ↑Token speed ↓Token speed ↑Prefill ↓Prefill ↑Context ↓Context ↑Ranked

Qwen2.5 7B

NVIDIA A100 40GB

User reports a crystallization experiment on Qwen2.5-7B-Instruct using an activation-steering engine called AkbasCore 3.0 / SEASC on 1× NVIDIA A100-SXM4-40GB. Setup uses LoRA gradient updates to transfer part of the induced layerwise representation geometry into the model weights, then removes the runtime steering motor and adapter. Training ran 1.549 min (~93 seconds) and consumed 2.6329 Wh of measured GPU-board energy at a time-weighted mean board power of 102.33 W. The locked final test showed +15.7881% progress with the motor off, best state at update 40, and automatic overshoot detection at update 70. An unresolved merge diagnostic reports 1.762% relative hidden-state drift.

DeepSeek V4 Flash 284B (13B active)

NVIDIA A100 40GB · llama.cpp

generation:: 16.1 tokens/s
quant:: Q8_K_XL (GGUF)

Reported by the source; GPU count, offloading and concurrent requests can change these figures. Prompt-processing speed is shown only when the source reports it; generation speed cannot tell us time to first token. Check the full setup before comparing.

User reports all experts on CPU with only 15.8 GB of 40 GB VRAM used.

Get a weekly email of new NVIDIA A100 40GB reports and newly released models that fit it.

Email me new reports

Reported configurations

Individual observations from this page, not expected speeds or a ranking. Select a model to open its report, with the source and offloading conditions. Reported context may be a limit rather than actual input length.

Model / hardware	Quant / engine	Reported context	Generation
DeepSeek V4 Flash 284B (13B active) NVIDIA A100 40GB	Q8_K_XL llama.cpp	Not reported	16.1 tokens/s