llamaperf

NVIDIA Jetson AGX Thor 128GB

NVIDIA · 128GB unified memory · 8 reports

Is this your card?

On the NVIDIA Jetson AGX Thor 128GB, Qwen3.8 125B · 6B active at 4-bit, the model most run on it, generates a median of 36.7 tokens per second from one plain run (community reports, as of 22 Nov 2026).

As of 22 Nov 2026, the models most run on the NVIDIA Jetson AGX Thor 128GB, each one size at one quant level, with the median generation speed of its plain runs (one device, one request, no speculative decoding):

Bars are medians on one scale, 0 to 36.7 tokens/s. Press ▶ to watch text arrive at a row's speed.

Engines people use on it: vLLM 2 (reports)

Run models on your NVIDIA Jetson AGX Thor 128GB? Add your numbers.

Your coding agent can run a speed test on your machine and send it here with a free account. Or paste a result you've already posted.

Reported performance on the NVIDIA Jetson AGX Thor 128GB

These reports mix GPU counts and memory setups, so a model listed here may have needed extra cards or CPU offloading. The calculator estimates which models fit in 128 GB of unified memory. Prompt processing is how fast the input is read, not how long you wait for the first word. How to measure time to first token.

Speeds are as each source reported them. GPU count, offloading and concurrent requests change them, and prompt-processing speed is input throughput, not time to first token. Check the full setup before comparing two reports.

Tone: mixedNVIDIA hardware
generation:
12.0 tokens/s
quant:
Q4

User reports Qwen3.8 27B Q4 at around 12 t/s on a Jetson Thor, below the ~20 t/s they expected from its memory bandwidth. No engine, context length or KV cache is stated; the figure is without MTP or DFlash. On an RTX 5070 Ti the same model reaches about 66 t/s, and DFlash on the Thor gives only a roughly 2x speedup, slightly better than MTP.

Nov 18, 2026

Qwen3.8 27B

NVIDIA Jetson AGX Thor 128GB · TensorRT Edge-LLM · 2,048 ctx

NVIDIA hardware
generation:
39.6 tokens/s
prompt processing (prefill):
926.0 tokens/s
quant:
INT4

User reports Qwen3.8-27B at 39.6 tok/s decode on a Jetson AGX Thor, with 926 tok/s prompt processing and 23.6 tok/s end-to-end. Setup is TensorRT Edge-LLM 0.10.1 with INT4 weights, a FlashHead LM head and MTP speculative decoding, at batch 1 with 2,048-token prompts and 128 tokens out, averaged over 20 requests. Peak memory was 24.8 GiB and the weights download is about 20 GiB. Accuracy against the FP16 model: GSM8K 96.4 vs 96.1, IFEval 83.5 vs 81.9, MMLU-Pro 79.1 vs 79.5. The user notes Jetson AI Lab reports about 35 tok/s decode with llama.cpp Q4_K_M plus MTP, without stating prompt length or accuracy.

Oct 26, 2026
Tone: mixedNVIDIA hardware
generation:
0.7 tokens/s
quant:
FP8 (FP8)

User reports MiMo 2.6 Pro at 0.646 t/s on a Jetson AGX Thor 128GB. Setup is FP8 weights of about 566GB read from disk mid-inference, with experts streamed through 128GB of unified memory. The user notes the model technically loaded and ran, and that every decoder layer, routed expert and SSD read was traced and replayable step by step.

Oct 17, 2026
NVIDIA hardware
generation:
27.7-34.4 tokens/s
quant:
NVFP4 (NVFP4)

User reports a repository of launcher scripts for running local LLMs on the 128 GB Jetson AGX Thor Developer Kit, with vLLM or NVIDIA's Thor-optimized llama.cpp container. The default profile is Qwen3.8-27B NVFP4 with DFlash2 speculative decoding, a 30.3 GB download, listed at a published 27.69-34.44 tok/s on Thor. Other profiles include Qwen3.6-35B-A3B NVFP4 + DFlash at 116.5 average and 139.1 peak tok/s, Nemotron Nano 9B v2 NVFP4 at 30 tok/s, Nemotron 3.5 Lightning NVFP4 + DSpark at 123.01-138.02 tok/s, Gemma 4 26B-A4B NVFP4 + MTP at 50 tok/s c1 and 180 tok/s aggregate c8, and Muse Glimmer 30B K-Quant + DFlash at up to 36 tok/s. The GLM-4.7-Flash and GLM-4.5-Air profiles are experimental with no published Thor speeds. The speeds come from different workloads and are not a controlled comparison.

Oct 9, 2026
NVIDIA hardware
generation:
36.7 tokens/s
prompt processing (prefill):
899.0 tokens/s
quant:
NVFP4 (NVFP4)
kv:
int8

User reports Qwen3.8-Flash-Next NVFP4 at 36.7 tok/s one-user decode and 899 tok/s prefill at 32k on a Jetson AGX Thor 128GB. Setup is TensorFold 0.6.0 with int8 KV cache, full 262,144-token window, --parallel 4. Prefill measured at 926 / 899 / 798 tok/s for 8k / 32k / 128k tokens; combined decode at 1 / 2 / 3 / 4 users is 34.8 / 53.7 / 70.3 / 76.7 tok/s; first answer to a 60k-token prompt takes 69.7 s. User compares against vLLM's FP8 build on the same box (38.4 tok/s decode, 2,101 tok/s prefill at 32k) and describes a local Thor prompt path with gated runs r09 and r11 reaching 1,549 tok/s prefill at 32k. 128 GB and 273 GB/s are published Thor specs, not re-measured.

Oct 4, 2026
NVIDIA hardware
generation:
25.0 tokens/s
prompt processing (prefill):
3557.0 tokens/s
quant:
NVFP4 (NVFP4)
kv:
8-bit

User benchmarks Qwen3.8-27B at 24.96 t/s generation and 3557 t/s prompt processing on a Jetson AGX Thor 128GB, using the Mjolnir vLLM image with the FA4 GEMV decode kernel. Setup is vLLM 0.30.0 with NVFP4 weights and an 8-bit KV cache at 8K context, single request. The GEMV kernel is default-on and dispatches only on M=1, head_dim=256, GQA shapes. The GEMV leg wins at c=1 but regresses at c=4 (57.74 t/s at 8K context, −15.4% versus stock vLLM), which the user attributes to an open investigation. Stock vLLM reaches 24.43 t/s and 2468 t/s prompt at the same setting.

Oct 4, 2026
Tone: positiveNVIDIA hardware
generation:
15.3-18.5 tokens/s
prompt processing (prefill):
90-170 tokens/s
quant:
Q4_K_XL (GGUF)
visioncodingagenticlong-context

User reports Qwen3.8-Flash-Next running on a Jetson AGX Thor 128GB with vision support, decoding at 15.3-18.5 tok/s on free-form text. Setup is llama.cpp (qwen4exp branch, commit d4a943f plus cherry-pick 24ea62d and canreuse-v2.patch) with UD-Q4_K_XL GGUF, 65536 context, ngram-mod speculative decoding, and the 51B n-gram table offloaded to CPU/NVMe via -ot per_layer_token_embd=CPU -lm mmap, leaving about 80GB resident. Prefill is 90-170 tok/s depending on caching; ngram-mod speculation reached 82.9 tok/s on verbatim code reproduction (91% draft acceptance, mean accepted span 59 tokens) but only fires on long untouched spans. The 120W power mode costs about 10% versus MAXN, and images cost a one-off 1-2s to encode without affecting decode speed.

Sep 27, 2026
NVIDIA hardware
generation:
139.1 tokens/s
quant:
NVFP4 (NVFP4)

User reports Qwen3.6-35B-A3B-NVFP4 at 139.1 tok/s on a single NVIDIA Jetson AGX Thor with 117 GB unified memory. Setup is vLLM built from source for sm_110a with DFlash speculative decoding (12 tokens), marlin MoE backend, flash_attn attention backend, 65536 context length, and 0.78 GPU memory utilization. The post also benchmarks Qwen3.5-4B-NVFP4 at 155.8 tok/s, Qwen3.6-27B-NVFP4 at 50.1 tok/s, and Qwen3.5-122B-A10B-NVFP4 at 52.6 tok/s, all at concurrency 1. The 122B requires cutlass MoE and TRITON_ATTN due to a Marlin crash at 256 experts, and achieves 27-42 tok/s with DFlash versus 10.9 tok/s autoregressive.

Sep 27, 2026

Get a weekly email of new NVIDIA Jetson AGX Thor 128GB reports and newly released models that fit it.

Email me new reports

Reported configurations

Individual observations from this page, not expected speeds or a ranking. Select a model to open its report, with the source and offloading conditions. Reported context may be a limit rather than actual input length.

Model / hardwareQuant / engineReported contextGeneration
Qwen3.8 27B
NVIDIA Jetson AGX Thor 128GB
Q4
Engine not reported
Not reported12.0 tokens/s
Qwen3.8 27B
NVIDIA Jetson AGX Thor 128GB
INT4
TensorRT Edge-LLM
2,04839.6 tokens/s
Mimo 2.6 1020B (42B active) Pro
NVIDIA Jetson AGX Thor 128GB
FP8
Engine not reported
Not reported0.7 tokens/s
Qwen3.8 125B (6B active) Flash-Next
NVIDIA Jetson AGX Thor 128GB
NVFP4
TensorFold
262,14436.7 tokens/s
Qwen3.8 27B
NVIDIA Jetson AGX Thor 128GB
NVFP4
vLLM
8,19225.0 tokens/s
Qwen3.6 35B (3B active)
NVIDIA Jetson AGX Thor 128GB
NVFP4
vLLM
65,536139.1 tokens/s

Frequently asked

How fast is the NVIDIA Jetson AGX Thor 128GB for local LLMs?

On the NVIDIA Jetson AGX Thor 128GB, Qwen3.8 125B · 6B active at 4-bit, the model most run on it, generates a median of 36.7 tokens per second from one plain run (community reports, as of 22 Nov 2026). Also reported: Qwen3.8 27B at 4-bit (12.0 t/s, one plain run).