llamaperf
← All articles

Measurement

How to measure time to first token on a local LLM

By llamaperf · · 6 min read

Quick answer

Send a streaming request, start a clock as it goes out, and stop the clock when the first chunk with visible text arrives. Write the prompt length down beside the result. Note whether the model was already loaded and whether the prompt was cached, because either can move the number by seconds.

What the wait is made of

Time to first token, or TTFT, is how long you stare at an empty reply before any text appears. In a chat it's often the thing that makes a setup feel slow, even when the words arrive quickly once they start.

What happens before the first word

If the model isn't in memory yet, the runtime first loads it from disk. Then it reads your whole prompt, a step called prompt processing or prefill, and only after that does it generate the first token. Ollama reports these steps separately in its API response: load_duration is "Time spent loading the model in nanoseconds" and prompt_eval_duration is "Time spent evaluating uncached prompt tokens in nanoseconds".

Why long prompts wait longer

Prompt processing has to get through every token you send, chat history included. A 20,000-token document takes far longer to read than a one-line question on the same setup, so a TTFT figure without the prompt length beside it can't be compared with anything. Long inputs need more memory as well, which our post on context length and the KV cache explains.

Timing it with Ollama

A small script that sends a streaming request and notes when the first text arrives is the simplest method that holds up. Ollama streams by default:

When true, returns a stream of partial responses From the stream parameter in the Ollama generate API reference

A small streaming script

It needs only Python's standard library. Put your prompt in prompt.txt and swap in a model you have installed.

import json, time, urllib.request

body = json.dumps({
    "model": "YOUR_INSTALLED_MODEL",
    "prompt": open("prompt.txt").read(),
    "stream": True,
}).encode()
req = urllib.request.Request(
    "http://localhost:11434/api/generate", data=body,
    headers={"Content-Type": "application/json"})

start = time.perf_counter()
first = None
with urllib.request.urlopen(req) as resp:
    for line in resp:
        chunk = json.loads(line)
        if first is None and (chunk.get("response") or chunk.get("thinking")):
            first = time.perf_counter() - start
        if chunk.get("done"):
            final = chunk

print(f"time to first token: {first:.2f} s")
print("load s:", final["load_duration"] / 1e9)
print("prompt tokens:", final["prompt_eval_count"])
print("cached prompt tokens:", final.get("prompt_eval_cached_count"))
print("prompt s:", final["prompt_eval_duration"] / 1e9)

What counts as the first token

Reasoning models stream their thinking before the answer, and the script above counts the first piece of thinking text as the first token. If what you care about is when the answer itself starts, check only response. Record which rule you use and keep it consistent.

Timing it with llama.cpp's server

The timings object

llama.cpp's server hands you most of this breakdown with no script at all. Every response includes a timings object, and the server README documents prompt_n (prompt tokens processed), cache_n (prompt tokens reused from the cache) and prompt_ms, the time spent on the prompt. Add the time for one generated token to prompt_ms and you have a close estimate of TTFT as the server sees it. For a wall-clock figure the streaming approach above works here too, and since a client-side stopwatch also counts network and client time, keep the two kinds of figure apart when you compare.

Prompt caching

The server reuses work from the previous request when the start of the prompt matches:

Re-use KV cache from a previous request if possible. This way the common prefix does not have to be re-processed, only the suffix that differs between the requests. From the cache_prompt option in the llama.cpp server README

Caching helps in daily use, but it changes benchmark work: send the same prompt twice and the second TTFT can fall to a fraction of the first because almost nothing got processed. Check cache_n, or Ollama's cached prompt count, before you believe a fast result. It's also why a chat tool that rewrites the start of its system prompt on every turn feels slow. The prefix never matches, so the whole prompt gets read again.

A test you can repeat

Separate cold starts from warm runs

Measure cold, warm and cached cases separately so the result identifies which work was timed. Cold means the model wasn't loaded, and load_duration will be large. Warm means the model is loaded and the prompt is new. Cached means you repeated a prompt that had already been processed. Compare hardware or settings with warm runs on fresh prompts, and change a word at the start of the prompt between runs so the cache can't help.

What to record

FieldWhy
Prompt tokensTTFT grows with prompt length
Cached prompt tokensA cache hit hides prompt processing
Load timeSeparates a cold start from a warm run
First-token ruleThinking text or answer text
Timing sourceServer timings or client stopwatch
RunsReport the median and keep every value

Compare a short prompt with a long one from your work, then include a typical input to check how the wait grows with length. The performance guide explains how TTFT relates to generation speed, the benchmark-reading guide covers the rest of a clean report. Our report template has fields for both numbers. When you submit results, give the prompt length and say whether the run was warm, since a TTFT figure without those two can't be lined up against anyone else's.

Frequently asked questions

How do I measure time to first token in Ollama?

Send a streaming request to the generate endpoint and time it from the moment it goes out to the first chunk carrying response or thinking text. The final chunk has load_duration and prompt_eval_count, so record those too and you'll know what the wait included.

Why is the first response slow and later ones fast?

Usually the first request had to load the model from disk, and the later ones reused cached prompt work. load_duration and the cached prompt token count will tell you which it was. For a fair comparison, use a warm run with a fresh prompt.

Is time to first token the same as prompt processing time?

On a warm model, nearly. TTFT is prompt processing plus the first generated token, plus loading when the model wasn't in memory, and a client-side stopwatch adds a little network and client time on top.

How do I make prompt processing faster?

Send fewer tokens, and keep the start of your prompt stable so the cache can reuse it. Having the whole model on the GPU helps as well, since prompt processing on the CPU is far slower.

Why does my coding agent reprocess the whole prompt every turn?

The cache only reuses a prefix that matches exactly. Most likely the tool changes something near the start of the prompt between turns, such as a timestamp or a mode switch in the system prompt, and everything after that point has to be read again.

What's an acceptable time to first token?

That's up to you and the job. For chat, most people want text within a couple of seconds, while a long document can reasonably take longer. Time your own prompts and decide what you'll put up with.