llamaperf
← All articles

Choosing a model

Small local LLM: how to pick one for modest hardware

By Dev Ramanathan · · 7 min read

Quick answer

A small local LLM suits short, focused jobs like summaries, rewrites and simple code help, and it fits modest hardware. Read the model card for what the model really loads and what its licence allows, then try five of your own tasks on it before you trust it with more.

What a small local LLM is good for

A small local LLM is a model with a few billion parameters or fewer. Picture a pocket dictionary next to a full encyclopedia. It answers the everyday questions quickly, it takes little room, and it runs on the computer you already own. Our local LLM hub and the guide to running an LLM locally cover the wider setup. This article covers one decision: how to choose among the small ones.

Match the job to the size

Small models are good at short, focused work. Qwen's own card for its 2B model says as much:

"In light of its parameter scale, the intended use cases are prototyping, task-specific fine-tuning, and other research or development purposes." (Qwen3.5-2B model card)

That fits the whole category. Summarizing an email thread, tightening a paragraph, pulling dates out of a message and explaining a ten-line function all sit comfortably in small-model territory. Long chains of reasoning, large codebases and obscure facts are where a bigger model tends to pull ahead, and you'll notice quickly when you try.

Where your hardware comes in

A model in this class leaves room on a laptop, and on a graphics card with 8 GB of VRAM. How much room is left depends on the quantization and the context length you ask for. The VRAM calculator checks your exact card or Mac against a specific model, and what Q4_K_M means explains the letters in the file names if they look like code.

Read the model card before you download

A model card is the README on a model's Hugging Face page. Three of its lines will save you an afternoon of downloading the wrong thing.

Effective parameters and what loads

Gemma 4's smallest model is called E2B, and the "2" trips people up. Google's card lists 2.3B effective parameters and 5.1B with embeddings. It explains the difference like this:

"These embedding tables are large but are only used for quick lookups, which is why the effective parameter count is much smaller than the total." (Gemma 4 model card)

So the name describes how much work the model does for each token, and the files still carry the embedding tables. When you compare a model labelled 2B with one labelled 3B, look at the total on the card. Then put the model into the calculator and let it do the memory sum, since the file you download is a different size again once it's quantized.

Licence and context length

Licences differ even inside one family. The Gemma 4 E2B card and the Qwen3.5-2B card both say Apache 2.0. The Qwen2.5-Coder-3B-Instruct card points to the Qwen Research License, which limits use to research or evaluation and tells commercial users to request a licence from Alibaba Cloud. If you plan to use a model at work, read this line before you build anything on it.

Each card also lists a context length: 128K tokens for Gemma 4 E2B, 262,144 tokens natively for Qwen3.5-2B and 32,768 for Qwen2.5-Coder-3B. A longer window costs more memory as it fills up, which context length and KV cache walks through. You don't have to run the maximum. Set the length your tasks need.

Three small models side by side

Here are two recent small models and an older code model, with what each one's own card says. Nothing in the table is a speed or memory figure. Those depend on your machine, so the calculator and the community reports are the place for them.

What the cards say

ModelParameters on the cardContext on the cardLicence on the cardInput and focus
Gemma 4 E2B2.3B effective, 5.1B with embeddings128K tokensApache 2.0Text, image and audio
Qwen3.5-2B2B262,144 tokens nativelyApache 2.0Text and image
Qwen2.5-Coder-3B-Instruct3.09B32,768 tokensQwen ResearchCode, text only

The sources are the Gemma 4 E2B card, the Qwen3.5-2B card and the Qwen2.5-Coder-3B-Instruct card.

How to read the table

The coder model is the oldest and has the shortest window and the tightest licence, and it is built for one job. If your work is code, that focus may matter more to you than the window. The other two take images, and Gemma 4 E2B takes audio as well. Whether your runtime passes an image or a sound to the model is a separate question, so check your tool's documentation before you plan around it.

Try it on your own work first

A review of a small model describes one person's tasks on one person's machine. Your own tasks are the test that counts, and running them is quick.

Run five of your own tasks

Pick five things you did this week. An email thread to summarize, a paragraph to tighten, a function to explain, a table to reformat and a question you needed answered. Paste the same prompts into two small models. Ollama's Gemma 4 page lists the e2b tag, so ollama run gemma4:e2b gets you started, and every other model has a library page with its own tags. Judge the answers first and the speed second. If a coding task is on your list, evaluating a local coding model has a method for it.

When to move up a size

Say two of your five answers come back wrong in ways you'd have to repair by hand. Step up one size and run the same five again. If the next size up doesn't fit your memory, running a local LLM on CPU only covers what to expect from system RAM. For a wider choice, choosing a local LLM for the hardware you own starts from your memory and works toward models.

Frequently asked questions

What is the best small local LLM?

The best one is whichever gets your own tasks right on your machine. Gemma 4 E2B and Qwen3.5-2B are fair places to start because both are Apache 2.0 and both are recent. For a ranking built from what people measure on their own hardware, see the best local LLM page on this site.

Can a small local LLM run without a graphics card?

Yes. Small quantized models run from system memory on the CPU, and the answers arrive more slowly than they would on a GPU. Our article on running a local LLM on CPU only explains what sets the speed.

Is a 2B model a 2 GB download?

No. The file size depends on the quantization, and the parameter count on the card may include more than the headline number. Gemma 4 E2B lists 2.3B effective parameters and 5.1B with embeddings. Put the model in the calculator for the memory estimate.

Can I use Qwen2.5-Coder-3B in a business?

Read its licence first. The Qwen2.5-Coder-3B-Instruct card is under the Qwen Research License, which allows non-commercial use and asks commercial users to request a licence from Alibaba Cloud. Other sizes in the family may carry different terms, so check the card of the exact model you download.

Can small models read images or audio?

Some can. The Gemma 4 E2B card lists text, image and audio input. The Qwen3.5-2B card describes a model with a vision encoder. Qwen2.5-Coder-3B takes text only. Your runtime also has to pass the image or sound along, which you can confirm in its documentation.

How do I know a small model is too small for my work?

Run five real tasks. When you'd have to rewrite the answers to two of them by hand, the model is too small for that job. Try the next size up before you change your tools.