Running a coding model locally with Ollama

Running a capable coding model on your own GPU takes one install and a handful of commands. The better part is what comes after: the model is not a black box. You can read how much memory it takes, how fast it writes, and how much text it can hold at once, and none of your prompts ever leave the machine.

Local inference: the model moves to your laptop

Local inference means running the model on your own machine instead of calling a hosted API. Ollama is the fastest way there. It downloads open-weight models, serves them behind a local HTTP API at http://localhost:11434, and runs them on your GPU. Open-weight means the trained weights, the numbers the model learned during training, are published for you to download and run, unlike API-only models such as Claude or GPT whose weights never leave the vendor.

A model family is a lineage: who trained the weights. Four are worth knowing by name:

Llama (Meta)
Qwen (Alibaba)
Gemma (Google)
DeepSeek (DeepSeek)

You pick a family and a size, and Ollama pulls a build compressed to fit in your machine’s memory.

Install Ollama and the model

Five commands take you from nothing to a model answering on your machine:

Install Ollama with Homebrew (on Linux or Windows, use the installer at ollama.com/download).
Start the server; it listens at http://localhost:11434.
Pull a coding model.
List what you pulled, with its size on disk.
Run it once with a prompt to confirm it answers locally.

brew install ollama
brew services start ollama      # serves on http://localhost:11434
# model names follow family:size: deepseek-coder-v2:16b is DeepSeek's coding model
# in its 16-billion-parameter build (what a parameter is comes with the spec sheet)
ollama pull deepseek-coder-v2:16b
# ollama list shows what you pulled, and its size on disk
ollama list

ollama list reports it on disk:

NAME                     ID              SIZE      MODIFIED
deepseek-coder-v2:16b    63fb193b3a9b    8.9 GB    2 minutes ago

ollama run writes the code and prints a --verbose footer:

# --verbose appends timing stats after the answer
ollama run deepseek-coder-v2:16b --verbose "Write a one-line hello world in Python."
 
```python
print("Hello, World!")
```
 
total duration:       5.994714959s
load duration:        5.659750709s
prompt eval count:    18 token(s)
prompt eval duration: 70.487ms
prompt eval rate:     255.37 tokens/s
eval count:           28 token(s)
eval duration:        262.95ms
eval rate:            106.48 tokens/s

The footer is two phases, reading then writing:

The model does not see letters. The text gets split into tokens, small chunks, roughly word pieces. The prompt above, plus the wrapper text Ollama adds around it, splits into 18 of them: the prompt eval count.
Before it can write anything, the model reads all 18 tokens to build up what you asked. That phase took 70.487ms, so the prompt eval rate, 18 tokens in 70.487 ms, is 255.37 tokens/s: its reading speed.
Only then does generation start: the 28-token answer, written at the eval rate of 106.48 tokens/s. That number is the model’s throughput.

Reading the spec sheet

ollama show prints the model’s spec sheet:

ollama show deepseek-coder-v2:16b

  Model
    architecture        deepseek2
    parameters          15.7B
    context length      163840
    embedding length    2048
    quantization        Q4_0
 
  Capabilities
    completion
    insert
 
  Parameters
    stop    "User:"
    stop    "Assistant:"
 
  License
    DEEPSEEK LICENSE AGREEMENT
    Version 1.0, 23 October 2023

parameters (15.7B): how many weights the model holds.
- Writing a token means running weights through math: the more weights take part, the slower each token comes out.
- In a dense model, every weight takes part in every token.
- DeepSeek-Coder-V2 is a mixture-of-experts model instead: the weights are grouped into small expert sub-networks, and a router picks the few experts relevant to each token, the way a clinic keeps many specialists on staff but sends you only to the two your case needs.
- Only about 2.4B of its 15.7B weights work on any one token, so it writes like a small model (the 106.48 tokens/s above) while carrying a big model’s knowledge.
- All 15.7B still have to sit in memory.
context length (163840): the window, the most tokens the model can hold at once.
- Prompt plus output share it: everything the model reads and everything it writes must fit inside.
- At ~160K this one is built for long context.
- That window is the biggest lever on memory.
embedding length (2048): the 2048 numbers that describe each token.
- They come from 16 attention heads, parallel readers of the code.
- Each head contributes 128 numbers: 16 × 128 = 2048.
quantization (Q4_0): the precision the weights are stored at, the size-versus-quality dial.
- Each weight is normally a 16-bit number, the precision the model was trained and shipped in.
- Q4 keeps only 4 bits of it. Four bits can land on just 16 possible values, so each weight is rounded to the nearest of them.
- Fewer bits per weight makes the model about 4× smaller, for a little lost precision.

The Capabilities block is what the model can do:

completion: generate text.
insert: fill-in-the-middle, the code-infill trick that marks a coding model.
No tools: function calling, where the model answers with a structured call for your code to execute instead of prose, is not in this model’s repertoire, so Ollama’s tools API will not drive it.
To gain a capability, pick a model that lists it (qwen2.5-coder, for instance, adds tools). Capabilities are set by the model and its template, the wrapper text Ollama puts around your prompt before the model sees it. Listing a capability and delivering it are different things; only a probe settles which one a flag is.

Understanding the embedding length

Each of the 16 heads reads a token against the rest of the code for a different relationship. Take total on the last line here:

def checkout(cart, user):
    total = 0
    for item in cart:
        total += item.price
    if user.is_member:
        total *= 0.9
    return total

Different heads link that total to different earlier tokens at the same time:

one head → total = 0, where it was declared
one head → total += item.price, where each item’s price is added in
one head → total *= 0.9, its most recent value, the member discount
one head → def checkout, marking it the function’s return value

Each head records its read as 128 numbers, a point in 128 dimensions, the same way (x, y) is a point in 2 dimensions and (r, g, b) a color in 3. Stack all 16 heads and you get the token’s full 2048-number description:

token "total"           128 numbers per head
  head  1   [ 0.12  -0.03   0.88   …   0.05 ]
  head  2   [-0.44   0.21   0.01   …  -0.10 ]
  head  3   [ 0.07   0.55  -0.32   …   0.19 ]
    ⋮
  head 16   [ 0.31  -0.09   0.42   …  -0.02 ]
  ──────────────────────────────────────────────
  16 heads × 128 = 2048 numbers

The values above are illustrative. Ollama reports memory and throughput, not the model’s internal activations, so a real per-head dump needs a framework like PyTorch.

Model observability

ollama ps and the /api/ps endpoint tell you how the model is running right now. The install’s ollama run loaded it at the default window, the window Ollama uses when you set nothing, so that is the state on display:

ollama ps

NAME                     ID              SIZE     PROCESSOR    CONTEXT    UNTIL              
deepseek-coder-v2:16b    63fb193b3a9b    13 GB    100% GPU     32768      4 minutes from now

PROCESSOR is the column that matters:

A model runs by holding its weights in memory, and there are two pools for that:
- the GPU’s own VRAM, fast at this kind of math
- ordinary system RAM, which the CPU reaches and is far slower
100% GPU means every layer fits in VRAM, so the GPU does all the work, at the 106.48 tokens/s the install run measured.
When a model is larger than the free VRAM, Ollama loads what fits on the GPU and spills the rest to system RAM; ollama ps then reads a split like 60% GPU / 40% CPU.
That CPU share is the part of the model running on the slow path, and throughput collapses, from tens of tokens per second to single digits.

/api/ps is the same report over HTTP, with exact byte counts:

curl -s localhost:11434/api/ps | jq -r '
  .models[] |
  "\(.name)\n  VRAM:    \(.size_vram/1e9*10|round/10) GB of \(.size/1e9*10|round/10) GB (\(.size_vram/.size*100|round)% on GPU)\n  Context: \(.context_length) tokens\n  Quant:   \(.details.quantization_level), \(.details.parameter_size) params"
'

deepseek-coder-v2:16b
  VRAM:    13.8 GB of 13.8 GB (100% on GPU)
  Context: 32768 tokens
  Quant:   Q4_0, 15.7B params

Even at the default window, the VRAM figure tops the 8.9 GB that ollama list reported on disk back at the install:

Disk holds only the quantized weights.
VRAM holds the weights plus a per-token cache that grows with the loaded window.

The context window is the biggest lever on memory

A larger context window is not free:

To avoid reprocessing the whole context on every new token, the model caches a small piece of data for each token it has already seen: the KV cache.
That cache grows with the window, and you can watch it move.

The 163840 from ollama show is a ceiling, not a fixed size:

You choose the real window per run with the num_ctx option, up to that ceiling, and the model loads at whatever you pick.
It is the same deepseek-coder-v2:16b throughout, only the window changes.

ollama run has no command-line flag for the window, so num_ctx is set another way:

interactive session: run the model, then /set parameter num_ctx 32768
server env: OLLAMA_CONTEXT_LENGTH=32768 ollama serve
API: options.num_ctx on the request

Load it at 32K (K = 1024 tokens), or 32 × 1024 = 32768 tokens. Read the footprint right after:

# num_ctx=32768 loads it with a 32K window; reply discarded, we just want the load
curl -s localhost:11434/api/generate \
  -d '{"model":"deepseek-coder-v2:16b","options":{"num_ctx":32768},"prompt":"hi"}' \
  >/dev/null
curl -s localhost:11434/api/ps \
  | jq -r '.models[] | "\(.context_length) ctx  \(.size_vram/1e9*10|round/10) GB VRAM  \(.size_vram/.size*100|round)% GPU"'

32768 ctx  13.8 GB VRAM  100% GPU

Then the same model at 128K, four times the window:

# same model, num_ctx=131072 reloads it with a 128K window; reply discarded, we just want the load
curl -s localhost:11434/api/generate \
  -d '{"model":"deepseek-coder-v2:16b","options":{"num_ctx":131072},"prompt":"hi"}' \
  >/dev/null
curl -s localhost:11434/api/ps \
  | jq -r '.models[] | "\(.context_length) ctx  \(.size_vram/1e9*10|round/10) GB VRAM  \(.size_vram/.size*100|round)% GPU"'

131072 ctx  28.3 GB VRAM  100% GPU

Same weights, same model:

Four times the window roughly doubles the memory, 13.8 GB → 28.3 GB.
That extra ~14 GB is the KV cache growing, the only thing the window touches.
On this machine it still fit in VRAM at 100% GPU; on a smaller one the 128K run would spill to CPU and crawl.
The window, not the weights, is what decides whether a model fits.

Memory is only half the cost of a big window:

Quality drifts too: long-context models are widely reported to grow less reliable as the context fills, well before the window runs out.
A large context is worth paying for only when you will really use it.

The families, side by side

Naming families is one thing; running them shows the trade-off. One coding build from each family on the same machine, same --verbose prompt:

The models cap their context windows at different ceilings, so memory is measured at 8K, the one window all four support, to keep the figures comparable:

model	params	max ctx	VRAM @ 8K	tokens/s	capabilities
`codellama:7b`	7B dense	16K	6.2 GB	61.7	completion
`qwen2.5-coder:7b`	7.6B dense	32K	4.9 GB	47.9	completion, tools, insert
`codegemma:7b`	9B dense	8K	7.4 GB	40.6	completion
`deepseek-coder-v2:16b`	15.7B MoE (~2.4B active)	160K	10.3 GB	106.5	completion, insert

Bigger is not slower, and size is not the only axis:

DeepSeek is the largest here at 15.7B yet the fastest by far, over double any of the three dense 7-9B models, because its mixture-of-experts design activates only ~2.4B parameters per token.
Among the dense models, more parameters cost a little speed: 7B codellama leads at 61.7, 9B codegemma trails at 40.6, with qwen between them; that whole spread is smaller than the jump to the MoE model.
Memory does not follow speed. All 15.7B of DeepSeek’s weights sit in VRAM even though only ~2.4B fire per token, so at a shared 8K window it is the heaviest (10.3 GB); among the dense models qwen is the lightest (4.9 GB) despite sitting mid-pack on parameters.
The window ceiling is its own axis: DeepSeek stretches to 160K and qwen to 32K, but codegemma caps at 8K and codellama at 16K, a real limit before you feed it a long file.
Capability is separate again: only qwen2.5-coder lists tools; the two code* builds do completion only, and DeepSeek adds insert but not tools.
Size, speed, memory, window, and capability move independently; you pick the corner that fits the job.

One more axis: instruct versus reasoning.

All four models above are instruct models, they answer immediately.
A reasoning model like deepseek-r1 thinks in the open first, spending tokens on a visible chain of thought before the answer.
It is slower, and stronger on step-by-step problems.
Pull one (ollama pull deepseek-r1:8b) and you watch it reason before it replies.

A local model is a drop-in for a cloud API

Ollama serves a second API path, /v1:

It speaks the OpenAI chat-completions shape, the request and response format OpenAI’s cloud API popularized.
Any code or tool that already calls a hosted model can point at Ollama by changing one thing, the base URL:

curl -s localhost:11434/v1/chat/completions \
  -d '{"model":"deepseek-coder-v2:16b","messages":[{"role":"user","content":"Say hello in one short line."}]}' \
  | jq '{object, model, choices: [.choices[0].message], usage}'

{
  "object": "chat.completion",
  "model": "deepseek-coder-v2:16b",
  "choices": [ { "role": "assistant", "content": " Hello!" } ],
  "usage": { "prompt_tokens": 15, "completion_tokens": 3, "total_tokens": 18 }
}

Same request shape, same response shape, from a server on your machine:

The response carries the same usage accounting a cloud API returns.
Swap https://api.openai.com/v1 for http://localhost:11434/v1 and existing code runs against the local model.

Pi, pointed at the local models

Pi, the coding agent from Claude Code vs Pi, ships knowing only hosted providers, Anthropic and Google. One JSON file, ~/.pi/agent/models.json, teaches it the local ones. It adds Ollama as a custom provider: a name, the /v1 base URL above, and one entry per model, ids exactly as ollama list prints them (deepseek-coder-v2:16b, qwen2.5-coder:7b, and qwen3:8b). The apiKey is a placeholder; Ollama ignores it, but Pi wants a value:

{
  "providers": {
    "ollama": {
      "baseUrl": "http://localhost:11434/v1",
      "api": "openai-completions",
      "apiKey": "ollama",
      "models": [
        { "id": "deepseek-coder-v2:16b", "contextWindow": 163840, "maxTokens": 8192 },
        { "id": "qwen2.5-coder:7b", "contextWindow": 32768, "maxTokens": 8192 },
        { "id": "qwen3:8b", "contextWindow": 40960, "maxTokens": 8192 }
      ]
    }
  }
}

contextWindow and maxTokens matter: Pi does not ask Ollama for a model’s limits, it believes this metadata (and defaults to 128K when the entry omits it). Declare each window from the model’s ollama show spec sheet, or Pi will happily overfeed a 32K model. pi --list-models ollama confirms the hookup:

provider  model                  context  max-out  thinking  images
ollama    deepseek-coder-v2:16b  163.8K   8.2K     no        no
ollama    qwen2.5-coder:7b       32.8K    8.2K     no        no
ollama    qwen3:8b               41.0K    8.2K     no        no

A capability flag is a claim, not a proof

Local inference without an external API only pays off if the model can do the job you hired it for, and for agent work the job is the tool call.

Three of the installed models compete for it, and their ollama show Capabilities blocks disagree:

model	capabilities	`tools`?
`deepseek-coder-v2:16b`	completion, insert	no
`qwen2.5-coder:7b`	completion, tools, insert	yes
`qwen3:8b`	completion, tools, thinking	yes

Whether a model with the flag can actually make a tool call is a different question, and one probe answers it: offer the model a single tool, write, and ask for a file. The same request goes to all three models; only the model field changes:

curl -s localhost:11434/api/chat -d '{
  "model": "deepseek-coder-v2:16b",
  "stream": false,
  "messages": [{"role":"user","content":"Create a file named hello.txt containing exactly the word: hello"}],
  "tools": [{"type":"function","function":{"name":"write","description":"Write a file","parameters":{"type":"object","properties":{"path":{"type":"string"},"content":{"type":"string"}},"required":["path","content"]}}}]
}'

The request has two moving parts:

messages carries the ask, one user turn: create a file named hello.txt containing exactly the word hello.

"messages": [
  {
    "role": "user",
    "content": "Create a file named hello.txt containing exactly the word: hello"
  }
]

tools hands the model the one function it may call to get that done: write, taking an object with path and content, both strings, both required. The parameters block is a JSON Schema describing the function’s arguments; the model reads it to shape a valid call.

"tools": [
  {
    "type": "function",
    "function": {
      "name": "write",
      "description": "Write a file",
      "parameters": {
        "type": "object",
        "properties": {
          "path": { "type": "string" },
          "content": { "type": "string" }
        },
        "required": ["path", "content"]
      }
    }
  }
]

A model that earns its flag reads the ask, matches it to the tool, and answers with a structured tool_calls field carrying write and {"path": "hello.txt", "content": "hello"}.

A subtle note: a file on disk is not expected, because /api/chat is text in, text out; executing the call is the caller’s job. The probe grades exactly one thing, whether tool_calls comes back populated. Everything else is just text.

Three models, three verdicts

deepseek-coder-v2: no flag, and the absence is enforced

deepseek-coder-v2 does not list tools, so the expectation is that it never even gets to answer, and the probe confirms it at the API:

{"error":"registry.ollama.ai/library/deepseek-coder-v2:16b does not support tools"}

Ollama rejects the request before the model ever sees the prompt.

qwen2.5-coder: a false positive

qwen2.5-coder:7b has the flag, so the same request goes through, and fails in a more interesting way:

curl -s localhost:11434/api/chat -d '{
  "model": "qwen2.5-coder:7b",
  "stream": false,
  "messages": [{"role":"user","content":"Create a file named hello.txt containing exactly the word: hello"}],
  "tools": [{"type":"function","function":{"name":"write","description":"Write a file","parameters":{"type":"object","properties":{"path":{"type":"string"},"content":{"type":"string"}},"required":["path","content"]}}}]
}' | jq -r '"tool_calls: \(.message.tool_calls)\ncontent:\n\(.message.content)"'

tool_calls: null
content:
{
  "name": "write",
  "arguments": {
    "path": "hello.txt",
    "content": "hello"
  }
}

The reply, read against the expectation:

tool_calls: null: the one field the probe grades is empty; Ollama parsed no call.
content carries the call anyway: a well-formed write payload with exactly the right arguments, delivered as prose.

The model understood the task, picked the right tool, and built valid arguments, yet failed on the only thing that counts: a populated tool_calls field. Ollama’s parser watches for exactly the shape the model’s own chat template demands:

<tool_call>
{"name": "write", "arguments": {"path": "hello.txt", "content": "hello"}}
</tool_call>

But the model wrote the JSON and skipped the tags. No tags, no parse: tool_calls stays null, and the call lands in content as prose. Is that the model’s fault? Let’s test the same request with qwen3:8b:

curl -s localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "stream": false,
  "messages": [{"role":"user","content":"Create a file named hello.txt containing exactly the word: hello"}],
  "tools": [{"type":"function","function":{"name":"write","description":"Write a file","parameters":{"type":"object","properties":{"path":{"type":"string"},"content":{"type":"string"}},"required":["path","content"]}}}]
}' | jq -r '"tool_calls: \(.message.tool_calls)\ncontent:\n\(.message.content)"'

tool_calls: [{"id":"call_6epdfuul","function":{"index":0,"name":"write","arguments":{"path":"hello.txt","content":"hello"}}}]
content:

The request is well formed and Ollama’s parsing works; the pipeline produces a structured call whenever the model plays its part. That is the intriguing part: the same prompt, delegated to two models of the same family, both flying the tools flag, and only one actually delivers. Same request in, and only the weights differ.

The qwen2.5-coder flag advertises the standard; the model delivers a dialect. For driving a stock agent, that is a false positive.

qwen3: the flag, delivered

qwen3:8b (ollama pull qwen3:8b), the same size class as qwen2.5-coder:7b and one generation newer, answers the same probe with a structured call:

{
  "content": "",
  "tool_calls": [{"id": "call_9lsm4tio", "function": {"index": 0, "name": "write", "arguments": {"content": "hello", "path": "hello.txt"}}}]
}

The spec sheet says what the chat template supports; only a probe says what the model does. Before building anything on a capability flag, make the model prove it once.

Tuning the footprint

When a big window pushes the KV cache past your VRAM, you have server-level knobs, distinct from the model’s own weight quantization. Start Ollama with:

OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

OLLAMA_KV_CACHE_TYPE=q8_0 stores the KV cache at 8 bits instead of 16:

That roughly halves its memory for a tiny quality cost, so a window that spilled to CPU may fit on the GPU again.
This quantizes the KV cache (a runtime lever), not the weights (Q4_0, baked into the model).
One caveat: these are set on the server process, so they do not apply when Ollama runs under brew services. Stop that and run ollama serve yourself to use them.

When running it yourself wins

Local inference earns its place for concrete reasons, in roughly this order:

Cost at volume. A hosted model bills per token; a local one is a fixed hardware-and-electricity cost. Past a certain call volume, running your own is simply cheaper.
Latency and control. No network hop, no rate limits, no per-call quota. The model answers as fast as your GPU allows, as often as you like.
Free experimentation. Iterate on internal tooling without a metered API in the loop, no bill for a bad prompt, no quota to nurse.
Full telemetry. The --verbose footer, ollama ps, and /api/ps report tokens, memory, and the loaded window straight from the runtime, not summarized behind a vendor dashboard.
Data locality. A prompt to a hosted model leaves your machine; a local one never does. The blast radius of a leak shrinks to this one machine, which matters most for sensitive or regulated data.

That is the whole loop: one brew install, one ollama pull, and a coding model is writing on your own GPU. From there --verbose, ollama ps, and /api/ps report its speed, memory, and window at every step, and every prompt stays on the machine. Nothing about it is a black box.

Cloud still wins when you need frontier quality no model on your GPU can match. The honest split:

reach for local on cost, control, and experimentation
reach for a hosted frontier model when the task demands its ceiling

Running a coding model locally with Ollama

Local inference: the model moves to your laptop

Install Ollama and the model

Reading the spec sheet

Understanding the embedding length

Model observability

The context window is the biggest lever on memory

The families, side by side

A local model is a drop-in for a cloud API

Pi, pointed at the local models

A capability flag is a claim, not a proof

Three models, three verdicts

deepseek-coder-v2: no flag, and the absence is enforced

qwen2.5-coder: a false positive

qwen3: the flag, delivered

Tuning the footprint

When running it yourself wins

Beyond the spec sheet: which local model you ship

Running a coding model locally with Ollama

Claude Code ships the batteries, Pi hands you the wires

How AI and Agentic Development Are Reshaping Software Engineering

Fixup commits: the clean way to fix past mistakes

Putting Claude Code minions to work

From Docker build to AWS in 5 commands

The right way to authenticate with AWS

How I built a blog that writes itself

One command to launch my entire dev environment