May 2026
Selecting a local model is not a fire-and-forget process.
Over a weekend of building an AI agent stack backed by Ollama and a consumer GPU, I hit seven problems that no single documentation page covers — because they only emerge when you combine real hardware, real frameworks, and real usage patterns. Each one silently degraded performance or broke the stack entirely, and each one had a simple fix once you knew where to look.
If you're running local inference for serious work — not just demos, but daily operational use — these are the pitfalls worth knowing about before they find you.
The model documentation says 128K tokens. The framework documentation says it supports long conversations. Both are true. Neither matters.
Ollama determines the default context window based on your GPU's VRAM. On a 16GB card, the default is 4,096 tokens — not the model's native capability. Every request runs at 4K regardless of what the model supports.
There is no warning. No log message. The model simply stops mid-response when it hits the limit, and the framework retries, and retries, and eventually gives up.
The fix: set `num_ctx` explicitly in a Modelfile for every model you run. Verify it with `/api/ps` at runtime. Never trust the default.
Having a GPU installed, drivers loaded, and CUDA/DirectML available does not mean your inference engine is using it.
A version-specific device detection bug in Ollama (v0.32.1 on Windows with DirectML) meant the RX 6900 XT was physically present but never loaded. Every model ran on CPU. Token generation was 5.7 tok/s instead of 40+. No error message. No warning in the logs. Just silently slow. Easy to miss on long running agentic based tasks that you expect to run overnight but a bit of a problem when time is compressed.
The fix was one version update. But the symptom — slow inference with no obvious cause — is the kind of thing that sends people back to cloud APIs thinking local isn't viable.
The check: hit `/api/ps` after loading a model. If `size_vram` shows 0, your GPU isn't loaded. Update Ollama, restart, and verify.
Windows has a built-in watchdog called TDR (Timeout Detection & Recovery). Its default timeout is 2 seconds. If a GPU kernel is busy for longer than that — which happens trivially during a large prompt prefill — Windows force-resets the driver.
The inference backend crashes. The client gets an HTTP 500. The server log shows the connection was "forcibly closed by the remote host." The Windows event log shows a LiveKernelEvent.
The default TDR timeout was designed for desktop rendering, not sustained GPU compute. On AMD hardware with DirectML, large prompts (8K+ tokens) trigger this reliably.
The fix: raise `TdrDelay` to 60 in the registry and reboot. This is a known pattern for GPGPU workloads on Windows, but almost nobody mentions it in the context of local LLM serving.
Modern models like `gemma4:12b` and `qwen3.5:9b` are thinking models. Before producing any visible output, they generate a thinking block — invisible reasoning that consumes the full token budget.
A model with `max_tokens: 2048` that generates 1,800 tokens of thinking leaves only 248 for actual content. With no output cap, a single turn can generate 5,000+ invisible tokens before producing a short answer. The user sees a 50-word response. The system generated 5,000 tokens to produce it.
The fix: for agentic tasks, disable thinking via `think: false` in the request body (Ollama's OpenAI-compatible endpoint honors this). Set an explicit `max_tokens` cap. The tradeoff is real — the model can't "reason through" complex problems — but for tool-calling workflows, the full token budget should go to content.
A 14B-parameter model on a 16GB card sounds like the best of both worlds. It isn't.
At 64K context, the model weights plus KV cache exceed available VRAM. The excess offloads to CPU. Prompt processing degrades from 73 tok/s to 27 to 17 as it churns through tokens. Client timeouts fire. The backend crashes.
On a 16GB card, the practical ceiling at 64K context is roughly 12B parameters. Beyond that, either reduce context length, choose a smaller model, or accept CPU-class performance.
The lineup that works on 16GB:
| Model | VRAM @ 64K | Speed |
|---|---|---|
| 8B (fast) | ~3 GB | fastest |
| 9B (balanced) | ~7.5 GB | ~57 tok/s |
| 12B (capable) | ~8.5 GB | ~40 tok/s |
| 14B+ | 14+ GB | CPU offload |
Size matters. Fit matters more.
Ollama's default `keep_alive` is 5 minutes. If your usage cadence exceeds that — one update every 30 minutes, for example — the KV cache is dropped between turns.
Every turn then re-prefills the full conversation history from scratch. At 313 tok/s with a 35K-token history, that's ~2 minutes of pure re-computation before any generation begins. The model is healthy. The GPU is loaded. The network is fine. But every turn takes 2 minutes because the context was thrown away. When engagement is based on spot checks and workflow approvals this is easy to miss but over time the lag adds noticable delay in everything you do.
The fix: set `OLLAMA_KEEP_ALIVE` to a value that matches your usage cadence. For interactive daily use, 24 hours is reasonable. The tradeoff is VRAM held for the cached session — on a 16GB card with a single model, this is fine.
This is the lesson that changed how we use the AI agent.
For two weeks, the agent kept claiming it had written files that didn't exist. "I've prepared a draft and added it to the inbox." Nothing on disk. Different sessions, different prompts, different days. It happened repeatedly.
My first assumption was a backend failure — the tool call was sent but silently dropped. Twenty-four controlled tests against the inference backend proved otherwise: every model, every schema, every configuration emitted clean tool calls at 100%. The backend was fine.
The failing session made 10 tool calls — reads, searches, browser executions — but zero writes. Then its final message, in prose, claimed it had written the file. The model fabricated the action. It narrated a completion it never performed.
This is not a backend failure. It is an instruction-following gap. The model chose prose over tool use.
The fix: add a grounding rule to the system prompt: never claim a file operation occurred unless a write tool call was actually emitted and returned success. Post-fix testing showed 10/10 clean tool calls with zero fabricated claims, across multiple models.
The broader lesson: treat agent output as untrusted until proven otherwise — including the agent's claims about its own actions. The model's confidence in its response has no correlation with whether it actually performed the task.
Running local LLMs for daily operational work is viable. The performance is good. The cost is fixed. The privacy is absolute. But it requires treating inference as infrastructure, not a black box.
The pattern across all seven lessons is the same: defaults are tuned for demos, not production. Context windows are too small. GPU detection is fragile. Thinking models consume budgets silently. Caches expire too quickly. And the model will confidently tell you it did something it didn't.
If you're building a local AI stack for real work, read the actual configuration at runtime. Verify GPU engagement. Set explicit context windows. Disable thinking for tool-calling workflows. And never trust the agent's self-report without checking the disk.
*Lessons from building a daily-use AI agent stack on Ollama (v0.32.11) with a consumer AMD GPU, August 2026. All configurations are reversible and documented.*
Built on a home lab, powered by local models, and owned by Andrew Katana.