Building Agent Systems on Small Local Models

The constraint comes first

Start with where the data lives. Inside an enterprise network — device configs, topology, routing state, audit trails, customer environments — the most valuable data is also the most sensitive. In many of these environments you simply cannot send that data to a hosted model. Not for cost reasons, for security and compliance ones: the data isn’t allowed to leave the building.

So the model has to run locally. And that’s where the second constraint bites.

A local frontier-class model is expensive — not per token, but in hardware. Running a 100B+ model at usable latency means a serious GPU footprint that most teams can’t justify per-workstation or per-site. Walk down from there and you land, in practice, on models in the ~30B range. That’s the sweet spot for local deployment today: it fits on hardware people actually have, and it runs at a latency you can build an interactive agent around.

The catch is that a 30B model is not a frontier model, and it will remind you constantly.

What a 30B model gets wrong

Three limits show up immediately when you try to build an agent on one:

1. Hallucination. Ask it to do multi-step work and it will happily invent a plausible-looking answer instead of doing the work — fabricate a summary, guess a value, loop on “let me just do X” without ever doing X.
2. Thin and stale knowledge. It doesn’t know your environment, your conventions, or the specifics of the task domain, and its general knowledge is both smaller and older than a frontier model’s.
3. Short usable context. The advertised context window is not the usable one. In practice you’re working with something closer to 8K tokens on the small end, 32K in the middle. Fill it with tool schemas and raw data and the model stops reasoning.

You don’t fix these by swapping the model — that door is closed. You fix them in the harness around it. Here’s how each one maps to a concrete technique.

Fixing hallucination: give it less rope, and less to invent

A small model hallucinates most when it’s asked to produce something it should be computing, or to plan something it should be following.

• Push work into code, not the model. When a task spans N items, don’t have the model orchestrate N tool calls — write one tool that iterates in Python and call it once. A change-plan generator that computes deterministically makes 0 database calls where a naive agent loop made 60–80. The model that never has to hold 80 intermediate results never gets to invent them.
• Tell it WHAT, not HOW. Hand a small model a per-step checklist and it executes mechanically, loops, and eventually hallucinates a “done.” Give it a goal plus constraints and it reasons. Across a five-attempt experiment, every “do step 1, then step 2…” prompt failed; “here is the goal and the constraints” succeeded.
• Grade the output deterministically. A cheap Python predicate that checks “did the agent actually produce the artifact, or just talk about it?” catches the fabricated-summary failure before it ever reaches a user — and costs nothing, versus an LLM grading an LLM.

Fixing thin knowledge: retrieve it, don’t expect it

The model doesn’t know your world, so you feed the relevant slice of it in at call time.

• Ground every answer in real data. The agent reads from the actual source of truth — the database, the live device — rather than from what the model “remembers.” Its job is to reason over retrieved facts, not to recall them.
• Inject knowledge as memory, not prompt edits. Capture domain knowledge and hard-won constraints as small, retrievable guides, and inject the relevant ones into every model call, scope-filtered and quota-capped. This is also how the system improves: a lesson learned once becomes a guide that steers every future run — far more durable than editing the prompt.

Fixing short context: spend the budget deliberately

Treat context as a hard budget and account for every token.

• Size everything to the tier. Assume ~8K / 32K / ≥200K usable context and scale accordingly — how many memories you recall, how many rows of data you attach. A small model never gets handed a wall it can’t reason over.
• Make the orchestrator a thin router. The top-level agent routes intent to one specialist and returns that specialist’s output verbatim — no re-synthesis. Capability lives in the specialists, so the router’s context stays tiny.
• Ration tools ruthlessly. Every tool definition costs ~150–300 tokens on every call — sixteen tools is ~3,200 tokens of permanent overhead, and a long tool list also raises tool-selection error. Cap the tool count hard (a handful per agent), keep tool docstrings short, and prefer plain scripts over heavyweight tool definitions.

The pattern

None of this is a model trick. It’s harness engineering: push work into deterministic code, ground answers in retrieved data, steer behavior through injected memory, and spend a tight context budget on purpose. Do that, and a model you can run on your own hardware — inside your own network, with nothing leaving the building — does work that “obviously needs a frontier model.”

Where we put this into practice

This is exactly the design philosophy behind OLAV — an open-source agent platform for network operations, built from the ground up to run on local small models. Every principle above is a load-bearing rule in it: the tiered context budget, the thin-router orchestrator, the tool caps, WHAT-not-HOW prompting, fat tools, and memory-based steering are all enforced in the architecture, not left to good intentions.

If you’re trying to build a serious agent on hardware you control, OLAV is a working reference for how the pieces fit together. It’s open source under BSL-1.1 — explore it at github.com/james-olavai/olav.