How to build an AI agent: what you need beyond the prompt

Agent vs chatbot: the difference is one word

A chatbot answers. An agent acts: it breaks the task into steps, calls functions inside your systems, checks the result and carries the task to the end. “Find the contract, calculate the balance, file the request, hand it to a manager” is an agent. “Tell me how to file a request” is a bot.

Everything else follows from that. The moment a system is allowed to change things, it needs authority boundaries, an action log and a way to roll back what it did. The model here is one component, not the whole product.

What an agent is made of

A working agent has six parts. Missing any one of them looks exactly like “the demo works, production does not”.

• Model and prompt. They set behaviour and reasoning format. They change most often and cost the least.
• Tools. Functions and APIs through which the agent reads and changes data. Each tool is described so the model needs no guesswork: what it does, what it takes, what it returns, what counts as an error.
• State. What the agent remembers inside one task and between tasks. Without explicit state it loses context halfway through a long scenario.
• An authority policy. What may be done without confirmation, what requires a human, what is never allowed. These are system rules, not a line in the prompt.
• Observability. A log of steps, calls and decisions: what the agent saw, what it called, what came back. Without it any error becomes unreproducible.
• Evaluation. A set of scenarios with known correct outcomes that every prompt or model change is checked against.

Six steps to a first working agent

Order matters more than speed. Each step removes a specific class of problems, and skipping one brings it back a month later.

• Pick one process. Not “let’s adopt AI” but “processing inbound partner requests”. A narrow process with a clear start and end.
• Write the steps down the way a human performs them. Without that description the agent automates your guesses about the process, not the process.
• Build the tools for those steps. There are usually fewer than expected: three to five functions cover most of the scenario.
• Fix the boundaries. What the agent does silently, what it shows to a human, what it never touches. Separately, a per-task limit: on steps, on time, on cost.
• Run it on real data in shadow mode. The agent proposes an action, a human approves it. That is how you collect the evaluation set and see the real edge cases.
• Remove the confirmation where the agent is consistently right, and only there. Widening authority is a separate decision per action, not one global switch.

Where agents break in production

A demo and an operating system differ not in model quality but in the fact that production brings repeats, failures and hostile input.

• Idempotency. The agent retried a step after a timeout and the request was filed twice. Every state-changing action must be safe to repeat.
• Cost. The think-call-think loop multiplies tokens. Without a per-task limit and a per-step cost breakdown the bill grows unnoticed.
• Silent failures. A tool returned an empty response and the agent kept reasoning as if it had data. An empty result must be an explicit error, not an invitation to improvise.
• Access rights. An agent acts on someone’s behalf. If it sees more than the employee does, what you built is not automation but a leak.
• Drift. You change the model or the prompt and behaviour shifts. Without an evaluation set you hear about it from users.

How to estimate cost up front

The cost of an agent is not a per-million-token price but a product: steps per task, tokens per step, tasks per day. Break the scenario into steps and price each: usually one or two steps account for most of the spend.

Then three levers work. Routing: a heavy model for the reasoning step, a light one for extraction and formatting. Caching: the same context should not be paid for twice. Context trimming: the agent needs the relevant fragment, not the whole document.

When you do not need an agent

An honest answer saves more money than a clever architecture. You do not need an agent if the process is deterministic and expressible as branching logic: a plain workflow or integration is cheaper and more reliable there.

You also do not need one when there is no source of truth. If the data sits in three places and disagrees with itself, the agent will confidently act on wrong data, faster than a human and at a larger scale.