Why AI observability isn't enough

A financial-operations team ran an autonomous agent that took approved invoices and entered them into the accounting system: vendor, amount, cost center, due date, one row per invoice. It was fast and it almost never errored. One month the numbers stopped tying out. An analyst traced it back: for a batch of invoices the agent had entered the correct amount, but under the wrong cost center. Right value, wrong field. Every field it touched contained a valid number. The entries posted. The agent's traces were clean, its latency was healthy, its outputs passed schema validation, and no guardrail had anything to object to, because nothing it wrote was malformed. The observability stack reported a flawless month. The books disagreed.1

Every enterprise that reaches this point has already spent real money on tooling. It has traces and spans, it has evals, it has guardrails, it has model dashboards. The natural conclusion is that the tooling was not deployed well enough, that one more integration or one more alert would have caught it. That conclusion is wrong, and understanding why is the point of this paper. The gap the cost-center error fell through is not a coverage problem. It is a problem of what these tools are for.

What observability does well

It is worth being fair to observability, because the argument here is not that it is a failure. It is a triumph. Distributed tracing, structured logs, latency and error dashboards, span-level timing: these turned opaque distributed systems into things engineers can reason about, and they are just as useful pointed at an agent. When an agent times out, loops, blows its token budget, calls a tool that returns a 500, or crashes mid-run, observability catches it and catches it fast. Those are real failures and you want to know about them the moment they happen.

The same is true of its neighbors. Evaluation tells you whether a model is capable before you ship it and whether a change made it better or worse. Guardrails stop known-bad actions in real time, keeping a value inside a limit or refusing a disallowed request. Model metrics track quality, drift, and cost at the level of the model itself. Every one of these does a real job. An enterprise running autonomous AI should have all of them. The problem is not that any of them is bad at its job. The problem is that none of their jobs is the one that decides trust.

The category error

Here is the distinction the whole paper turns on. All of these tools observe the agent. They watch the machine run. Correctness does not live in the running. It lives in the result the running produced, which is a different object in a different place.

The cost-center agent ran perfectly. Every fact observability can know about a run was good: it started, it stepped through its logic, it wrote valid values, it finished. But "was the invoice booked to the right account?" is not a fact about the run. It is a fact about the accounting system a moment later, and about a definition of correctness (right amount and right cost center) that the agent's own logs simply do not contain. You could instrument that agent with the best observability stack in the world and add a hundred more spans, and not one of them would turn red, because there was nothing wrong with the running. The wrongness was in the outcome, and the outcome was somewhere the tools were not looking.

This is why it is a category error rather than a coverage gap. A coverage gap you close by instrumenting more. A category error you cannot close by doing more of the same thing, because the answer you need is not of the kind the tool produces. You can measure a run in infinite detail and never obtain a judgment about a result. To get that, something has to look at the result and compare it to what "right" was supposed to mean, which no tool that only watches the agent can do.

Why green dashboards lie

Follow the consequence and you arrive at the most disorienting experience in operating autonomous AI: the dashboard is entirely green and the work is wrong. Not amber, not flickering. Green. Every panel healthy, every check passed, and underneath it a batch of invoices booked to the wrong accounts.

The reason is structural, not a matter of tuning. Every signal on that dashboard is computed from the agent's own activity. Latency comes from the agent's timing. Error rate comes from the agent's exceptions. Eval scores come from grading the agent's output against a rubric. Guardrail status comes from the agent's actions against a set of rules. When a decision is confident, well formed, and wrong, it produces clean activity, so every signal derived from that activity reads healthy. The dashboard is not malfunctioning. It is reporting exactly what it measures, which is the agent, faithfully. It is green because the agent's behavior was green. The outcome, which no panel on it is measured against, was red.

This is why you cannot alert your way out of the problem. An alert fires on a threshold crossed in the data you collect. If the data you collect is all activity, your alerts can only ever tell you the activity looked abnormal. The dangerous failures do not look abnormal. They look like ordinary, successful work, because that is exactly what they are until you check them against reality.

What each tool can and cannot answer

Laid side by side, the pattern is unmistakable. Each of the four tools an enterprise already runs answers a genuine question about the agent, and each is silent on the one question that decides whether the decision was right.

The tool	What it answers well	What it cannot answer
Observability traces, logs, latency, errors	Did the agent run, and did it run cleanly? Where did it spend time, what did it call, where did it break?	Was the decision it produced right? Correctness is not in the trace.
Evaluation rubrics, test sets, scoring	Is the model capable? Did this change make outputs better or worse against a benchmark?	Did this specific decision achieve the real outcome days later, in the system where it landed?
Guardrails limits, filters, policy checks	Did the agent attempt a known-bad action? Was a value out of bounds or a request disallowed?	Did an allowed action produce the right result? A valid move can still be the wrong one.
Model metrics quality, drift, cost	Is the model healthy on average? Is it drifting, and what is it costing?	Was this one decision correct? An average is not a verdict on the case in front of you.

The table is not an indictment. It is a map of responsibilities. Read across any row and the tool is excellent at the thing in the middle column and structurally silent on the thing in the right one. The right column is a single question wearing four disguises: was the outcome right? That question is what Decision Assurance exists to answer, and the reason it is a distinct discipline rather than a feature you could add to any box in this table.

So what does look at the result?

If no tool that watches the agent can judge the outcome, the fair question is what can. The honest answer is that nothing divines it, Provy included. The result has to be connected back. Provy does not reach into your accounting system, your ticket queue, or your CRM and read them. It compares two things you supply: a short contract written before the run, and the settled result reported back to it afterward.

The contract is a plain checklist of what a good result means, one condition at a time: the amount matches, the cost center matches, the ticket did not reopen. It is written up front, in business terms, before anyone knows how the run will go. When the real result is known, it is pushed back to Provy and tagged to the same work item, by the system of record or by a person. Provy compares the two condition by condition and marks the run a success only if every condition held. Where no result has been connected yet, the honest verdict is "not measured," never a quiet pass.

This is why the discipline is reconciliation, not observation. Provy is the referee that holds the decision up against the settled result, not a camera inside your books. The check gets its strength the same way a financial reconciliation does: it sets an independent record of what happened next to what was expected, and the two have to agree. Take the cost-center month and run it through that lens.

Compose, do not replace

The wrong lesson to draw is that observability and its neighbors should be thrown out. The right lesson is that they compose with something above them. Observability tells you the agent ran and how. Evaluation tells you the model is capable. Guardrails keep it inside the lines. Decision Assurance tells you whether the decision, having run cleanly and stayed inside the lines, actually achieved the outcome it was for. Each layer answers a question the one below it cannot, and an enterprise operating autonomous AI needs all of them.

In practice

One idea, seen across different kinds of work. These are illustrations of the concept, not product walkthroughs.

The cost-center error is the whole argument in one incident. Perfect observability, valid outputs, no guardrail breach, and a month of books that did not tie out. Nothing that watched the agent was wrong. Everything that watched the agent was beside the point, because the failure was in the result, and the result is exactly what none of them measured. The next paper takes up the tool that does: Outcome Intelligence, measuring AI by the result it was meant to produce rather than the activity it produced along the way.