Agent observability · Runtime proof
AI Agent Observability: Named Gaps, Not Green Averages
AI agent observability is a named inventory of watched, unwatched, and accepted-unwatched — plus a signed receipt per governed action, not a green average.
Every agent platform now ships a dashboard. Latency, token spend, tool-call volume, error rate — a wall of green that tells you the system is busy and mostly healthy. That is monitoring, and monitoring has its place. But AI agent observability, the thing a security reviewer, a board, or a regulator actually asks for, is a different question: of everything this agent can do, what exactly is being watched, what is not being watched, and what have we deliberately accepted unwatched? If you cannot answer that as an enumeration — a list with names on it — you do not have observability. You have an average.
This page is about that difference. It covers why dashboard averages are where agent failures hide, what a named inventory looks like in practice, how signed receipts turn “watched” into “provably governed,” and how to tell monitoring, observability, and evidence apart when vendors use the words interchangeably. It builds on our longer treatment of runtime coverage for agentic AI security.
Why AI agent observability is not a dashboard
Averages are where agent failures hide. Say your panel shows 99% of tool calls “covered.” Covered out of what denominator? Which calls sit in the other one percent? If the agent makes ten thousand tool calls a day, that sliver is a hundred actions a day nobody saw. And for an agent that holds credentials, hits payment APIs, and writes to systems of record, the incident you will eventually have to explain lives in that sliver — not in the green 99%.
A dashboard answers “is the system healthy right now?” Observability answers a harder question: “can I ask something new about what happened and get a true answer?” For agents, the new question is almost always some version of: which actions were governed, which were not, and can you prove it? No aggregate metric can answer that. Only an inventory can.
Agents make this worse in a specific way: they chain. A single user request fans out into a dozen tool calls across five systems, and each call's input depends on the last call's output. An average over that chain tells you the chain mostly worked. It cannot tell you which link was never watched — and a compromise only needs one.
There is a design rule we hold to, and it is worth stealing whatever tooling you end up with: the panel never renders whole-green. A coverage view that can show 100% has either named zero gaps — which is dishonest — or defined “coverage” to exclude the unwatched — which is circular. An honest panel keeps the accepted-unwatched set on its face at all times, so nobody in the room can mistake silence for safety. Green is a state you earn per named action, not a background color for the things you forgot to list.
The named inventory: coverage as enumeration, not percentage
The deliverable of agent observability is a list, not a number. Every capability the agent holds is named into exactly one of three buckets:
- Watched. A control sits in the action path — permit, deny, override, or escalate — and every decision mints a signed receipt.
- Unwatched. A known gap. It has an owner, a ticket, and a date, and it sits on the panel visibly as a gap rather than dissolving into an average.
- Accepted-unwatched. A deliberate, signed-off risk acceptance with an expiry date. Someone with authority chose this, in writing, and the choice is on the panel too.
The discipline is that a gap is named, never averaged away. You can compute a percentage from the inventory if a board deck demands one, but the enumeration is primary and the percentage is derived. Reverse that order and you get the classic failure mode: a green number whose denominator nobody can produce, covering — or rather not covering — a set of actions nobody can name.
A fixture example, using our standard test org nimbus. Their support agent holds fourteen tools. Eleven are watched: a guard sits in the path and receipts every decision. Two are unwatched and named as gaps on the panel — calendar writes and CRM note writes; receipting for both is designed, not yet live. One is accepted-unwatched: read-only access to the knowledge base, signed off through the end of the quarter. That statement is auditable end to end. “93% coverage” is not, because it cannot tell you which seven percent — and the seven percent is the whole point.
The inventory is a living artifact, not a slide. It is reviewed on a cadence, every accepted-unwatched entry carries an expiry, and every tool the agent gains lands in a bucket the day it is granted — not the quarter after.
Receipts: the per-action record
The inventory tells you what is watched. Receipts tell you what actually happened inside the watched set. Each time the guard decides — permit, deny, override, escalate — it mints a signed receipt at the moment of enforcement. The receipt is an Ed25519-signed, hash-chained record, anchored to an RFC 6962 transparency log, and it verifies offline against a pinned key. Checking it requires no trust in us; the cryptography is the whole claim.
Because the receipts are hash-chained, the record is tamper-evident: drop one, alter one, reorder them, and verification fails. Because they are anchored to a transparency log, the chain itself cannot be quietly rewritten after the fact.
The receipts are OVERT receipts. OVERT is the open, royalty-free receipt standard we publish at overt.is: the format is public, the verifier is public, and nothing about checking a receipt requires our infrastructure or our permission. A standard you can read is the difference between “trust our dashboard” and “check it yourself.”
Receipts are content-free by construction. They carry hashes, counters, and labels — never prompts, never payloads, never personal data. That is what makes them safe to hand to an auditor, an insurer, or a regulator: you can prove an action was governed without disclosing what the action contained. You can verify a receipt yourself, in your browser, or see how receipts assemble into an auditor-ready bundle in the sample evidence pack.
Two limits, stated plainly, because they are the difference between evidence and marketing. First, receipts prove faithful recording of what passed the boundary — not that every action was captured. That is exactly why the named inventory exists: the unwatched set is where receipts are not, and the panel says so. Second, receipts do not prevent the next incident, and neither do we. They make what happened provable, after the fact, by someone who was not in the room. Prevention is an adjective; a receipt is a record.
For the deeper treatments: what makes AI attestation independent covers why a system vouching for itself is a log rather than evidence, and AI data security: verify the receipt walks the anatomy of a single artifact.
AI agent monitoring vs observability vs evidence
Vendors use these three words as synonyms. They are not, and the distinctions are load-bearing when you are the one answering questions after an incident.
- AI agent monitoring is metrics and alerts about system health: latency, error rates, token spend, tool-call volume. It tells you that something is wrong. Necessary, and nowhere near sufficient.
- AI agent observability is the ability to ask new questions about behavior — and for agents, the first new question is always coverage. It tells you what you can and cannot see, as a named inventory.
- Evidence is a signed, independently verifiable record of what a control did at the action boundary. It tells someone who was not in the room what held — without taking the operator's word for it.
The failure modes pair off. Monitoring without observability hides gaps inside averages — the 99%-green problem above. Observability without evidence is still self-report: you know what you watch, but you are vouching for it yourself. And evidence without an inventory proves only what happened inside the watched set, with nothing to say about what was never watched. You need all three layers, in that order. We wrote about the third layer in AI in cyber security: the evidence layer, and about what this looks like under a real attack class in prompt injection: prove what held.
What to demand from any vendor
Whether you buy from us or assemble this yourself, the questions are the same, and any serious vendor should answer them without adjectives:
- Show the list, not the number. Ask for the watched / unwatched / accepted-unwatched enumeration. If the coverage panel can render whole-green, ask what it excludes.
- Ask for per-action artifacts, not aggregates. A signed receipt per governed action that you can verify offline, against a pinned key, without the vendor's infrastructure or permission.
- Ask what crosses the trust boundary. The answer should be hashes and labels — never your prompts or your customers' data.
- Ask for stated limits. Receipts prove faithful recording, not completeness. A vendor who claims “comprehensive coverage,” or that their tooling prevents incidents, is selling adjectives. Ask for the denominators, exclusions, and intervals instead.
- Ask who vouches. If the party attesting is the party being governed, it is a log, not evidence. Independence is structural, not contractual.
If a vendor answers all five cleanly, the word “observability” means something. If not, you are buying a dashboard with a better name.
If you're deploying agents into regulated or high-stakes environments, get runtime coverage and see what a named inventory plus signed receipts looks like against your own tool calls. You can also verify a receipt yourself, or read the open standard at overt.is.