Agent observability · Runtime proof
AI Agent Observability: Named Gaps, Not Green Averages
Observability reports what instrumented systems recorded. Glacis adds explicit control scope and signed operational evidence for consequential actions routed through a configured boundary. It also names the watched, unwatched, and accepted-unwatched paths.
Many agent platforms ship a dashboard. Latency, token spend, tool-call volume, error rate: a wall of green that can show the system is busy and mostly healthy. That is monitoring, and monitoring has its place. For a security reviewer, board, or regulator, another useful question is: of the capabilities in the declared inventory, what is watched, what is not watched, and what has been accepted unwatched? Without that enumeration, a coverage average is difficult to interpret.
This page is about that difference. It covers why dashboard averages can hide important gaps, what a named inventory looks like in practice, how signed receipts add a checkable record to watched actions, and how to tell monitoring, observability, and evidence apart when vendors use the words interchangeably. It builds on our longer treatment of runtime coverage for agentic AI security.
Why AI agent observability is not a dashboard
Averages are where agent failures hide. Say your panel shows 99% of tool calls “covered.” Covered out of what denominator? Which calls sit in the other one percent? If the agent makes ten thousand tool calls a day, that sliver is a hundred actions a day nobody saw. And for an agent that holds credentials, hits payment APIs, and writes to systems of record, the incident you will eventually have to explain lives in that sliver, not in the green 99%.
A dashboard answers “is the system healthy right now?” Observability supports a harder question: “can I ask something new about what happened and get a supportable answer?” For agents, that question is often some version of: which actions were inside the declared control scope, which were not, and what records exist? An aggregate metric alone cannot answer that; it needs a defined inventory behind it.
Agents make this worse in a specific way: they chain. A single user request fans out into a dozen tool calls across five systems, and each call’s input depends on the last call’s output. An average over that chain tells you the chain mostly worked. It cannot tell you which link was never watched. A compromise only needs one.
There is a design rule we hold to, and it is worth considering whatever tooling you use: keep accepted-unwatched and known-unwatched paths visible. A 100% figure is meaningful only for a stated denominator and time interval; it says nothing about capabilities omitted from that denominator. A named inventory helps prevent silence from being mistaken for safety.
The named inventory: coverage as enumeration, not percentage
The useful deliverable of agent observability is a list behind the number. Capabilities in the declared inventory can be assigned to three buckets:
- Watched. A configured control sits in the routed action path (permit, deny, override, or escalate) and covered decisions can emit signed records.
- Unwatched. A known gap. It has an owner, a ticket, and a date, and it sits on the panel visibly rather than dissolving into an average.
- Accepted-unwatched. A documented risk acceptance with an owner and expiry date, shown on the panel alongside the watched set.
The discipline is that a gap is named, never averaged away. You can compute a percentage from the inventory if a board deck demands one, but the enumeration is primary and the percentage is derived. Reverse that order and you get the classic failure mode: a green number whose denominator nobody can produce, covering (or rather not covering) a set of actions nobody can name.
Consider a fixture using our test organization, nimbus. Its support agent has fourteen inventoried tools. Eleven are watched: a configured guard sits in the routed path and can record its covered decisions. Calendar writes and CRM note writes are unwatched and named as gaps; receipting for both is designed, not yet live. One is accepted-unwatched: read-only access to the knowledge base, signed off through the end of the quarter. That enumeration is more reviewable than “93% coverage” because it exposes the stated denominator and the named remainder.
The inventory should be a living artifact, not a slide. Review it on a cadence, give accepted-unwatched entries an owner and expiry, and classify newly granted tools promptly rather than waiting for a quarterly review.
Receipts: the per-action record
The inventory declares what is watched. Receipts add a signed account of reported decisions inside that set. A configured guard can emit an Ed25519-signed, hash-linked record for a covered permit, deny, override, or escalation decision. A verifier can check that record offline against a pinned key. A valid signature supports integrity and key attribution for the covered fields; it does not establish that the signer told the truth, that the control was effective, or that every action was captured.
Hash links can make alteration or reordering detectable within the chain presented for verification. A separately published anchor can add evidence that a commitment existed by a particular point. Neither mechanism detects an event that never entered the covered route or a claim that was false when signed.
The receipts are OVERT receipts. OVERT is the open, royalty-free receipt standard we publish at overt.is: the format is public, the verifier is public, and nothing about checking a receipt requires our infrastructure or our permission. A standard you can read is the difference between “trust our dashboard” and “check it yourself.”
A deployment can configure receipts to carry hashes, counters, labels, and selected metadata instead of raw prompts or payloads. Whether protected payloads remain inside a particular boundary depends on deployment routing, configuration, and egress controls; the receipt alone does not prove locality. You can Verify a record yourself, in your browser, or see how receipts assemble into a review bundle in the sample evidence pack.
Two limits matter. First, receipts do not prove completeness: they cover records produced on named paths, while the inventory must disclose known unwatched and accepted-unwatched paths. Second, receipts do not prevent an incident, certify safety or compliance, or prove control effectiveness. They make a bounded account independently checkable after the fact.
For the deeper treatments: what makes AI attestation independently verifiable explains the distinction between checking a signature and trusting the underlying claim, and AI data security: verify the receipt walks the anatomy of a single artifact.
AI agent monitoring vs observability vs evidence
Vendors use these three words as synonyms. They are not, and the distinctions are load-bearing when you are the one answering questions after an incident.
- AI agent monitoring is metrics and alerts about system health: latency, error rates, token spend, tool-call volume. It can indicate that something may be wrong.
- AI agent observability supports new questions about behavior and scope. For agents, a key question is which inventoried paths are watched and which are not.
- Operational evidence is a signed, scoped record of a reported control decision at the action boundary. Another party can check its integrity and signer attribution without treating that check as proof of truth or effectiveness.
The failure modes pair off. Monitoring without an inventory can hide gaps inside averages. Observability without signed evidence relies on the operator’s records alone. Evidence without an inventory describes only the covered set and says nothing about omitted paths. The three layers answer different questions and should be assessed together. We wrote about the evidence layer in AI in cyber security, and about one attack class in prompt injection: prove what held.
What to demand from any vendor
Whether you buy from us or assemble this yourself, the questions are the same, and any serious vendor should answer them without adjectives:
- Show the list behind the number. Ask for the watched / unwatched / accepted-unwatched enumeration and its denominator.
- Ask for scoped artifacts, not only aggregates. Can a supported record be checked offline against a pinned key, and which actions can emit one?
- Ask what crosses each trust boundary. Require a deployment-specific data-flow account; do not infer payload locality from the presence of hashes or signatures.
- Ask for stated limits. Receipts support integrity and signer attribution for covered claims, not completeness, effectiveness, safety, or incident prevention. Ask for denominators, exclusions, and intervals.
- Ask who signs and operates the path. Disclose key custody, signer provenance, and organizational relationships. A second signature or service-operated witness does not by itself establish organizational independence.
If a vendor answers all five cleanly, the word “observability” means something. If not, you are buying a dashboard with a better name.
If you’re deploying agents into regulated or high-stakes environments, Talk to us and see what a named inventory plus signed receipts looks like against your own tool calls. You can also Verify a record yourself, or read the open standard at overt.is.