AI runtime security

This guide covers post-deployment testing, monitoring, configured controls, incident containment, and scoped operational evidence. The NIST AI RMF is voluntary guidance; applicable legal and sector duties depend on the system, role, and jurisdiction.

What is AI runtime security?

AI runtime security is the post-deployment discipline of testing, monitoring, constraining, and responding to AI behavior in operation. For configured paths, teams can preserve evidence of which control was supposed to run and what it reported. That evidence supports review; it does not prove that every event was observed, a control was effective, or a framework requirement was met.

Think of it as the difference between inspecting a bridge once before opening day and instrumenting it with sensors that monitor stress, vibration, and load every second it’s in service. The inspection matters. But the sensors are what keep people safe.

Why runtime security matters

AI systems don’t stand still. Even when the model weights don’t change, the environment around them does:

• Prompts evolve. Teams update system prompts, add context, change instructions. Each change can subtly alter behavior.
• User inputs shift. Real-world users find interaction patterns that testing never anticipated. Adversarial users probe for weaknesses.
• Upstream models change. If you’re calling a hosted model via API, the provider may update the model behind the same endpoint. Your system prompt stays the same; the behavior underneath it changes.
• Oversight expectations evolve. Some EU AI Act roles and covered systems have post-market duties; NIST AI RMF offers voluntary monitoring guidance; Colorado’s current SB 26-189 instead centers on ADMT transparency and review records.

A model that passes every evaluation on Tuesday can behave differently by Friday. Runtime security fills that gap.

How it differs from pre-deployment testing

Pre-deployment testing answers the question: “Does this model behave correctly right now, on these inputs?” That’s a valuable question. But it’s not the only one.

Both can be necessary. Pre-deployment testing establishes a baseline; post-deployment monitoring tests how behavior changes in use. NIST AI RMF is a voluntary framework, while applicable laws and sector rules may impose their own ongoing risk-management or monitoring duties.

What post-deployment supervision looks like

Effective AI runtime security combines several capabilities:

1. Automated red teaming on a schedule

Rather than running penetration tests only once before launch, teams can run scoped adversarial probes on a recurring or event-triggered cadence. The open-source autoredteam project exercises configurable categories against supported endpoints; results describe the tested configuration and should not be treated as universal production coverage.

2. Behavioral drift detection

Teams can compare selected evaluation or production signals with defined baselines and investigate material changes. Statistical methods such as CUSUM can be appropriate for some stable, measurable signals, but detection quality depends on instrumentation, thresholds, sampling, and coverage. Glacis does not claim built-in CUSUM detection through autoredteam.

3. Governance framework mapping

Configured mappings can relate findings to relevant concepts in OVERT, NIST AI RMF, and MITRE ATLAS. A mapping helps reviewers navigate the evidence; it does not make a finding audit-ready or establish that a framework requirement is satisfied.

4. Operational evidence

Runtime findings can be recorded in signed, append-only evidence structures. A valid signature and available log proof support integrity for covered fields; they do not independently establish that a test ran as claimed, that monitoring was effective, or that every relevant event was captured.

Who needs AI runtime security?

Organizations deploying AI systems into consequential workflows may need a post-deployment security and supervision program proportionate to the use case. Examples include healthcare systems using ambient scribes or clinical decision support, financial-services firms running credit or fraud models, and AI labs operating products at scale.

The regulatory and assurance landscape is use-case-specific. The EU AI Act assigns post-market duties to covered roles and systems. The NIST AI RMF is voluntary guidance that treats monitoring as part of risk management. Colorado’s current SB 26-189 does not impose the repealed high-risk-system monitoring regime.

Getting started

One practical starting point is a scoped scan. autoredteam is open source and runs configurable probes against supported model endpoints. Its current README describes 19 attack categories and estimates a full real-model run at roughly 5–20 minutes depending on probes and judge configuration; it does not claim universal endpoint support, CUSUM drift detection, or automatic OVERT mapping for every finding.

From there, define a recurring or event-triggered test cadence, connect findings to the relevant review workflow, and declare which production paths can produce operational evidence. The objective is explicit coverage and defensible records, not a universal monitoring claim.