AI Explainability & Transparency Guide

What is AI Explainability?

AI explainability (XAI) is the ability to understand and articulate how an AI system produces its outputs. At its core, explainability enables humans to comprehend, trust, and effectively manage AI decision-making.^[4]

A system is explainable when you can answer questions like:

• Which inputs influenced this decision? What features or data points drove the prediction?
• How were inputs weighted? Which factors mattered most, and which were ignored?
• Why this decision versus alternatives? What would need to change for a different outcome?
• Is the logic consistent and fair? Does the model apply the same reasoning across similar cases?

Explainability vs. Interpretability vs. Transparency

These terms are often conflated but represent distinct concepts:

Key distinction: You can have explainability without interpretability. A deep neural network is not interpretable. You cannot inspect its millions of parameters and understand its logic. But you can use techniques like SHAP or LIME to generate explanations for specific predictions, making the model explainable even though it remains fundamentally uninterpretable.^[5]

The Four Levels of Explainability

Models fall into four categories based on how explainability is achieved:^[4]

Why Explainability Matters

Explainability has evolved from academic curiosity to business imperative. Four forces are driving this shift:

1. Trust and User Adoption

People affected by consequential decisions need information they can use to understand, question, and, where applicable, challenge the outcome. Operators also need enough insight to decide whether an output is appropriate for the specific case.

Research shows that appropriate explanations increase user trust and model adoption, but only when the explanations are accurate and meaningful. Oversimplified or misleading explanations can backfire, creating false confidence in unreliable systems.^[6]

2. Regulatory Compliance

Some legal regimes impose transparency, information, or review duties in defined circumstances:

• GDPR Article 22 (2018): Right not to be subject to solely automated decision-making with legal or similarly significant effects. Articles 13-15 require “meaningful information about the logic involved.”^[3]
• EU AI Act Article 13 (relevant Annex III high-risk obligations apply from 2 December 2027 under the AI Omnibus in force since 27 July 2026): High-risk AI systems must be “designed and developed in such a way to ensure that their operation is sufficiently transparent to enable users to interpret the system’s output and use it appropriately.”^[7]
• Colorado SB 26-189 (compliance from January 1, 2027): Repeals and replaces the 2024 Colorado AI Act. Requires deployers of covered automated decision-making technology (ADMT) to disclose, after an adverse consequential decision, the ADMT’s role and the principal personal data it processed, plus how consumers can exercise their rights.^[8]

3. Model Debugging and Improvement

Explainability tools reveal when models learn spurious correlations or fail to generalize. Classic examples include:

Without explainability techniques, such failures remain hidden until production deployment, when they’re costliest to fix.

4. Fairness and Bias Detection

Explainability enables detection of discriminatory patterns. The COMPAS recidivism algorithm, used to inform bail and sentencing decisions, was found to exhibit racial bias partly through explainability analysis showing race-correlated features (zip code, prior arrests) disproportionately influenced risk scores for Black defendants.^[10]

ProPublica’s investigation revealed COMPAS was twice as likely to incorrectly label Black defendants as high-risk compared to white defendants (45% vs. 23% false positive rate). Explainability techniques made the bias measurable and actionable.^[10]

Types of Explanations

Not all explanations serve the same purpose. Understanding the different types helps you choose appropriate techniques for your use case.

Global vs. Local Explanations

Use global explanations for model validation, bias assessment, and understanding system-wide behavior. Use local explanations for regulatory compliance, individual decision justification, and debugging specific failures.

Model-Agnostic vs. Model-Specific

Explainability Techniques

The XAI toolkit has expanded since DARPA’s XAI program launched in 2016. Widely used techniques include the following; suitability depends on the model, decision, audience, and explanation objective.^[11]

SHAP (SHapley Additive exPlanations)

SHAP, introduced by Lundberg and Lee in 2017, uses game theory (Shapley values) to assign each feature an importance value for a particular prediction. SHAP has become the de facto standard for feature importance explanations.^[12]

When to use SHAP: Feature importance for individual predictions, model debugging, identifying bias sources, explaining tree-based models (TreeSHAP is particularly fast), regulatory compliance documentation.

Limitations: Computationally expensive for large models, assumes feature independence (can be problematic with correlated features), requires careful baseline selection.

LIME (Local Interpretable Model-agnostic Explanations)

LIME, developed by Ribeiro et al. in 2016, explains individual predictions by fitting a simple interpretable model (like linear regression) locally around the prediction being explained.^[13]

When to use LIME: Quick explanations for debugging, text classification explanations, image classification with superpixel highlighting, situations where speed matters more than theoretical guarantees.

Limitations: Explanations can be unstable (small input changes produce different explanations), no theoretical guarantees like SHAP, requires careful tuning of locality parameters.

Attention Mechanisms

Attention mechanisms, central to transformer architectures, expose attention weights that can support analysis of token relationships. Those weights do not by themselves establish why an output occurred or provide a causal explanation.^[14]

When to use attention: Natural language processing tasks, machine translation explanations, document summarization justification, any transformer-based model where you need to show which inputs influenced outputs.

Limitations: Attention weights don’t always correspond to true feature importance, multi-head attention can be difficult to interpret, attention is correlational not causal.

Counterfactual Explanations

Counterfactual explanations answer: “What would need to change for the model to produce a different output?” They’re particularly valuable for regulatory compliance and user-facing explanations.^[15]

When to use counterfactuals: Credit decisions (GDPR/ECOA compliance), hiring/admission decisions, medical diagnosis (“what symptoms would change the diagnosis?”), any domain where users need actionable feedback.

Limitations: May suggest unrealistic or unethical changes, can be expensive to compute, not always unique (multiple counterfactuals may exist).

Feature Importance and Permutation Importance

Feature importance techniques measure how much each feature contributes to model performance. Permutation importance shuffles feature values and measures the drop in model accuracy. Features causing large drops are deemed important.^[16]

When to use feature importance: Model selection and comparison, feature engineering validation, compliance documentation showing what data the model uses, communicating with non-technical stakeholders.

Transparency Requirements

Beyond explainability techniques, transparency requires documentation that makes AI systems understandable at the system level. Two frameworks have emerged as standards:

Model Cards

Introduced by Mitchell et al. (Google) in 2019, model cards provide structured documentation of ML models including intended use, training data, performance across demographics, ethical considerations, and known limitations.^[17]

Some model and system providers publish model cards or system cards, but availability and naming change by release. For high-risk systems, the EU AI Act imposes specified technical-documentation and information duties; it does not generally mandate a document called a “model card.”^[7]

System Cards and Documentation

System cards extend model cards to describe complete AI systems including data pipelines, human-in-the-loop components, deployment infrastructure, monitoring procedures, and incident response plans. ISO 42001 and the EU AI Act require system-level documentation beyond individual model cards.^[18]

Regulatory Requirements

Explainability and transparency have shifted from optional features to regulatory mandates. Here’s what’s required and when:

GDPR Automated-Decision Transparency and Safeguards

GDPR Article 22 restricts decisions based solely on automated processing that produce legal or similarly significant effects. In the relevant automated-decision cases, Articles 13 to 15 require meaningful information about the logic involved, significance, and envisaged consequences; Article 22 also provides safeguards including human intervention and the ability to contest a decision where an exception permits the processing.^[3]

Enforcement: The applicable GDPR fine tier depends on the provision and circumstances; the Regulation’s upper tier can reach €20 million or 4% of worldwide annual turnover.^[3]

EU AI Act Article 13: Transparency and Information to Deployers

The EU AI Act requires that high-risk AI systems be designed to ensure “sufficiently transparent” operation enabling users to “interpret the system’s output and use it appropriately.”^[7] The AI Omnibus entered into force on 27 July 2026. Relevant Annex III high-risk obligations apply from 2 December 2027; relevant Annex I product-embedded obligations apply from 2 August 2028.

Penalties: Under Article 99, non-compliance with the listed operator obligations for high-risk systems is generally subject to a ceiling of €15 million or 3% of worldwide annual turnover, whichever is higher. The €35 million or 7% tier applies to prohibited practices under Article 5; SME calculations use the lower applicable amount.^[7]

Colorado SB 26-189 (Automated Decision-Making Technology)

In May 2026, Colorado repealed and replaced its 2024 AI Act (SB 24-205) with SB 26-189, signed May 14, 2026. The “high-risk AI system” / reasonable-care framework no longer governs; the law now regulates “covered automated decision-making technology (ADMT)” used to materially influence a consequential decision. Substantive compliance commences January 1, 2027 (the duties are enacted but not yet operative). Under the new framework, developers must provide deployers with documentation supporting appropriate use, and deployers must, in the consequential-decision context, disclose:^[8]

• The role the ADMT played in the consequential decision
• The principal categories of personal data processed and how they informed the outcome
• The consumer’s rights, including how to exercise them
• How to request correction of inaccurate personal data and, where commercially reasonable, meaningful human review

This disclosure is generally triggered within 30 days of an adverse consequential decision, alongside a clear-and-conspicuous pre-use notice. Note that SB 26-189 dropped several duties that the 2024 Act had carried: the reasonable-care duty against algorithmic discrimination (now addressed under existing Colorado anti-discrimination law), mandatory algorithmic impact assessments, the small-deployer size exemption, and the NIST AI RMF / ISO 42001 rebuttable-presumption safe harbor. Those frameworks remain sound practice for documenting and explaining decision logic, but Colorado law no longer treats them as a codified affirmative defense. Exact contours of “materially influence” await Attorney General rulemaking due by January 1, 2027.^[8]

Explainability by Use Case

Different domains have different explainability requirements driven by regulatory context, risk levels, and user needs.

Healthcare and Clinical AI

Healthcare represents the highest-stakes domain for explainability. Physicians need to understand AI recommendations to maintain clinical accountability, and regulators require transparency for medical device approval.

Common techniques: Saliency maps for medical imaging (showing which pixels influenced a diagnosis), SHAP for predictive models (identifying risk factors), attention visualizations for clinical notes, counterfactuals for treatment recommendations.

Example: Google’s diabetic retinopathy detection system uses attention maps to highlight specific retinal regions (hemorrhages, exudates) that drove the diagnosis, enabling ophthalmologists to validate the AI’s reasoning.^[19]

Financial Services and Credit Decisions

Financial services face the most mature explainability requirements due to decades of anti-discrimination regulation.

Common techniques: Counterfactual explanations (“if your income were $X higher, you would qualify”), feature importance rankings (showing top factors in credit decisions), adverse action reason codes, SHAP values for individual decisions.

Challenge: Balancing explainability with model performance. Simpler, more interpretable models may have lower predictive accuracy. Many banks use “challenger models”, which are interpretable models that validate complex model decisions.^[20]

HR and Employment Decisions

AI hiring and HR systems face increasing scrutiny following discrimination lawsuits and regulatory investigations.

Regulatory context: The EU AI Act classifies employment AI as high-risk. NYC Local Law 144 (2023) requires bias audits for automated employment decision tools. Illinois Artificial Intelligence Video Interview Act requires notice and consent for AI-analyzed video interviews.^[22]

Common techniques: Feature importance to identify resume/assessment factors, bias testing across demographic groups, counterfactuals for candidate feedback, LIME for explaining why specific candidates were recommended or rejected.

Legal Services and Contract Analysis

Legal AI faces unique explainability requirements because attorneys maintain professional liability for work product, including AI-assisted analysis.

For legal research, contract analysis, and document review, useful transparency includes source tracing, quoted context, confidence limits, version history, and a clear boundary between retrieved material and generated analysis. These artifacts support review but do not make an output legally correct.

Common techniques: Citation tracing (showing source documents), attention visualization (highlighting contract clauses that drove extraction results), confidence scoring with explanations for low-confidence outputs, version tracking showing how AI suggestions evolved.

Implementation Framework

Building explainable AI systems requires intentional design, not post-deployment retrofitting. Here’s a practical framework:

Phase 1: Requirements Definition

Phase 2: Technique Selection

Choose explainability techniques based on your requirements:

Phase 3: Validation and Testing

Explainability techniques can be wrong. Validate explanations through:

• Sanity checks: Do explanations align with domain knowledge? If a medical model highlights random pixels, it’s likely wrong.
• Consistency testing: Do similar inputs produce similar explanations? Inconsistent explanations suggest instability.
• Ablation studies: Remove features identified as important. Does performance drop as predicted?
• Expert review: Have domain experts review explanations for a sample of decisions. Do they agree?

Phase 4: Documentation and Monitoring

Document your explainability approach in model cards and system documentation. Monitor explanation quality over time. Model drift can cause explanation drift.

Implementation checklist

Explainability workstreams to assess

Glacis does not infer or validate a foundation model’s internal reasoning. These workstreams belong to the deployer’s separate explainability program; signed operational records may preserve covered claims or artifact identifiers, not establish that an explanation is faithful.

1

Define Stakeholder Needs

Map who needs explanations (regulators, users, auditors, internal teams) and what type (global, local, counterfactual). Applicable requirements and the decision context inform what is needed.

2

Choose Techniques Early

Select explainability techniques during model design, not after deployment. Constraints like real-time explanations or specific regulatory formats influence technique selection and model architecture.

3

Validate Explanation Quality

Test explanations with domain experts. Check consistency, sanity, and alignment with ground truth. Bad explanations are worse than no explanations. They create false confidence.

4

Document and Monitor

Create model cards documenting explainability approach, validation results, and known limitations. Monitor explanation drift as models and data evolve. Archive explanations for audit trails.

Key principle: Explainability is not a feature you add at the end. It’s a design constraint that shapes model selection, feature engineering, and deployment architecture from day one.

Communicating Explanations

Technical explainability methods produce outputs like SHAP values, attention weights, and feature importance scores. Translating these into meaningful explanations for different audiences is as important as generating them.

Explanations for Technical Audiences

Data scientists and ML engineers benefit from detailed technical explanations:

• SHAP value distributions and dependence plots
• Feature importance rankings with confidence intervals
• Partial dependence plots showing feature relationships
• Model architecture diagrams and training metrics

Explanations for Non-Technical Users

End users need simple, actionable explanations in natural language:

Explanations for Regulators and Auditors

Compliance audiences need evidence that explanations are accurate, complete, and non-discriminatory:

• Model cards with transparency documentation
• Fairness metrics across demographic groups
• Validation evidence showing explanation accuracy
• Audit trails showing how explanations were generated and verified

Tools & Platforms

The explainability tooling ecosystem has matured significantly. Here are the leading open-source and commercial options:

Open Source Tools

Commercial Platforms

Enterprise AI governance platforms increasingly include built-in explainability capabilities:

• IBM watsonx.governance: Model explanation dashboards, fairness metrics, automated documentation generation
• Credo AI: Explainability assessment as part of AI governance workflows, alignment with regulatory requirements
• Holistic AI: Bias and explainability testing integrated with compliance automation
• Arize AI: Model monitoring and observability with built-in explainability for production systems

Frequently Asked Questions

References