AI Supply Chain Security Guide

What is AI Supply Chain Security?

AI supply chain security extends traditional software supply chain security to encompass AI-specific components: training datasets, pre-trained models, fine-tuning data, evaluation benchmarks, and model deployment infrastructure. While traditional supply chains track code dependencies, AI supply chains must also track data lineage, model provenance, and computational resources.

The AI Attack Surface

AI systems introduce attack surfaces that don't exist in traditional software:

• Data poisoning: Attackers inject malicious examples into training data, causing models to learn backdoors or biases
• Model poisoning: Pre-trained models from repositories like Hugging Face contain embedded backdoors
• Dependency confusion: ML-specific packages on PyPI, conda, npm targeted with typosquatting and malicious uploads
• Provenance attacks: Models and datasets with falsified origins or licensing claims

The complexity stems from AI's multi-layered supply chain. A single deployed model might depend on a base model from Hugging Face, fine-tuning data scraped from the web, Python packages from PyPI, CUDA libraries from NVIDIA, and cloud infrastructure from AWS—each representing a potential compromise point.

The AI Supply Chain

Understanding the AI supply chain requires mapping dependencies across five distinct layers:

Each layer creates dependencies that must be tracked, verified, and monitored. A study of 1,000 AI projects found the median project has 103 dependencies with 16 known CVEs—significantly higher than traditional software projects.^[1]

Attack Vectors in AI Supply Chains

AI supply chain attacks exploit trust relationships and opacity in the ML development process. Here are the primary attack vectors documented in the wild:

1. Model Poisoning

Attackers distribute pre-trained models containing backdoors that activate on specific triggers. These attacks are particularly insidious because:

• Models are large binary files that can't be easily code-reviewed
• Backdoors can be embedded in ways that don't affect normal performance
• Detection requires specialized testing on trigger patterns
• Organizations rarely verify model provenance beyond "it's from Hugging Face"

2. Dependency Confusion & Typosquatting

ML developers rely heavily on package managers like PyPI and conda. Attackers exploit this through:

• Typosquatting: Registering packages with names similar to popular libraries (e.g., "torch" vs "pytorch-torch")
• Dependency confusion: Uploading malicious packages with identical names to internal packages
• Malicious updates: Compromising legitimate package maintainer accounts

Research by Checkmarx identified over 10,000 malicious packages on PyPI specifically targeting ML developers, with names mimicking TensorFlow, PyTorch, and popular transformer libraries.^[3]

3. Data Poisoning

Training data poisoning targets the datasets used to train or fine-tune models. Attack methods include:

• Injecting mislabeled examples that degrade model performance on specific inputs
• Contributing poisoned samples to open datasets (Common Crawl, LAION-5B)
• Manipulating web content that gets scraped into training corpora
• Exploiting user-generated content in fine-tuning pipelines

Data poisoning is particularly effective because organizations rarely verify provenance of training data, and detecting poisoned samples requires knowing what to look for.

4. Model Repository Compromise

Platforms like Hugging Face host hundreds of thousands of models. Security risks include:

• Malicious pickle files in model checkpoints (pickle deserialization = arbitrary code execution)
• Models claiming to be one architecture while actually being another
• Falsified model cards and evaluation metrics
• License washing (commercial models redistributed as open source)

Hugging Face has implemented security features including pickle scanning and model signing, but 80% of organizations using third-party models don't verify signatures or scan for malicious code.^[4]

Data Supply Chain Risks

Training data represents a critical attack surface because it directly shapes model behavior. Key risks:

Data Provenance Challenges

Most AI teams cannot answer basic provenance questions about their training data:

• Where did each training example originate?
• Who labeled or annotated the data?
• What transformations or filtering were applied?
• What is the licensing status of source material?
• When was the data collected and is it still valid?

The EU AI Act Article 10 requires "data governance and management practices" including provenance tracking for high-risk systems. Organizations without data lineage capabilities will struggle to demonstrate compliance.^[8]

Third-Party Dataset Risks

Popular datasets like ImageNet, Common Crawl, and LAION face ongoing security and compliance concerns:

• Data poisoning: Malicious actors contributing samples to open datasets
• Copyright infringement: Datasets containing copyrighted material without license
• Privacy violations: Personal data scraped without consent (LAION-5B removed after privacy complaints)
• Bias and representation: Systematic over/under-representation of demographic groups

Web Scraping Provenance

Many foundation models train on web-scraped data (Common Crawl, C4, The Pile). This creates provenance challenges:

• No ability to verify individual web pages weren't tampered with
• Scraped content may contain adversarially-placed poisoning examples
• Legal status unclear (ongoing litigation against OpenAI, Meta, Stability AI)
• No mechanism to remove or update training data post-scraping

Model Supply Chain Risks

Pre-trained models have become infrastructure for modern AI. Over 500,000 models are hosted on Hugging Face alone, with billions of downloads monthly. This centralization creates systemic risk.

Pre-trained Model Risks

Hugging Face Security Features

Hugging Face has implemented several security controls, but adoption is inconsistent:

• Pickle scanning: Automated scanning for malicious pickle files (but can be bypassed)
• Malware scanning: ClamAV scanning of uploaded files
• Model signing: Optional GPG signatures for model verification (low adoption)
• Security reporting: Vulnerability disclosure program

Despite these features, research shows only 3% of downloaded models have verified signatures, and most organizations don't perform local security scanning before deployment.^[9]

Fine-tuning Attack Vectors

Even when starting with a trusted base model, fine-tuning introduces risks:

• Fine-tuning data poisoning (injecting backdoor triggers during fine-tuning)
• Parameter extraction attacks (fine-tuning to extract training data from base model)
• License violations (fine-tuning non-commercial models for commercial use)
• Drift from safety alignment (fine-tuning removing safety controls from base model)

Software Supply Chain Risks

AI projects depend on complex software stacks spanning ML frameworks, data processing libraries, and deployment tools. Each dependency represents potential compromise.

The ML Dependency Problem

Analysis of 1,000 AI projects on GitHub revealed:

• Median of 103 transitive dependencies
• Median of 16 known CVEs in dependency tree
• 97% had at least one vulnerable dependency
• 68% pinned zero dependencies (exposing to supply chain attacks)

This is significantly worse than traditional software. AI projects average 40% more dependencies and 2.3x more vulnerabilities than non-AI projects of similar size.^[1]

PyPI Package Ecosystem Risks

PyPI hosts over 500,000 packages, but has minimal security controls:

• No code signing requirements
• No mandatory 2FA for package maintainers (until 2023)
• No automated malware scanning of uploads
• Easy to register typosquat packages

Checkmarx research documented over 10,000 malicious packages targeting ML developers with techniques including:

• Typosquatting popular packages (tensorflow-gpu, pytorch-cuda, transformers-dev)
• Dependency confusion (uploading packages with internal corporate names)
• Compromised maintainer accounts
• Malicious code in setup.py executing during pip install

Critical ML Library Vulnerabilities

Dependency Pinning Failure

Most ML projects use unpinned dependencies (e.g., transformers>=4.0 instead of transformers==4.36.1). This means:

• Builds are non-reproducible
• Malicious package updates auto-install
• Breaking changes cause production failures
• Impossible to track exact dependency versions for compliance

Research shows 68% of AI projects have zero pinned dependencies, despite this being a basic supply chain security control.^[1]

Infrastructure Supply Chain Risks

AI infrastructure dependencies create additional attack surfaces:

Cloud Provider Risks

• Metadata exfiltration: Cloud APIs exposing model weights, training data, or credentials
• Shared infrastructure: GPU sharing enabling side-channel attacks
• Access control failures: Misconfigured S3 buckets exposing model checkpoints
• Supply chain concentration: Heavy reliance on AWS/GCP/Azure creates single points of failure

GPU Supply Chain

The AI industry's dependence on NVIDIA GPUs creates supply chain concentration:

• NVIDIA controls >95% of AI GPU market
• CUDA software is proprietary and closed-source
• No practical alternatives for training large models
• Supply constraints enable counterfeiting and gray-market hardware

Model Hosting Services

Third-party model hosting (Replicate, Together AI, Anyscale) introduces risks:

• No visibility into infrastructure security controls
• Model weights stored on third-party infrastructure
• Inference data potentially logged or used for training
• Service outages impacting production AI systems

AI SBOM and ML-BOM

Traditional Software Bills of Materials (SBOMs) don't capture AI-specific components. The industry is developing two complementary approaches:

AI SBOM: Extending Traditional SBOM

AI SBOM extends formats like SPDX and CycloneDX to include AI components:

• ML framework dependencies (PyTorch, TensorFlow versions)
• Model files as software components
• Data processing pipeline dependencies
• Training and inference infrastructure

The Linux Foundation's SPDX 3.0 specification (released 2024) includes AI/ML extensions, providing standardized fields for model metadata and dataset references.^[10]

ML-BOM: Machine Learning Bill of Materials

ML-BOM is a specialized format capturing ML-specific supply chain information:

Organizations like Google and Microsoft are developing internal ML-BOM tooling, but no industry-wide standard exists yet. NIST is working on ML-specific SBOM guidance as part of the AI Safety Institute.^[11]

Implementing ML-BOM

Practical ML-BOM implementation requires:

• Automated extraction: Tools that generate ML-BOM from model artifacts and training pipelines
• Cryptographic hashing: SHA-256 hashes of all model weights and datasets
• Provenance tracking: Chain of custody from data collection through deployment
• Version control: ML-BOM versioned alongside model versions
• Signing: Digital signatures on ML-BOM for tamper detection

Secure Model Sourcing

Organizations deploying third-party models should implement verification controls:

Model Verification Checklist

Model Provenance Tracking

Implementing model provenance requires tracking:

• Source: Original repository, organization, download URL
• Integrity: SHA-256 hash of model weights
• Signature: GPG signature verification if available
• Lineage: Base model → fine-tuned versions → deployed versions
• Modifications: All fine-tuning, quantization, or pruning operations
• Evaluation: Benchmark results, bias testing, safety evaluations

Model Signing

Organizations should implement model signing for internal models:

• GPG sign model artifacts before uploading to internal registries
• Verify signatures before deployment
• Maintain key rotation and revocation procedures
• Log all signature verification attempts

Hugging Face supports model signing via GPG, but adoption remains low. Organizations should sign internal models even if third-party models lack signatures.^[12]

Dependency Management for AI Projects

Securing the software supply chain requires rigorous dependency management:

Dependency Scanning

Implement automated scanning for:

• Known CVEs: Use tools like pip-audit, safety, or Snyk
• Malicious packages: Detect typosquatting, dependency confusion
• License compliance: Track GPL, commercial licenses in dependencies
• Unmaintained packages: Flag dependencies with no recent updates

Dependency Pinning Best Practices

# Bad - allows any version >= 2.0
torch>=2.0.0
transformers>=4.0.0

# Good - pins exact versions
torch==2.1.2
transformers==4.36.1
tokenizers==0.15.0

Dependency Auditing Workflow

Establish a regular auditing workflow:

1. Weekly scans: Automated scanning for new CVEs in dependencies
2. Monthly reviews: Manual review of dependency update recommendations
3. Quarterly upgrades: Coordinated dependency upgrades with testing
4. Incident response: Emergency process for critical vulnerabilities

Private Package Repositories

For production systems, consider using private package mirrors:

• Mirror PyPI packages to internal Artifactory/Nexus repository
• Security scan all packages before adding to internal mirror
• Prevent direct PyPI access from production environments
• Maintain allow-list of approved packages and versions

Regulatory Requirements

Supply chain security is increasingly required by regulation and industry standards:

EU AI Act Supply Chain Requirements

The EU AI Act Article 11 requires providers of high-risk AI systems to:

• Implement "data governance and management practices" including provenance tracking
• Document training, validation, and testing datasets
• Maintain technical documentation on data sources and preprocessing
• Ensure transparency of AI system supply chains

High-risk provisions take effect August 2026—organizations without supply chain documentation will face non-compliance.^[8]

NIST Secure Software Development Framework (SSDF)

NIST SP 800-218 SSDF explicitly includes AI/ML in scope. Practice PO.3.2 requires:

"Create an inventory of the software's components, including commercial software, open-source software, in-house software, and AI/ML components."

Federal contractors and enterprises selling to government must demonstrate SSDF compliance, which now includes AI supply chain transparency.^[13]

Executive Order 14110 Requirements

EO 14110 (October 2023) requires developers of foundation models to:

• Report safety test results to the federal government
• Share information about cybersecurity measures
• Provide transparency into model provenance and training data
• Implement measures to detect AI-generated content

While primarily aimed at frontier model developers, the transparency requirements establish precedent for supply chain documentation across the AI industry.^[14]

Implementation Checklist

Organizations should implement supply chain security controls across six areas:

GLACIS Supply Chain Security Framework

Frequently Asked Questions

References