Shadow AI Governance Checklist

Governance is the step after detection and classification. This checklist walks you through the process of taking a shadow AI system you've discovered and making it compliant with EU AI Act and operational best practices.

The Governance Lifecycle

This checklist covers 8 phases:

1. Discovery & Assessment, Document the shadow AI system
2. Classification & Legal, Determine its risk level
3. Governance Implementation, Build audit trail, policy, enforcement
4. Performance Monitoring, Test for bias and drift
5. Compliance Documentation, Prepare for regulation
6. Regulatory Readiness, Get sign-off and incident response ready
7. Ongoing Compliance, Monthly/quarterly/annual reviews
8. Continuous Improvement, Share learnings across organization

Timeline: 4-6 months from discovery to compliance-ready.

Phase 1: Discovery & Initial Assessment

Step 1.1: Document the Shadow AI System

Create a system record with:

• System name/description
• Owner/responsible team
• When discovered (date)
• Detection method (DNS logs, firewall, employee report)
• Date of first deployment (if known)

Step 1.2: Understand System Purpose & Scope

• What does this system do?
• What decisions does it make?
• How many employees/customers use it?
• How frequently is it used?
• What data does it process? (PII? Health? Financial?)

Step 1.3: Preliminary Risk Screening

• Does it process personal data?
• Does it make automated decisions affecting individuals?
• Is it in a regulated industry?
• Could it cause harm if it fails?

Phase 2: Classification & Legal Assessment

Step 2.1: Formal Classification

Use the AI Classification Guide (see related guides) to determine:

• Is it prohibited? (If yes, discontinue immediately)
• Is it High-Risk? (If yes, full governance required)
• Is it General-Purpose? (Limited transparency obligations)
• Is it Low-Risk? (Standard data protection)

Step 2.2: Substantial Modification Assessment

Check if your governance approach could accidentally make you the provider:

• Will you modify prompts? Will you filter outputs?
• Will you change model weights? Will you add training data?
• If all are NO → Safe to proceed as deployer
• If any are YES → Risk becoming provider; consult legal

Phase 3: Governance Implementation

Step 3.1: Audit Trail Setup

Every decision the AI makes must be logged with context.

• Audit trail system selected & deployed
• Captures: Decision / Timestamp / User / Authorization / Output
• Logs are immutable (cannot be deleted/modified retroactively)
• Logs are cryptographically signed (tamper-proof)
• Logs retained for minimum 3 years
• Access control implemented (only scoped users can view)
• All log access is itself logged
• Regular backup of logs (offsite, immutable)

Step 3.2: Policy & Authorization Framework

Define what the AI system is scoped to do.

• Policy document created (intended scope, decision limits)
• Escalation thresholds defined (when to escalate to human)
• Exception procedures documented
• Signed by business owner & CISO
• Policy is machine-readable (enforcement system can parse it)
• Policy version control implemented
• Policy cryptographically signed (hash included in every log entry)

Step 3.3: Enforcement System

Implement controls to prevent the AI from violating its policy.

• Enforcement layer deployed
• System decisions routed through enforcement layer
• Enforcement layer intercepts decisions before execution
• Can be monitored without enforcement (audit-only mode)
• Easy rollback if enforcement causes problems
• Enforcement rules tested (10+ violation scenarios)
• All violations correctly blocked/flagged
• Performance impact measured & acceptable

Step 3.4: Human Oversight Mechanism

• Human review process defined (when review is required, by whom)
• Review SLA specified (how quickly must review happen)
• Review documentation requirements established
• Appeal/override process defined
• Reviewers trained on system scope, bias detection, escalation
• Review system monitored (time to review, override rate, feedback captured)

Phase 4: Performance Monitoring & Bias Testing

Step 4.1: Baseline Performance Metrics

• Accuracy: ____% (measured on validation set)
• Precision: ____% (false positive rate)
• Recall: ____% (false negative rate)
• Latency: ____ ms
• Uptime target: ____%
• Initial performance measured and documented
• Comparison to human baseline (if applicable)
• Acceptable performance thresholds defined

Step 4.2: Bias & Fairness Testing

• Protected characteristics identified (gender, race, age, disability, socioeconomic)
• Test scenarios designed for each characteristic
• Fairness metrics defined (e.g., Disparate Impact Ratio)
• Initial bias testing completed (500+ samples with known traces)
• Results disaggregated by protected group
• Discrimination indicators measured
• If bias found, mitigation strategy created
• Quarterly bias testing scheduled
• Production data monitored for group disparities

Step 4.3: Performance Drift Detection

• Performance monitoring dashboard created
• Tracks accuracy, precision, recall over time
• Alerts if metrics drop below thresholds
• Data drift detection implemented
• Incident response process defined (detect → investigate → remediate)

Phase 5: Compliance & Documentation

Step 5.1: Technical Documentation Package

• System description (what it does, how it works)
• Training data documentation
• Model architecture & performance metrics
• Risk assessment (what can go wrong)
• Mitigation measures documented
• Testing & validation results
• Performance monitoring process
• Human oversight process
• Risk management plan (harms, probability, severity, mitigation)

Step 5.2: User Notification & Transparency

• Users notified of automated decision
• Clear notice: "An AI system was used to make this decision"
• Provided at decision point (not buried in T&Cs)
• Explanation available on request (plain language)
• Explanation includes key factors in decision

Step 5.3: Appeals & Rights Process

• Clear steps to appeal a decision
• Escalation to human review
• Timeline for response: _____ days
• Right to appeal documented
• No penalty for appealing
• All appeals logged
• Appeal outcomes tracked

Phase 6: Regulatory Readiness

Pre-Inspection Checklist

• Technical documentation complete & accessible
• Risk assessment signed
• Performance metrics & bias testing results current
• Audit trail complete & sample entries provided
• Query examples demonstrated (show decisions by date, violations, metrics)
• Log integrity verified (tamper-evident)
• Latest bias testing completed within 90 days
• No evidence of discrimination (or mitigation documented)
• Legal review completed
• Compliance officer sign-off obtained

Incident Response Plan

• Definition of "incident" (e.g., High-Risk decision outside policy)
• Notification procedures (who to contact)
• Investigation process
• Remediation process (how to fix affected decisions)
• Regulator notification process (if required)
• Key contacts identified & escalation path defined

Phase 7: Ongoing Compliance & Monitoring

Monthly Monitoring

• Performance metrics reviewed
• Alert logs reviewed (concerning patterns?)
• Audit trail integrity verified
• Issues reported to leadership

Quarterly Assessment

• Bias testing completed
• Performance drift analysis
• Policy violations reviewed
• Appeals/override patterns analyzed
• Compliance status reported to leadership

Annual Comprehensive Review

• Full bias testing (all protected characteristics)
• Performance benchmarking
• Documentation updated
• Risk assessment revisited
• Classification re-validated (still High-Risk?)
• Legal/compliance sign-off renewed

Phase 8: Continuous Improvement

• Document governance experience (what worked, what was hard)
• Create playbook/runbook for other High-Risk systems
• Train other system owners on compliance process
• Share best practices across organization