EU AI Act gap assessment: the 12-point self-check before your conformity assessment

Before engaging a conformity assessment body (CAB) to validate your AI system against the EU AI Act, a meticulous gap assessment is essential. This self-diagnostic ensures alignment with legal requirements before incurring assessment costs, mitigating risks of non-compliance or costly corrections. Unlike generic compliance frameworks, the EU AI Act’s tiered risk-based approach demands granular scrutiny—particularly for high-risk AI systems. Below is a structured 12-point self-check, designed to identify operational gaps, documentable evidence, and a scoring methodology to prioritize remediation efforts.

1. Risk Classification and Categorization

The EU AI Act categorizes AI systems into four risk classes (Unacceptable, High, Medium, Low) based on core functionality, impact on human rights, and societal risks. A preliminary assessment must determine whether your system falls under High-Risk (e.g., hiring, law enforcement, critical infrastructure) or Medium/Low-Risk (e.g., customer service chatbots, basic predictive analytics). Evidence required: A functional specification document (or equivalent) outlining:

1. Primary purpose and intended use cases
2. Data sources and processing methods
3. Potential adverse impacts (e.g., bias, transparency, accountability)
4. Regulatory triggers (e.g., EU-wide deployment, sensitive data)

2. Technical Compliance with Annex II

High-Risk AI systems must comply with Annex II’s technical requirements, which include transparency, human oversight, and robustness. Key sub-requirements include:

• Transparency: Users must be informed about AI’s existence and its role (e.g., "This response is generated by an AI system").
• Human Oversight: A human must be able to intervene at any stage, with records of such interventions.
• Robustness and Performance: AI must not introduce significant risks (e.g., hallucinations, bias, failure modes).
• Data Quality: Input data must be accurate, relevant, and free from manipulation.

Evidence required: A technical compliance log detailing:

1. Transparency notices deployed across all user interfaces
2. Processes for human intervention (e.g., escalation paths, documentation templates)
3. Robustness testing results (e.g., failure rate benchmarks, bias audits)
4. Data validation protocols (e.g., sample records of input/output validation)

3. Data Ethics and Bias Mitigation

The EU AI Act prohibits AI systems that rely on discriminatory or harmful data (e.g., racial, gender, or disability-based datasets). For High-Risk systems, additional safeguards apply, such as:

• Dataset Audits: Third-party bias assessments must be conducted on training and deployment data.
• Diversity in Training Data: Data should reflect diverse populations to avoid underrepresentation.
• Continuous Monitoring: Bias flags must be tracked and addressed in real-time.

Evidence required:

1. Bias audit reports (e.g., from a certified auditor) with performance metrics (e.g., error rates by demographic)
2. Dataset diversity metrics (e.g., percentage representation of underrepresented groups)
3. Bias monitoring logs (e.g., alerts triggered, corrective actions taken)
4. Policy on data source validation (e.g., third-party verification, provenance checks)

4. Documentation and Record-Keeping

Compliance with the EU AI Act requires detailed documentation to demonstrate adherence to legal obligations. For High-Risk systems, this includes:

• System Design Documentation: Architectural diagrams, risk assessments, and technical specifications.
• Training and Testing Records: Proof of validation (e.g., performance benchmarks, user testing reports).
• Incident Logs: Records of non-compliance events, corrective actions, and follow-ups.
• User Interface Transparency: Screenshots or logs of transparency notices deployed.

Evidence required: A structured compliance dossier (e.g., in a shared folder or database) containing:

1. System architecture diagrams
2. Risk assessment reports (e.g., ISO 21434 alignment)
3. Validation test results (e.g., accuracy, fairness benchmarks)
4. Incident response templates and audit trails
5. Transparency notice examples (e.g., for different user roles)

5. Human Rights and Societal Impact

High-Risk AI systems must demonstrate no adverse impact on human rights (e.g., autonomy, privacy, equality). This includes:

• Impact Assessments: A formal risk assessment identifying potential harms (e.g., job displacement, surveillance risks).
• Mitigation Strategies: Proposed safeguards to address identified risks (e.g., fairness audits, user consent mechanisms).
• Stakeholder Engagement: Feedback from affected communities (e.g., employees, customers, civil society).

Evidence required:

1. Risk assessment report with human rights impact analysis
2. Mitigation plan (e.g., with timelines for implementation)
3. Stakeholder feedback logs (e.g., surveys, interviews)
4. Policy on human rights compliance (e.g., internal governance structure)

6. Third-Party Dependencies

If your AI system relies on third-party components (e.g., cloud providers, libraries, or APIs), these must also comply with the EU AI Act. Key considerations include:

• Vendor Compliance: Assess whether third-party providers meet EU AI Act obligations (e.g., transparency, risk management).
• Data Transfer Risks: Ensure no sensitive data is transferred outside the EU without safeguards (e.g., Standard Contractual Clauses).
• Audit Trails: Maintain records of third-party interactions (e.g., API calls, data flows).

Evidence required:

1. Vendor compliance agreements (e.g., SLA clauses on AI Act alignment)
2. Data transfer documentation (e.g., proof of EU-wide data processing)
3. Audit logs of third-party dependencies (e.g., usage statistics, error reports)
4. Policy on third-party risk assessment (e.g., regular reviews)

7. User Rights and Accountability

High-Risk AI systems must provide users with clear rights and recourse mechanisms, including:

• Explanations: Users must receive understandable explanations for AI-generated outputs.
• Access to Data: Users must be able to request their data (e.g., via a privacy portal).
• Recourse Mechanisms: Users must have a way to appeal AI decisions (e.g., human review process).
• Transparency Notices: Clear disclosure of AI’s role in decision-making.

Evidence required:

1. User rights documentation (e.g., FAQs, help center articles)
2. Data access procedures (e.g., request templates, processing steps)
3. Recourse pathways (e.g., escalation forms, human review workflows)
4. Transparency notice examples (e.g., for different user types)

8. Continuous Monitoring and Improvement

The EU AI Act requires ongoing monitoring to detect and address emerging risks. Key activities include:

• Performance Monitoring: Real-time tracking of AI behavior (e.g., error rates, bias flags).
• Incident Response: Protocols for detecting and mitigating non-compliance events.
• Regular Audits: Scheduled reviews of AI performance and compliance.
• Feedback Loops: Mechanisms for users to report issues (e.g., bug reports, compliance alerts).

Evidence required:

1. Monitoring dashboards (e.g., with alerts for anomalies)
2. Incident response playbooks (e.g., with escalation paths)
3. Audit schedules (e.g., quarterly review templates)
4. User feedback channels (e.g., ticketing system logs)

9. Technical Documentation for Conformity Assessment

If you proceed to a conformity assessment, you must provide specific technical documentation to the CAB. Key requirements include:

• System Design: Detailed architecture diagrams and component specifications.
• Risk Assessment: Formal justification of why the system meets Annex II requirements.
• Validation Evidence: Proof of testing (e.g., performance benchmarks, bias audits).
• Compliance Logs: Records of all transparency notices, human intervention logs, and incident responses.

Evidence required:

1. Technical specifications (e.g., API contracts, data flow diagrams)
2. Risk assessment report (e.g., with Annex II alignment)
3. Validation test reports (e.g., with pass/fail criteria)
4. Compliance logs (e.g., with timestamps and outcomes)

10. Legal and Regulatory Compliance

The EU AI Act requires specific legal obligations, such as:

• Registration: High-Risk systems must be registered in the EU AI Act Registry.
• Public Communication: Clear labeling of AI systems (e.g., "AI-powered" vs. "human-generated").
• Data Protection: Compliance with GDPR (e.g., consent management, data minimization).
• Liability Framework: Clarity on accountability (e.g., who is responsible for AI errors).

Evidence required:

1. Registry submission (e.g., proof of registration with EU AI Act portal)
2. Labeling policy (e.g., with examples of user-facing disclosures)
3. GDPR compliance reports (e.g., with data processing impact assessments)
4. Liability policy (e.g., with risk-sharing agreements)

11. Post-Compliance Monitoring

Even after assessment, continuous monitoring is mandatory. Key activities include:

• Periodic Audits: Quarterly or annual reviews of AI performance.
• Incident Reporting: Systems for tracking and addressing non-compliance.
• User Feedback: Mechanisms for collecting and acting on user reports.
• Technical Updates: Regular reviews of AI models and dependencies.

Evidence required:

1. Audit schedules (e.g., with review dates and responsibilities)
2. Incident response logs (e.g., with corrective actions)
3. User feedback templates (e.g., with response workflows)
4. Technical update logs (e.g., with model versioning and validation)

12. Third-Party Certification (Where Applicable)

For High-Risk AI systems, third-party certification may be required to demonstrate compliance. Key considerations include:

• Certification Body Selection: Choose a CAB accredited by the EU (e.g., via EAC or national regulators).
• Certification Process: Follow the CAB’s procedures for technical and administrative assessments.
• Evidence Submission: Provide all required documentation (e.g., technical specs, risk assessments).
• Certification Decision: Ensure the CAB issues a compliance certificate (or refusal).

Evidence required:

1. List of accredited CABs (e.g., from EU AI Act Registry)
2. Certification contract (e.g., with scope and timeline)
3. Submitted documentation (e.g., proof of compliance review)
4. Certification outcome (e.g., compliance declaration)

Scoring Summary: Multiply each checkpoint’s score by its weight (e.g., Risk Classification = 4, Technical Compliance = 3) to generate a total compliance score. A score above 60% suggests readiness for a formal assessment; scores below 40% indicate critical gaps requiring remediation. Action: Prioritize remediation efforts based on the highest-scoring gaps, ensuring alignment with the EU AI Act’s risk-based approach before engaging a CAB. This proactive approach minimizes costs and delays, while also building a robust compliance framework for ongoing operations.