EU AI Act Compliance for High-Risk AI Systems: What Your Organization Needs to Know

A Q&A about the EU AI Act with Schellman CEO Avani Desai on risk classification, AI literacy, and the August 2 deadline

Schellman CEO, Avani Desai, recently joined a DataCamp panel webinar, "Deadline Approaching: EU AI Act Compliance for High-Risk AI Systems", alongside Jessica Eaves Mathews (Managing Attorney at Leverage Legal Group) and Jason M. Loring (Partner at Jones Walker, co-chair of its Privacy, Data Strategy and AI team).

The panel broke down what the EU AI Act requires, how risk classification works in practice, and what organizations that are still behind on compliance should prioritize first.

Below is an edited Q&A drawn from Avani's perspective, covering what the Act actually requires, why generative and agentic AI raise the stakes, and what organizations should be doing right now, including for the Article 50 transparency obligations that took effect on August 2.

Q: Why does AI need to be regulated?

A: The real issue is that AI has moved from generating content to influencing decisions and increasingly taking actions. Once a model is embedded in hiring, lending, healthcare, cybersecurity, or critical infrastructure, a model failure can become a real operational, financial, or human-impact event.

Technically, AI systems are probabilistic. Their behavior can change based on the prompt, context window, retrieval source, model version, system instructions, tools, or downstream integrations. A model can perform well in a controlled benchmark but behave differently in production because the data distribution changes, or because users interact with it in ways the developer never anticipated.

With agentic systems, the risk expands further as the model is no longer just producing text. It may have identity, permissions, memory, API access, and the ability to execute transactions. That widens the attack surface to include prompt injection, data poisoning, retrieval manipulation, insecure tool use, exclusive agency, and privilege escalation.

Regulation is the accountability layer. It forces organizations to define intended purpose, test foreseeable failure modes, document limitations, maintain human oversight, monitor production behavior, and establish responsibility when something goes wrong. The EU AI Act is fundamentally trying to balance innovation with protection of health, safety, and fundamental rights.

Q: At a high level, what does the EU AI Act cover?

A: The EU AI Act regulates the lifecycle of an AI system, covering how it's developed, trained, validated, placed on the market, integrated, deployed, monitored, and changed. It also allocates obligations across the supply chain, so responsibilities differ depending on whether you're the provider, deployer, importer, distributor, product manufacturer, or downstream integrator.

A company may say, "We didn't build the foundation model," but integrating it into a recruitment platform or materially modifying it can bring on additional responsibilities regardless.

The Act addresses prohibited practices, high-risk systems, transparency requirements, general-purpose AI models, conformity assessment, technical documentation, post-market monitoring, incident reporting, market surveillance, and enforcement. This is much closer to a product assurance and operational-governance regime than a simple privacy regulation.

Q: Are there industries particularly affected by the EU AI Act, and what does the risk classification system involve?

A: In addition to industry, decision types should be considered. The most affected use cases are systems that materially influence a person's rights, safety, livelihood, or access to an essential service, such as hiring, employee monitoring, education admissions, credit decisions, insurance, healthcare, biometric identification, critical infrastructure, law enforcement, immigration, and judicial processes.

Financial services, healthcare, employment, and critical infrastructure will likely feel this most directly, since those environments already require traceability, validation, controlled change, access management, and defensible decisions. The AI Act adds AI-specific requirements on top of those existing control environments.

The key technical question is: what is the model allowed to influence or execute? A generative AI assistant summarizing an internal meeting may be relatively low risk. The same model connected to an HR system and permitted to rank candidates becomes a very different system. The underlying foundation model may be identical, but the risk changes because of the data, decision logic, user population, level of automation, and consequence of error.

That question is exactly what the Act's risk classification system is built around. Classification is based on both the intended purpose and the context of use.

• Prohibited practices: uses the EU considers fundamentally incompatible with its values such as certain manipulative techniques, social scoring, exploitation of vulnerabilities, and particular biometric or emotion-recognition uses.

• High-risk AI: permitted, but requiring a much stronger control environment because they affect safety or fundamental rights. Examples include employment decisions, education, access to credit or essential services, certain biometric applications, law enforcement, migration, and justice.
  
• Transparency-obligation systems: may not require the full high-risk control stack, but users must be told when they're interacting with AI or when content has been generated or manipulated by AI.
  
• Minimal risk: most ordinary business applications.
  

Classification can't be a one-time label attached to a model. It has to be tied to a documented system boundary: What model is used? What data is retrieved? What tools can it call? What decisions can it influence? Is there human review? Has the intended purpose changed? Those details determine the risk.

Q: Is generative AI treated differently from traditional machine learning in the EU AI Act?

A: There are really two regulatory layers:

1. Use-case risk: A traditional ML model used for credit scoring may be high risk; a generative model used to recommend whether someone should receive credit could also be high risk. The regulation looks at function and impact, not whether the system is generative.

2. General-purpose AI (GPAI) model: Providers have obligations around technical documentation, downstream information-sharing, and copyright policies. For GPAI models with systemic risk, expectations get more technical, around model evaluations, adversarial testing, incident reporting, and cybersecurity protections, and those obligations began applying in August 2025.

Generative AI also creates distinct technical risks, involving hallucination, prompt injection, model inversion, data leakage, retrieval poisoning, harmful-content generation, synthetic-media risks, and unpredictable behavior from probabilistic outputs. NIST's Generative AI Profile specifically addresses the fact that generative AI creates risk patterns not fully captured by traditional software testing.

Q: What steps are needed for high-risk systems to comply with the EU AI Act?

A: Think of compliance as a complete system of controls across design, development, deployment, and production:

1. A documented risk-management process, including intended purpose, foreseeable misuse, affected populations, and mitigation decisions.

2. Data governance spanning lineage, bias analysis, and privacy controls across training, validation, and test datasets.
3. Technical documentation and traceability with the ability to reconstruct what model version, configuration, data sources, and prompts produced a given output or action.
4. Logging including infrastructure logs, but expanding to model inputs/outputs, tool calls, retrieval sources, human overrides, and incident events.
5. Meaningful human oversight involving a person clicking "approve" after the AI has already shaped the decision isn't real oversight. The human needs enough information, authority, and time to actually challenge the system.
6. Accuracy, robustness, cybersecurity, change control, post-market monitoring, and, where required, conformity assessment.

The control environment has to remain effective when the model, dataset, prompt framework, retrieval layer, or external provider changes.

Q: What happens if a high-risk AI system fails to comply with the EU AI Act?

A: The financial penalties can be significant, but the operational impact may be even more serious. A regulator may require corrective action, restrict the use of the system, require withdrawal from the market, or stop deployment altogether. If the AI system sits inside a key customer workflow, that could interrupt revenue or create contractual issues before the fine is even calculated.

The highest penalty tier under the Act can reach €35 million or 7% of worldwide annual turnover for certain violations. Other categories carry different thresholds.

There's also evidence risk. If an organization can't produce its system inventory, testing results, risk assessment, model documentation, logs, and approval history, it becomes very difficult to show that reasonable controls existed.

And enterprise customers are already pushing these obligations through contracts. Even companies not directly targeted by a regulator may face customer audits, representations and warranties, indemnification requirements, security questionnaires, and termination rights related to AI.

Q: What does the August 2 transparency deadline for the EU AI Act require, and how should organizations respond?

A: On August 2, 2026, the transparency obligations under Article 50 begin to apply. Organizations need to address situations where individuals interact with AI systems and where AI-generated or manipulated content is produced or distributed, including notifying people when they're interacting with an AI system in relevant circumstances, and addressing labeling or disclosure for certain synthetic or manipulated content, including deepfakes and some public-interest content. The Commission has also issued guidelines to support implementation.

Organizations need to understand where content is generated, how it moves through the technology stack, whether metadata survives downstream processing, and whether the disclosure remains visible when content is copied, exported, compressed, or republished. It's partly a legal question, but it's also a content-provenance and system-design question.

Start with an Article 50 use-case inventory: identify every chatbot, virtual agent, synthetic voice system, image-generation tool, video-generation tool, and content-publishing workflow. Then determine whether the organization is acting as provider or deployer, and what disclosure obligation applies.

Technically, test:

• Whether the disclosure appears before or at the point of interaction

• Whether it's clear to a reasonable user
• Whether generated content carries appropriate marking or metadata
• Whether labels persist after export or transformation
• Whether APIs preserve provenance information
• Whether downstream systems can strip or overwrite the marking
• Whether exceptions are documented
• Whether the organization retains evidence that the control operated

Check third-party model and platform contracts too because a deployer may depend on the provider to supply machine-readable markings, metadata, or documentation. And connect this to change management: if a product team swaps the model or content-generation provider, the transparency control shouldn't silently disappear.

Q: What is the AI literacy requirement in the EU AI Act?

A: The EU AI Act requires providers and deployers to ensure a sufficient level of AI literacy among the people operating or using AI on their behalf and the appropriate level depends on their technical knowledge, experience, education, context of use, and the people potentially affected.

The AI literacy requirement in the EU AI Act points toward role-based literacy:

• A board member needs to understand material risk, governance, accountability, and investment decisions.

• A developer needs to understand evaluation, prompt injection, data leakage, model drift, adversarial testing, secure integration, and logging.
• A procurement team needs to understand vendor responsibilities, data use, intellectual property, model updates, audit rights, and downstream dependencies.
• An HR user needs to understand automation bias, discrimination risk, human review, and system limitations.

In addition to tracking training completion, you should track competencies, role mappings, control ownership, exceptions, and whether users can recognize when a system is outside its approved intended purpose.

Q: Beyond the EU AI Act, what else should organizations track and where can they go deeper?

A: The bigger issue is regulatory convergence. The EU AI Act is the most comprehensive horizontal AI law, but organizations also have to consider privacy, cybersecurity, product safety, consumer protection, employment, discrimination, intellectual property, and sector-specific regulation.

In the US, the AI governance landscape is more fragmented across state AI laws, privacy laws, biometric laws, employment rules, consumer-protection enforcement, and industry-specific obligations. Colorado's AI law is notable because it focuses on high-risk AI used in consequential decisions and includes requirements around risk management, impact assessments, notices, and discrimination risk.

AI governance and cybersecurity governance can't remain separate. An AI system may introduce new attack paths through model endpoints, vector databases, plugins, tool connectors, and autonomous agents.

For primary sources and implementation guidance on the EU AI Act, start with the regulation itself on EUR-Lex, then the European Commission's AI Act portal, implementation FAQs, Service Desk, and guidance from the European AI Office.

For general-purpose AI specifically, the Commission's GPAI guidelines and Code of Practice cover the operational expectations for documentation, transparency, systemic risk, safety, and security. And for technical risk management, NIST AI RMF and its Generative AI Profile are useful as they create a common structure around Govern, Map, Measure, and Manage, and don't replace the law, but help translate legal concepts into measurable controls, testing, monitoring, and governance evidence.

Final advice for EU AI Act Compliance

Start by building an accurate system inventory and architecture view for each AI system, identifying the model, owner, intended purpose, data sources, integrations, permissions, decision impact, and human oversight. Then classify the use case and assign the appropriate control set.

From there, build a proper evaluation program and observability from day one. Traditional software testing asks, "Did the system produce the expected output?" AI testing has to ask broader questions: how does it behave across populations, can the retrieval layer be poisoned, does performance degrade after a model update, can an agent exceed its authority?

The companies that handle regulation well are the ones that can produce evidence showing how the system was designed, tested, approved, monitored, changed, and governed. That's the difference between saying you have responsible AI and actually being able to prove it.

To learn more about the AI governance landscape or how to prepare for compliance, contact us today.