EU AI Act Compliance

EU AI Act Compliance – Create a Framework for Trustworthy AI Systems

This Act establishes a wide-ranging, sector-neutral regulatory framework aimed at laying the groundwork for AI governance across the EU, with potential repercussions for companies and legislative efforts globally.

Download the FREE eBook Now

 

On July 12, 2024, the European Union (EU) introduced Regulation (EU) 2024/1689, known as the AI Act, marking the world’s first comprehensive regulation of artificial intelligence (AI).

1. AI OVERVIEW

Artificial Intelligence (AI) represents a paradigm shift in technology, employing selflearning algorithms that evolve their decision-making processes through the analysis of vast amounts of data.

Unlike traditional software that operates on a static set of instructions, AI dynamically adapts during a training phase and continues its utility in deployment by applying generalised learnings to unseen data. However, this adaptive nature leads to unique challenges, including the risks of overfitting, where models overly specialise in their training data, and bias, which can lead to discriminatory outcomes.

Recognising these risks, the European Union has introduced stringent regulations to govern AI systems that impact its jurisdiction.

As of February 2025, banned practices are to be strictly avoided, while high-risk systems will be heavily regulated starting August 2026.

These regulations aim to ensure that AI systems are both reliable and ethical, balancing innovation with accountability.

The EU AI Act aims to create a framework for trustworthy AI systems.

The way AI systems are trained on data means they can pick up and even amplify biases from the real world. Imagine an AI system used in hiring that unknowingly favours certain groups over others, or an AI in healthcare that works less effectively on people with darker skin tones.

That’s incredibly unfair and can have real consequences.

I think regulators have to be really strict about how AI systems are tested for fairness and discrimination. And not just before they go live, but checking-in regularly throughout their use to make sure they’re not causing unintended harm. Companies shouldn’t be able to just put out an AI system and hope for the best.

The biggest thing for building trust in AI is transparency. It gets scary when you feel like there’s this powerful technology out there making decisions about you for reasons you don’t understand. It feels like you lose control. I think it’s important for people to be able to easily find out how an AI system works – like the data it uses, the processes it follows, and if mistakes happen, what went wrong.

Also, there should always be a human you can go to if you think an AI-made decision was unfair. ‘It shouldn’t be the technology with the final say.’

• autonomy of a system’ (some degree of independence of actions from human involvement).

The adaptiveness that an AI system could exhibit after deployment, refers to selflearning capabilities, allowing the system to change while in use.

• ‘inference’ a key characteristic of AI systems is their capability to infer AI system’ means a machine based system designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predicitions, content, recommendations, or decisions that can influence physical or virtual environments.
• Machine based
• Varying levels of autonomy (even minimum fulfils)
• May exhibit adaptiveness (disregard)
• Infers to influence environment

Which systems are getting regulated?

• Calculator: not regulated (They are not displaying adaptiveness, they are not displaying any inference. )
• Website with cookies to remember dark theme: not regulated (it’s displaying adaptiveness maybe, but it’s not displaying inference. )
• Manually constructed system with expert rules: regulated (perspective issue)
• Machine learning system with automatically inferred rules: regulated
• Machine learning system with online continuous learning: regulated

Many people, if asked, wouldn’t consider a system based on human-constructed rules (without machine learning) to be artificial intelligence. Most would likely say it doesn’t qualify as AI, as it relies on human expertise rather than machine learning. However, if this rule-based system infers new insights from historical data, which it often does, it still performs functions like AI. Consequently, even these systems would likely fall under the scope of the AI Act and be subject to regulation, despite public perception.

Why so broad? Future proof and motivating

Examples:

• Smart marketing campaigns
• Risk/insurance calculation
• Website adjustment engines
• Predicting weather
• Car assistance systems
• Fraud detection
• Face ID recognition
• Server failure prediction
• Chatbots

Examples of artificial intelligence (AI) span a wide range of applications across industries. In marketing, AI is used to predict the optimal product to recommend to a customer, the best time to make a recommendation, and the most effective
communication channel, such as phone, email, or letter. In banking and insurance, AI helps calculate risks and premiums, assessing loan repayments and corporate risk. AI-powered website engines adjust layout and content based on user behaviour,
while AI also plays a role in weather prediction, energy forecasting for solar power, and car assistance systems that read road signs or speed limits. Financial services utilise AI for fraud detection in transactions, and smartphones leverage face recognition for security. AI is also employed in server maintenance, predicting hard disk failures, and in popular applications like chatbots and virtual assistants, such as ChatGPT and cloud-based services.

‘Publicly accessible space’:

Definition:

• Accessible to an undefined number of people
• Irrelevant of private/ public ownership
• Not covered: online spaces
• ‘emotion recognition system’:

Definition: an AI system identifying or inferring emotions or intentions using biometric data

• Focus on identifying emotions through biometric data, not physical states or obvious expressions alone
• Emotions could be considered high risk/ banned
• Exclusions: physical states like pain/ fatigue (systems detecting pilot/ driving fatigue), mere detection of expressions, gestures, or movements unless used to
  infer emotions. – focus on identifying emotions through biodata, not physical states or obvious expressions alone.

‘provider’ natural or legal person/ other body that develops an AI system or general purpose AI model or AI developed and places it on the market or into service under its own name or trademark.

‘deployer’ natural or legal person or other body using an AI system under its authority except where the AI system is used in the course of a personal non professional activity

• Even supply chains are getting regulated: authorised representative, importer, distributor, operator

When comparing AI providers and deployers, the distinction lies in the roles they play in developing or utilising AI systems. Providers are typically responsible for building and customising AI systems. This can involve creating AI from scratch using tools like Python or R, or training untrained models from vendors (e. g. , SAS) on their own data. Providers can also engage in transfer learning, using pre-trained models (e. g. , from platforms like OpenAI or Hugging Face) and fine-tuning them for specific tasks. Despite not training the models from the ground up, these companies still develop complete AI systems, thus qualifying as providers. On the other hand, deployers are companies that implement AI systems developed by others. For example, if a company contracts a vendor like OpenAI to deploy ChatGPT on their servers without modifying the model, they act as a deployer. The distinction often revolves around whether the company is involved in creating and training the AI system (provider) or simply hosting and using it (deployer).

‘intended purpose’: the use for which an AI system is intended by the provider (specific context and conditions of use)

‘reasonably foreseeable misuse’: the use of an AI system in a way that is not in accordance with its intended purpose

‘profiling’ any form of automated processing of personal data consisting of the use of personal data to evaluate certain personal aspects relating to a natural person – often used in contexts such as targeted advertising, credit scoring, and assessing risks in various sectors

Providers of open-source models will be regulated to a much smaller degree compared to the providers of vendor models:

• Documentation
• Free and open-source limitations
• Systemic risk exceptions
• Transparency for AI models: publicly share model parameters, architecture, summarisation of training data, and usage; ensure original provider credit and distribution terms respected

You have to basically reveal everything about the structure of your model, which some people or some providers might be hesitant to do because then they are revealing basically their secrets and their work, which was very expensive. And this is something that many of these vendors are hesitant to do because they are worried that publishers will be suing them, or artists writers will be suing them for improper use of their artistic works.

Open-source AI models hold a crucial place in the IT world, offering alternatives to vendor solutions and fostering innovation through freely available frameworks and libraries. However, there is concern that strict regulations under the EU AI Act could stifle this innovation by imposing burdens on developers who often maintain these projects as a hobby without financial compensation. To address this, the EU AI Act includes significant exceptions for open-source AI models, aiming to ease regulatory pressure. Providers of open-source models must fulfill certain obligations, such as maintaining technical documentation, ensuring transparency by sharing model architecture, parameters, and even training data. This transparency differs from traditional open-source software, where only code disclosure is required. However, models posing systemic risks, such as misuse for harmful purposes, may still face stricter regulation despite being open-source. This approach aims to balance regulation with fostering innovation in the open-source AI community, ensuring that AI development continues without being overly hindered by compliance requirements.

A Precedent: the European unions AI act is one of the worlds first major attempts to draw regulatory boundaries around AI technology. It goes beyond considering benefits toward directly confronting potential harms.

Title II’s significance: these clauses don’t merely outline risky applications that need oversight – they define practises strictly banned throughout the EU.

• 6 months to enter into force

2. Introduction to the Regulations

The Regulation’s purpose is to enhance the EU’s internal market by fostering the development and adoption of safe, human-centred AI, while protecting health, safety, rights, democracy, and the environment from AI’s potential harms. It establishes harmonised guidelines for marketing, operating, and using AI systems across the EU. These include banning certain practices, imposing specific requirements for high-risk systems, enforcing transparency, regulating general-purpose AI models, and ensuring market surveillance. The scope covers EU-based and international entities when their AI is used within the EU, exempting systems for military or national security use, personal, non-professional usage, and research activities before market launch. Detailed definitions clarify roles, responsibilities, and standards, particularly regarding high-risk and biometric AI applications. It promotes AI literacy among providers and deployers to enhance responsible usage.

Chapter II, Article 5 of the AI Act outlines prohibited AI practices to prevent harmful manipulation, exploitation, or unjustified discrimination of individuals. It forbids AI systems that use subliminal or deceptive methods that impair individuals’ informed decision-making, exploit vulnerabilities like age or disability, or employ social scoring that leads to unjust or unrelated negative treatment. AI applications for predictive risk assessments or criminal profiling solely based on personality traits are banned, except when used to support assessments with objective evidence. The Act restricts the unauthorised creation of facial recognition databases and the use of AI to deduce sensitive information, like race or religious beliefs, from biometric data. “Real-time” remote biometric identification in public spaces is restricted to urgent law enforcement needs, such as locating missing persons or preventing serious threats, but requires prior judicial approval, impact assessments, and strict usage controls. Law enforcement must notify relevant authorities about biometric system usage, and annual reports are mandated to maintain transparency.

Chapter III, Article 6 of the EU AI Act establishes the criteria for classifying AI systems as “high-risk.” An AI system is deemed high-risk if it serves as a safety component in products regulated by EU law or if it must pass a third-party conformity assessment. Annex III lists additional high-risk applications, although exemptions apply if the AI poses minimal risk to health, safety, or fundamental rights, such as systems performing limited or assistive tasks without altering decision-making outcomes.

Providers must conduct document assessments if they consider an Annex III AI to be low-risk and comply with registration requirements. The European Commission will provide guidelines by 2026 to clarify high-risk classifications, amend conditions as needed based on evidence, and ensure protections remain robust and consistent.

3. Theme 1: Manipulation & Deception

The principle: Artificial intelligence must not be designed in a way that tricks or secretly influences people to make decisions harmful to themselves or others.

Article 5:

‘Subliminal techniques beyond a person’s consciousness.’ subtle cues in the design of an AI interface, potentially not things a user even consciously notices.

‘. . purposefully manipulative or deceptive techniques’ AI is explicitly engineered to mislead or nudge a person in a predetermined direction.

‘. . with the objective to or the effect of materially distorting. . ’ even unintending consequences matter if they cause serious disruption of decision making ‘…. Causing significant harm.’ Consider harm as: economic loss, psychological distress, damage to reputation, etc

i.e for an online gambling platform, AI alters visuals or sounds to increase a sense of urgency, leading to excessive betting behaviour when they intended to stop.

Or a political bot on social media, targeting individuals based on browsing history, spreading misinformation and leading to polarised view.

Or automated financial advisor who ‘nudges’ a customer through emails into a mortgage, whereas the customer doesn’t have sufficient financial literacy.

4. Theme 2: Exploiting Vulnerabilities

Highlights the ethical responsibility to prevent AI from being designed or deployed in ways that exploit the vulnerable, such as children, the disabled, or those facing economic hardship. The principle is that AI should not manipulate individuals based on their age, disability, or socio-economic status, leading to harmful actions. AI’s strength in recognising patterns and behaviours can be misused to target vulnerable groups, exacerbating issues like financial stress or lack of critical thinking. Scenarios include AI exploiting children’s lack of impulse control for in-app purchases or preying on individuals in financial desperation with predatory loans. The EU recognises the harm these manipulations can cause, emphasising that AI’s power imbalance over individuals must not be leveraged for profit or influence. This responsibility extends beyond malicious intent, calling for developers and companies to safeguard against unethical outcomes, especially in sensitive sectors like healthcare, marketing, and finance, to ensure AI does not reinforce existing social inequalities.

5. Theme 3: Discriminatory Social Scoring

The theme of Discriminatory Social Scoring centres on preventing AI systems from creating unfair ranking systems that assign scores to individuals based on their social behaviours or characteristics, leading to unjustified discrimination. Title Two of the AI Act aims to prohibit AI systems that compute such scores, ensuring human dignity is respected and biases are not perpetuated. Key points include:

1. Social behaviour evaluation: AI systems that evaluate or classify people based on social behaviour or characteristics, such as online activity, spending habits, or social interactions, can unfairly reduce individuals to mere data points.

2. Detrimental and disproportionate treatment: Social scoring is harmful when it leads to two specific outcomes:

• Detrimental treatment in unrelated contexts: A score generated for one reason impacts areas of life that are unrelated (e.g., a social media post affecting job prospects).
• Disproportionate treatment: The consequences of the score do not match the seriousness of the action (e.g., an old comment resulting in long-term penalties).

3. Industry impact: Financial and insurance industries, which use AI to generate risk scores, need to be cautious about how their models use data, ensuring inputs are relevant and outputs are fair. For example, using spending habits to determine loan eligibility may lead to disproportionate penalties if irrelevant factors like past impulsive spending are factored in.

4. Proxies for discrimination: Even when explicitly discriminatory inputs like age are excluded, AI models can still rely on “proxies” (e. g. , spending patterns) that indirectly replicate the effects of those inputs, raising ethical concerns.

Example scenarios highlight the risks of AI-based scoring:

• An insurance company adjusting premiums based on lifestyle predictions from unrelated data (e. g. , takeout spending).
• The risk of unsupervised AI models, which group individuals based on data, potentially incorporating hidden discriminatory factors.

Key Points:

• The EU act focuses on preventing unfair and opaque scoring systems that harm individuals based on factors beyond their control.
• AI should not dictate all aspects of life or perpetuate societal inequalities.
• Even if AI scores are accurate, misuse can cause unjust discrimination by reducing people to data points without considering the broader human context.

6. Mass Surveillance and Biometric Identification

The theme of Mass Surveillance with Biometric Identification in the AI Act centres on prohibiting the use of real-time facial recognition and other biometric tracking technologies in public spaces without stringent justification and oversight, due to their threat to privacy and personal freedom.

Key points include:

1. Real-time biometric identification: This involves monitoring individuals in public spaces (e. g. , streets, parks, transportation hubs) using AI-powered systems that constantly track their facial or biometric data, far beyond traditional security measures. Such technology enables widespread, continuous surveillance of anyone in public, turning people into traceable data points.

2. Strict limits for law enforcement: The AI Act allows exceptions in very specific cases, including:

• Locating missing or abducted children.
• Addressing imminent threats to life or serious terrorist attacks.
• Identifying suspects in active investigations of serious crimes.

These uses require clear, exceptional justification, with an emphasis on immediate danger and not broad, untargeted surveillance.

3. Misuse and abuse: The Act highlights potential scenarios where mass surveillance could be misused:

• Political suppression: Governments using facial recognition to track and silence protesters, undermining freedom of assembly.
• Commercial exploitation: Businesses using biometric tracking to monitor consumer behaviour, invading privacy to build marketable profiles.

4. Prohibition on predictive policing: The Act bans the use of AI for predictive policing, which involves forecasting crimes based on biometric data or profiles, as it risks reinforcing harmful biases and stereotyping.

To summarise the key takeaways:

• Erosion of anonymity: Biometric tracking threatens the privacy people expect in public, making them constantly identifiable and traceable.
• Burden of proof on authorities: Any use of biometric surveillance must be backed by a serious, well-justified case, with evidence that less intrusive methods are insufficient.

The AI Act maintains a strict stance on mass surveillance, demanding ongoing scrutiny and a balance between security and individual freedoms.

7. Classification Rules for High-Risk AI Systems

Identified as those that either function as a:

1. (IN) safety component of a product regulated by specific EU legislation listed in Annex I and are mandated to undergo an external conformity evaluation as per those Annex I regulations (e. g. Machinery, Medical devices, Toys, Lifts, Pressure equipment, Recreational watercraft, Civil aviation security); OR

2. (IN) are associated with the use cases outlined in Annex III Derogation (OUT): AI systems shall not be considered as high risk if they do not pose a significant risk of harm, to the health, safety or fundamental rights of natural persons, including by not materially influencing the outcome of decision making. This is the case if one or more is true:

• If the AI system carries out a narrow procedural task;
• The AI system is intended to improve the result of a previously completed human activity;
• Identifies patterns or deviations in decision making without intending to substitute or sway the prior human judgement without adequate human oversight;
• Executes a preliminary task for an evaluation significant to the purposes outlined in Annex III;
• Obligation: Providers of AI systems falling under Annex III, which believe their system does not constitute a high risk, shall document such an assessment prior to its market launch or service provision;
• (Back IN) AI systems that profile individuals, defined as the automated processing of personal data to evaluate various aspects of an individual’s life such as work performance, financial status, health, personal interests, reliability, behaviour, locations, or movements, are very likely to be classified as high risk.

Summarised:

The discussion revolves around the classification of high-risk AI systems under the EU AI Act, focusing on two main categories:

1. Safety Components of Regulated Products: High-risk AI systems are those used as safety components in heavily regulated industries, such as machinery, medical devices, lifts, and toys, which are already subject to strict EU laws. Companies in these sectors, already familiar with regulation, will have to conform to additional AI regulations, but only when AI is used as a safety component

2. Broader Use Cases in Annex III: The second, more impactful category concerns AI systems used in broader sectors like education, finance, and insurance. For instance, AI used to grade students’ work or assess creditworthiness would be classified as high-risk. Annex III lists a wide range of areas where AI systems will fall under this category, which is new for many industries.

Derogations and Obligations:

• Derogation: AI systems can avoid being classified as high-risk if they do not pose significant harm to health, safety, or fundamental rights. For example, if an AI performs a narrow procedural task (e. g. , organizing students’ assignments), it may qualify for this exception.
• Obligation: Even when using the derogation, companies must document and assess their AI systems before launching them.
• Profiling Exception: If the AI profiles individuals (e. g. , assessing financial status or behaviour), the derogation is overridden, and the system remains high-risk regardless.

In summary, the EU AI Act’s high-risk classification impacts industries already under strict regulation, as well as new sectors, especially when AI is used for sensitive tasks like grading or financial evaluations.

Derogation

Derogation, a way for AI systems to avoid being classified as high-risk under the EU AI Act. There are several scenarios in which AI systems can potentially be exempted from the high-risk label, even if they fall under the broad areas listed in Annex III.

Key points:

Narrow Procedural Tasks: AI systems that perform minor, non-critical tasks (like organising data or handling administrative duties) can qualify for derogation. For example, if AI is used in an educational setting for minor tasks but not for grading students or making impactful decisions, it wouldn’t be considered high-risk.

1. Enhancing Human Work: If AI is used to suggest improvements on previously completed human activities (e. g. , suggesting edits to an essay after a teacher has graded it), it may avoid the high-risk classification. The crucial factor is that the final decision remains with the human.

2. Decision-Making Support: AI systems that merely highlight trends or patterns without directly influencing decision-making (e. g. , flagging performance data trends for a manager) could also qualify for derogation. However, there is concern about potential risks if humans blindly rely on AI outputs.

3. Preparatory Tasks: AI systems that organise or prepare information to assist human decision-makers (e. g. , organising applicant data for housing assistance) may avoid being classified as high-risk as long as the final decision rests with a human. Overall, the positioning of AI within a workflow (whether it replaces or supports human decision-making) is key to determining whether it qualifies for derogation. Its predicted that an industry of consultants might emerge to help businesses strategically design AI systems to avoid high-risk classification.

Biometrics

1. Legal Restrictions: Biometric and emotion recognition systems may be subject to strict national or union laws. Before proceeding with such AI systems, organisations must ensure they are not violating banned practices.

2. Remote Biometric Identification: AI systems used for real-time identification in public spaces (like airports) based on biometric data are likely to be high-risk, especially when scanning large populations for safety purposes, such as identifying missing persons or criminals.

3. Biometric Categorisation: AI systems that categorise individuals based on sensitive attributes (e. g. , gender, ethnicity) using facial features could lead to discrimination and are also classified as high-risk. An example is targeted advertising based on inferred personal characteristics.

4. Emotion Recognition: AI systems that assess emotional states (e. g. , in workplaces or retail environments) are considered high-risk due to the interpretive nature of emotions and the potential privacy invasion. These systems monitor facial expressions or vocal tones to provide feedback on emotions, which raises significant ethical concerns.

These technologies often fall under the high-risk category due to their potential to affect individuals without their consent, and that businesses must carefully assess legal and ethical implications before deploying them.

Critical Infrastructure

AI systems intended to be used as safety components in the management and operation of critical digital infrastructure, road traffic, or in supply of water, gas, heating or electricity.

Critical Infrastructure: This term was used during the COVID-19 pandemic to grant special privileges to essential industries. AI regulations may reuse the same concept for industries like digital infrastructure, road traffic management, and utilities (water, gas, electricity).

Safety Components: Not all AI systems in critical infrastructure are considered high-risk—only those directly related to safety components. These could be systems where failure could endanger human safety or critical operations. The definition of safety components remains open to interpretation, however leaning toward a broad understanding that includes systems embedded in processes or products critical to safety.

Examples of Critical Infrastructure AI:

• Cybersecurity for Cloud Services: AI systems monitor and defend against cyberattacks, ensuring the safety of digital infrastructure. – systems failure or misuse could compromise large scale digital services, exposing sensitive data or causing service outages that affect millions of users
• Data Centre Management: AI-driven systems manage cooling, energy, and server maintenance to prevent failures. – given the role of data centres in supporting internet services, any malfunction or mismanagement by the AI could lead to widespread disruptions of online services and critical applications.
• Road Traffic Systems: AI can dynamically manage traffic flows using cameras and real-time data to ensure road safety. – an error in the system could lead to accidents, severe traffic jams or delayed emergency response, directly impacting public safety and mobility.
• Autonomous Vehicle Coordination: AI systems might coordinate autonomous cars, directing them to avoid traffic or accidents. – a failure in the system could result in vehicle collisions, endangering passengers, pedestrians, and other road users as well as disrupting traffic flow.
• Utility Supply (Water, Gas, Electricity): AI systems can optimise electricity grids or water treatment processes to balance supply and demand, or ensure proper filtration. – if the AI system fails, it could cause widespread power outages or imbalances in electrical supply, affecting households, businesses, and hospitals.
• Automated water treatment plans; AI systems can control the operations of a water treatment plant, adjusting chemical dosages, filtering processes, and water distribution based on real time data from various sensors – a malfunction could lead to unsafe water quality, either through contamination or inadequate supply, posing serious health risks.

Implications of AI systems in education as categorised under the EU AI Act, particularly those deemed high risk. Key points include:

1. AI in Admissions: AI systems can be used for admissions to educational and vocational institutions, such as automated university admission systems that screen and rank applicants. The EU emphasises that AI should not negatively influence this process, acknowledging the risks involved.

2. Evaluating Learning Outcomes: AI can also evaluate student performance, especially in grading open-ended assessments like essays and projects. While multiple-choice tests don’t involve AI, using AI to assess complex assignments presents high risks. He suggests universities will need to adapt their evaluation methods in light of students increasingly using generative AI tools like ChatGPT.

3. Adaptive Learning Platforms: These platforms adjust to individual student performance, offering tailored exercises based on their progress. AI-driven assessments can also adapt during exams to present more challenging questions as students demonstrate proficiency, classifying these systems as high-risk.

4. Assessing Educational Levels: AI systems may also determine the level of education individuals can access, particularly in the context of vocational training subsidies funded by the EU. If AI makes decisions about who receives these subsidies, it falls under high-risk AI.

5. Monitoring Student Behaviour: AI systems can monitor for cheating during exams, utilising technologies like facial recognition and keystroke analysis. While such remote proctoring is becoming common, it is classified as high risk due to privacy concerns and potential biases.

The role of AI in employment and worker management, highlighting two main areas under the EU AI Act that are classified as high risk:

1. AI in Recruitment and Selection:

AI systems are increasingly used for recruitment processes, such as automating the screening of resumes and evaluating candidates based on criteria like keywords and experience. While these systems can improve efficiency, they carry significant risks due to potential biases in the training data. If the AI is trained on data reflecting biased hiring practices, it may perpetuate those biases, leading to unfair job opportunities. There is caution against hiring managers using generative AI tools like ChatGPT to summarise CVs, emphasising that while AI can be beneficial, it can also misinterpret or introduce bias into the selection process.

2. AI in Work Relationships:

AI systems are also being developed to make decisions regarding task allocation, promotions, and performance evaluations. For instance, AI can monitor employee productivity and assign tasks based on performance metrics. However, this can lead to unfair advantages for high performers, limiting opportunities for others who might not be recognised as such by the AI.

There’s potential for AI-driven performance monitoring systems that evaluate employee behaviour and communication styles. While these tools could provide insights into performance, their use raises privacy concerns and ethical questions regarding employee surveillance.

“Access to Essential Services and Benefits” within the EU AI Act, expressing dissatisfaction with how various distinct areas are grouped together. Here are the key points:

1. General Critique:

Combining diverse areas such as creditworthiness, insurance, and emergency services under one umbrella is problematic. The importance of these categories varies significantly, particularly emphasising the critical nature of emergency services compared to financial assessments.

2. AI in Public Assistance:

AI systems are expected to assess eligibility for public benefits like healthcare and unemployment assistance, which could replace human decision-makers in government offices. There are risks associated with such systems, particularly if they are not designed to be fair and unbiased.

3. AI in Creditworthiness:

AI will also be used to evaluate the creditworthiness of individuals, impacting the banking sector. While systems for detecting financial fraud are exempt from being classified as high risk, the broader credit assessment tools will face strict regulations. This highlights concerns from the banking industry regarding existing regulations and how the new AI Act will impose additional burdens on credit assessment processes, which are already heavily monitored.

4. AI in Insurance:

AI systems assessing life and health insurance applications will predict risks based on factors like medical history and personal habits. This area is also classified as high risk, impacting how insurers evaluate applicants.

5. AI in Emergency Services:

AI may be used to classify and prioritise emergency calls for services such as police and medical aid. It’s suggested that while this could enhance efficiency, it must be handled cautiously due to the stakes involved in emergency response.

The most significant impact will likely be on the banking industry, particularly regarding the evaluation of creditworthiness and loan approvals, due to the stringent regulations introduced by the AI Act.

AI Systems for Crime Victim Risk Assessment

• Predictive Analysis: AI systems are being developed to analyse crime data and demographics to identify individuals or areas at higher risk of becoming victims of crimes. This includes assessing historical crime patterns to allocate police resources effectively.
• Self-Fulfilling Prophecy: A critical issue with predictive policing: when police are dispatched to specific areas based on AI predictions, they may uncover more crime in those areas. This can create a feedback loop where increased policing leads to higher reported crime rates, which in turn reinforces the AI’s predictions. This self-fulfilling prophecy can result in biased outcomes and exacerbate existing societal issues.

1. AI for Evaluating Evidence

• Forensic Evidence Assessment: AI systems can be used to evaluate the reliability of forensic evidence (e.g. DNA samples, fingerprints, digital evidence) during criminal investigations. This application is considered high risk due to the potential implications of inaccurate assessments on legal outcomes.

2. AI in Risk Assessment for Offending

• Re-Offending Predictions: AI tools may also be employed to assess the risk of individuals re-offending after release from prison. Unlike the extreme scenario depicted in the film Minority Report, which involves predicting crimes before they occur, this application focuses on analysing data from convicted individuals to model their likelihood of re-offending based on historical behaviour.
• Regulatory Considerations: While this type of predictive modelling is not classified under prohibited practices, it still falls into the high-risk category, necessitating careful consideration and regulation due to its potential consequences on individuals’ lives and public safety.

Key points:

The deployment of AI in law enforcement raises complex ethical and practical challenges, particularly regarding bias, accountability, and the potential for unintended consequences. All discussed AI applications within law enforcement are categorised as high risk, indicating the need for stringent oversight and regulation to ensure responsible use and mitigate potential harms associated with these technologies.

“Migration, Asylum, and Border Management” within the context of the EU AI Act:

Security, Irregular Migration, and Health Risks: AI systems are intended to assess various risks associated with individuals entering or within the territory of EU member states. This includes evaluating security risks, irregular migration, and health risks.

Specific Applications

• Border Security Assessments: AI can be used to analyse data such as travel history, visa records, and social media activity to flag individuals who may pose security threats at borders. This application is classified as high risk due to the implications of automated decision-making.
• • Visa Application Risk Assessment: Another use case involves AI tools used by consulates to evaluate visa applications. These systems would analyse factors such as an applicant’s travel history, financial status, and ties to their home country to assess the likelihood of overstaying or engaging in illegal activities. This application also falls under the high-risk classification.

The key takeaway is that while human assessments at borders are not regulated, the use of AI in these processes elevates the risk and necessitates careful oversight due to the significant implications for individuals’ rights and security.

“Administration of Justice and Democratic Processes” within the context of the EU AI Act:

This category covers AI systems used in the judicial sector and for influencing democratic processes. Despite having only two points, the implications are significant.

AI in Judicial Authorities

AI in legal and democratic processes pose significant high-risk implications. In the legal realm, AI tools can assist judicial authorities by researching case law and suggesting precedents, potentially impacting judicial outcomes. Sentencing prediction systems that recommend penalties based on past cases also fall under high-risk due to their direct influence on sentencing decisions. A critique arises over the lack of regulation for generative AI used by private firms for legal interpretations, raising concerns about people relying on cheaper AI tools over qualified legal professionals, which could lead to misinterpretations. In democratic contexts, AI used in elections—such as influencing outcomes or microtargeting voters with personalised messages—also carries high-risk, as these tools can subtly manipulate public opinion by exploiting emotional reactions. The Cambridge Analytica scandal shows how such practices can impact voter behaviour, with recommender systems on social media often fuelling this through targeted content aimed at swaying users, driven by ad revenue.

Obligations for High-Risk AI Systems

1. Risk management system (Article 9)
2. Data governance (Article 10)
3. Documentation (Article 11)
4. Record-keeping (Article 12)
5. Transparency (Article 13)
6. Human Oversight (Article 14)
7. Accuracy (Article 15)
8. Robustness (Article 15)
9. Cybersecurity (Article 15)

The first obligation for high-risk AI systems is establishing a risk management system. This is described as a structured, ongoing process that must be documented and maintained throughout the AI system’s lifecycle.

Key Components of a Risk Management System

.1. Ongoing Process: The risk management system is not a one-time checklist but a continuous and iterative process that needs regular updates as the AI system evolves, and new risks are identified.

2. Steps in the Risk Management Process:

• Identify and Analyse Risks: Start by identifying all possible risks that the AI system may pose to health, safety, or fundamental rights, including foreseeable risks based on intended use.
• Estimate and Evaluate Risks: Assess the likelihood of identified risks occurring and their potential impact, considering normal and foreseeable misuse scenarios.
• Post-Market Risk Evaluation: After deployment, continue evaluating new risks based on real-world data and user feedback.
• Adopt Targeted Measures: Implement specific measures to mitigate identified risks, such as updating algorithms or improving user instructions.
• Focus on Mitigatable Risks: Concentrate efforts on risks that can be reasonably addressed through design or user training.
• Ensure Acceptable Residual Risks: Evaluate whether remaining risks are acceptable after implementing mitigation measures.
• Testing: Conduct thorough testing before market introduction to ensure consistent performance and effective risk management.
• Consider Vulnerable Groups: Take additional measures to protect vulnerable groups, such as minors or those in precarious situations.
• Integration with Other Processes: Integrate the risk management process with other existing EU legal frameworks, like GDPR.

Practical Implementation: Case Study on AI Hiring Tool

1. Appointment of a Team: A cross-functional team (including AI developers, HR professionals, data privacy officers, and legal experts) is appointed to oversee the risk management process for the AI hiring tool.

2. Establishing Risk Management Process: The team documents a clear risk management process to be followed throughout the tool’s lifecycle.

3. Initial Risk Identification: Conduct workshops to brainstorm potential risks like bias in candidate selection, data privacy issues, and misinterpretation of resumes.

4. Risk Analysis and Documentation: Analyse identified risks for their potential impact and likelihood, prioritising them for mitigation strategies.

5. Implementing Design Solutions: Incorporate design features to mitigate risks, such as algorithms for bias mitigation and data anonymisation.

6. Continuous Monitoring: Regularly monitor for new risks during development, allowing for adjustments as needed.

7. Pre-Deployment Testing: Test the AI tool extensively, including simulated and real-world conditions, to ensure fairness and accuracy.

8. Defining Metrics and Thresholds: Set clear performance metrics for the AI tool, such as acceptable error rates and bias levels.

9. Deployment: After successful testing, deploy the AI tool while continuing to monitor its performance and gather user feedback.

10. Regular Reviews and Updates: Conduct regular reviews of the AI system’s performance, especially after significant updates.

11. Risk Mitigation and Communication: Implement additional measures for identified risks that cannot be eliminated, including human oversight and user training, while ensuring transparency with job applicants about the AI tool’s use.

12. Consideration for Vulnerable Groups: Address the potential impacts on vulnerable groups and implement additional safeguards to ensure fairness.

Practical Case Study – Key Steps in Data Governance:

Assemble a Data Governance Team: A cross-functional team, including data scientists, legal experts, compliance officers, and risk managers, oversees the entire data lifecycle, ensuring adherence to regulatory and ethical standards.

1. Define Data Governance Protocols:

Develop protocols for data management, covering design choices, data collection processes, and data preparation (cleaning, annotating, enriching) to ensure data is accurate and compliant with legal standards.

2. Ensure Data Quality and Relevance:

Review data sets to ensure relevance, completeness, and accuracy. Ensure diversity in data to avoid regional or socioeconomic biases, which is crucial for making fair credit decisions across demographics.

3. Bias Analysis and Mitigation:

Conduct detailed bias analysis to identify patterns leading to unfair outcomes, such as demographic overrepresentation or historical biases. Corrective measures include rebalancing datasets or algorithmic techniques to reduce bias.

4. Processing Special Categories of Personal Data:

If necessary, use sensitive data (e. g. , ethnicity, health) to detect and correct bias, with strict safeguards such as limited access, data encryption, and deletion after use.

5. Rigorous AI System Testing:

Test the AI system extensively to ensure consistency and fairness in creditworthiness evaluation. Conduct continuous bias testing across diverse applicant profiles to confirm fair treatment across all demographics.

6. Monitoring and Updates Post-Deployment:

Continuously monitor the AI’s performance after deployment to identify new risks or biases. This ensures the system remains fair and compliant.

7. Comprehensive Documentation:

Document every step of the data governance process, including data sources, preparation methods, and bias mitigation strategies, for audits and regulatory inspections.

8. Regular Compliance Reviews:

Schedule reviews to ensure that data governance practices are up-to-date with legal changes and evolving AI system use.

8. Summary of Article 11 on documenting high-risk AI systems:

The main requirement of Article 11 of the EU AI Act is technical documentation for highrisk AI systems.

Key points:

1. Pre-market Documentation: The technical documentation must be prepared before the AI system is deployed or placed on the market, and it must be kept up to date.

2. Compliance Focus: The documentation must demonstrate that the AI system complies with all applicable legal requirements, particularly around cybersecurity, accuracy, robustness, and more.

The structure of the documentation has been split into three key parts:

1. General System Description:

This covers the intended purpose (e. g. , fraud detection), foreseeable misuse, interactions with other systems, and any hardware or software dependencies. It also includes details on how the AI system interacts with external systems or APIs, versioning, and user instructions.

2. AI System Development Details:

This includes: The development methodology (e. g. , Agile). Any pre-trained models or third-party tools used (e. g. , TensorFlow). Design specifications and the logic behind design choices. Information on the data sets used, bias mitigation efforts, human oversight, and validation/testing procedures. Cybersecurity measures to address threats like data poisoning or adversarial attacks.

3. Monitoring and Post-Deployment Functioning: This part focuses on monitoring the AI system’s performance after deployment, including: Performance metrics related to real-world use. Assessment of cybersecurity risks. Addressing foreseeable unintended outcomes and risks.

These are draft recommendations based on experience, and further guidelines may emerge by 2025-2026. The documentation must be formal, thorough, and continuously updated as the AI system evolves.

9. Summary of Article 12 on recordkeeping for high-risk AI systems:

The main idea is to ensure traceability, compliance, and monitoring of the system’s operations. Using a case study of a traffic signal control AI system, the steps to implement record-keeping are outlined:

1. Event Logging:

Log key actions and events (e. g. , traffic signal changes or detection of anomalies). Ensure timestamping to trace the timing of each action. Log data sources (e. g. , traffic cameras or sensors) that influence the AI’s decisions.

2. Compliance with Article 12:

Risk identification: Logs should capture potential system risks (e. g. , signal malfunctions or issues with emergency vehicle routing). Post-market monitoring: Use logs for ongoing audits and performance assessments. Operational monitoring: Keep track of daily AI decisions to ensure regulatory compliance.

3. Session Logging:

Record the start and end times of each AI session when managing traffic signals to monitor the frequency and duration of use.

4. Logging Reference Databases:

Track which datasets (e. g. , historical traffic data) were used in the AI’s decisionmaking.

5. Input Data Logging:

Log the specific input data (e. g. , vehicle speed or emergency detection) that triggered each decision to adjust signals.

6. Identification of Human Oversight:

Record the identities of human operators who review or override AI decisions to ensure accountability. In conclusion, the traffic signal control system’s record-keeping ensures traceability, monitoring, and regulatory compliance, providing a model for handling high-risk AI systems.

10. Summary of Article 13: Transparency and Provision of Information to Deployers

Article 13 of the EU AI Act focusing on transparency and the provision of information to deployers of high-risk AI systems.

Key differences between Article 11 and Article 13:

• Article 11 concerns internal technical documentation aimed at ensuring the AI system’s compliance with the law. This documentation is primarily for the provider and regulators.
• Article 13, on the other hand, applies when a provider sells or licenses the AI system to deployers (other companies). It specifies the documentation and information that

providers must give deployers without disclosing sensitive company know-how. Providers are expected to maintain two sets of documentation:

1. Internal documentation (for the company and regulators as per Article 11).

2. Customer-facing documentation for deployers as required by Article 13.

The information provided to deployers should cover:

• Intended purpose and contact details of the provider.
• Characteristics, capabilities, and limitations of the AI system, including accuracy metrics, robustness, and cybersecurity measures.
• Risks in intended use, such as potential misuse.
• Explanation of output, helping deployers understand how the AI system arrives at decisions.
• Input data specifications to ensure proper operation (e. g. , data types required by the AI model).
• Guidelines for interpreting outputs, for instance, defining score thresholds for decision-making (e. g. , loan approval).
• Human oversight, detailing how human review or intervention should be applied.
• Logging mechanisms, specifying how the system logs its operations.
• In summary, Article 13 ensures that providers give deployers relevant, understandable documentation without exposing proprietary information, allowing them to operate the AI system effectively while maintaining transparency and compliance.

11. Human Oversight (Article 14)

High risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use Human oversight shall aim to prevent or minimise the risks to health, safety or fundamental rights that may emerge when a high-risk AI system is used in accordance with its intended purpose or under conditions of reasonably foreseeable misuse, in particular where such risks persist despite the application of other requirements set out.

12. Accuracy (Article 15)

High risk AI systems shall be designed and developed in such a way that they achieve an appropriate level of accuracy, robustness, and cybersecurity, and that they perform consistently in those respects throughout their lifecycle.

To address the technical aspects of how to measure the appropriate levels of accuracy and robustness set out in paragraph 1 and any other relevant performance metrics, the Commission shall, in cooperation with relevant stakeholders and organisations such as metrology and benchmarking authorities, encourage, as appropriate, the development of benchmarks and measurement methodologies.

The levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use.

13. Robustness (Article 15)

High risk AI systems shall be as resilient as possible regarding errors, faults or inconsistencies that may occur within the system or the environment in which the system operates, in particular due to their interaction with the natural persons or other systems. Technical and organisational measures shall be taken in this regard.

The robustness of high risk AI systems may be achieved through technical redundancy solutions, which may include backup or fail safe plans. High risk AI systems that continue to learn after being placed on the market or put into service shall be developed in such a way as to eliminate reduce as far as possible the risk of possibly biased outputs influencing input for future operations (feedback loops), and as to ensure that any such feedback loops are addressed with appropriate mitigation measures.

14. Data centric robustness:

• High quality input features (NOT just ‘data’)
• Data diversity and representation (data augmentation, targeted data collection, stratified sampling)
• Bias detection
• Fairness metrics (disparate impact)
• Unexpected data points (data validation)

15. Model-centric robustness:

• Robust model architectures (e. g. ensemble)
• Cost sensitive learning (i. e. exposing true costs of errors to the model for training)

16. Cybersecurity:

High risk AI systems shall be resilient against attempts by unauthorised third parties to alter their use, outputs or performance by exploiting system vulnerabilities.

The technical solutions aiming to ensure the cybersecurity of high risk AI systems shall be appropriate to the relevant circumstances and the risks. The technical solutions to address AI specific vulnerabilities shall include, where appropriate, measures to prevent, detect, respond to, resolve and control for attacks trying to manipulate the training data set (data poisoning), or pre-trained components used in training (model poisoning), inputs designed to cause the AI model to make a mistake (adversarial examples or model evasion), confidentiality attacks or model flaws

Resilience Against Unauthorised Manipulation:

High-risk AI systems must resist manipulation by unauthorised parties, who might exploit vulnerabilities to alter system outputs, performance, or usage.

Technical defences should match the specific threats and context of each high-risk AI application.

Cybersecurity Measures:

Cybersecurity solutions should address vulnerabilities unique to AI, with tools to prevent, detect, and respond to attacks on various fronts:

Data Poisoning: Inserting misleading data into the training set to alter model behaviour, a risk heightened by the drive for richer data sets.

Model Poisoning: Attacking pre-trained components used in training to embed hidden behaviours or backdoors, especially concerning with open-source models obtained from platforms like Hugging Face.

Adversarial Attacks: Feeding inputs crafted to make the AI model err, often by inducing it to misclassify.

Confidentiality Attacks: Targeting data privacy by extracting sensitive information from the model, such as intellectual property or private customer data.

Examples and Evolving Challenges:

Data Poisoning: Real-world example in a bank, where fake data manipulated a loan model, changing loan approval decisions.

Model Poisoning and Gating: Using pre-trained components like Meta’s gated models poses risks, as malicious intent in these models could lead to targeted attacks, especially when the creator knows the model’s intended use. Model Evasion:

Post-training, models are vulnerable to attacks aimed at tricking them into incorrect outputs.

Adversarial Examples: A specific type of attack where inputs are subtly altered (e. g. , a manipulated sticker on a stop sign) to cause incorrect predictions.

General Evasion: Attackers might exploit model weaknesses without needing access to training data, using slight input perturbations to uncover vulnerabilities.

Testing for Robustness:

Testing how models react to input changes (e. g. , adjusting age or occupation in a loan application) helps identify when model responses shift undesirably, revealing weaknesses similar to those exploited in adversarial attacks.

Confidentiality Attacks:

Model Stealing: Repeatedly querying a model to recreate a similar one without direct access, a process called “model distillation. ” This may pose risks, especially for highvalue or sensitive models.

Data Exfiltration: Attempts to retrieve the model’s training data (e. g. , reproducing copyrighted material or sensitive data) violate intellectual property and data security.

Inference Attacks: Extracting sensitive or private information from a model, like personal data, poses significant risks, especially if the model was trained on confidential data.

17. Practical Implementations

Current AI Development Workflow:

Idea Generation: AI developers rely on domain experts to propose initial ideas where AI might add value, such as document categorization, customer predictions, or chatbot functionality.

Concrete Proposals: Promising ideas are refined into detailed project proposals, specifying data use, targeted users, cost, timeline, and team structure. This stage filters out non-viable ideas.

Prototype Development:

Testing Validity: Instead of full-scale implementation, a small prototype or experimental model is built to test the idea’s viability, aiming to verify the approach before full investment. These prototypes typically take between one and twelve months, often benchmarked at three months.

Full Development and Deployment:

Only successful prototypes advance to full development and deployment, while unsuccessful prototypes are abandoned.

Regulatory Obligations:

When deploying high-risk AI systems, developers must adhere to various obligations, including post-market monitoring and continuous risk management documentation. These requirements are clear for full deployment but become problematic between the proposal and prototype stages.

Survival Rates of Prototypes:

Experience shows that only about 10% to 30% of prototypes advance to full deployment. This is a typical part of the innovation process, as many prototypes fail during testing.

Impact of Regulations:

If regulations impose obligations before the prototyping phase, it may hinder innovation by increasing costs and complexity. Lengthy documentation and approval processes could lead teams to narrow down their testing to fewer ideas, reducing experimentation and exploration of potentially valuable projects.

Potential Redesign of Prototyping Processes:

To mitigate the impact of regulations, companies might adapt their prototyping processes. Instead of immediately building an AI system, they could use simpler, non-AI methods for initial testing. For example, manual data analysis could be employed to identify potential customer churn before implementing a more complex AI solution.

18. Direct Interaction with natural persons

Scope of Obligations:

These obligations apply only to AI systems intended for direct interaction with people, such as chatbots. They do not apply to systems like credit scoring, where users do not interact directly.

Disclosure Requirement:

Providers must ensure that users are informed when they are interacting with an AI system. This includes clear communication that a chatbot or virtual assistant is not a human. This practice has been seen in the industry and aims to improve transparency.

Consumer Perception:

Research suggests that consumers may value products enhanced by AI less than those involving genuine human interaction. Therefore, providers might feel tempted to obscure the use of AI, which is why the obligation for disclosure is emphasised.

Exceptions:

There are exceptions to the disclosure requirement. For instance, if it is obvious to a reasonably observant person that they are interacting with an AI (such as a physical robot with an AI assistant), explicit disclosure may not be necessary.

Additionally, AI systems authorised by law for criminal detection, prevention, investigation, or prosecution are exempt from these obligations. This could include scenarios where AI-generated deepfake voices are used by law enforcement without disclosing their artificial nature.

Key distinctions between synthetic content and deepfakes:

Synthetic Content:

While not explicitly defined in the AI Act, synthetic content is understood as any content generated by AI systems. This includes images, text, and other media produced using AI, regardless of the input.

The rationale is that if an AI system is involved in creating content, that content is considered synthetic. For instance, images generated by Midjourney or text produced by ChatGPT are both classified as synthetic content.

The definition is broad, meaning any output from an AI system is deemed synthetic, regardless of how much human input is involved.

Deepfakes:

Deepfakes are specifically defined in the AI Act as AI-generated or manipulated media (image, audio, video) that resembles existing individuals, objects, or events.

A key component of the definition is that deepfakes must falsely appear authentic or truthful. This means that if people can easily recognize a deepfake as inauthentic, it may not qualify as a deepfake under the AI Act.

This definition raises concerns as it may provide a defence for those creating harmful deepfakes, allowing them to argue that their content doesn’t meet the criteria for being a deepfake.

Concerns and Implications:

Concern about the vagueness of the deepfake definition, suggesting it could lead to challenges in regulation and accountability.

There is a worry that the definition might enable individuals to escape consequences for producing harmful deepfake content by claiming it is not perceived as authentic by a majority.

Key Distinction:

Synthetic content is the broader category encompassing any AI-generated content.

Deepfakes fall within this category but are specifically those forms of synthetic content that resemble real people or events and falsely appear authentic.

Synthetic Key Points:

1. Scope of Obligations:

The focus is on obligations for providers of AI systems (e. g. , OpenAI, Midjourney) that generate synthetic audio, images, videos, or text.

These obligations do not cover scenarios where users of social media platforms post synthetic content.

2. Marking Outputs:

Providers must ensure that AI-generated outputs are marked in a machinereadable format, allowing other algorithms to easily recognise them as synthetic. This obligation aims to enhance transparency regarding AI-generated content.

3. Technical Solutions:

Providers should implement effective, interoperable, and reliable technical solutions for marking synthetic content, taking into account the costs and limitations of different content types.

Examples of potential marking methods include:

• Digital Watermarking: An invisible mark embedded within the image that can be detected by algorithms.
• Metadata Embedding: Including information in the header of documents to indicate AI generation.
• Steganography: A technique for hiding information within images or text that could be used positively to mark synthetic content discreetly.
• New File Formats: Introduction of specific file extensions for AI-generated content, though this is still speculative.

1. Content Variability:

The obligation for marking may vary based on the type of synthetic content and its realism. For example, realistic images might face stricter marking requirements compared to more abstract content.

The cost and complexity of implementing markings for physical representations of digital content (like billboards) may influence regulatory obligations.

1. Exceptions:

The obligations do not apply if the AI system is merely assisting with standard editing or if it does not substantially alter the input data.

For instance, using AI for proofreading may exempt the output from the marking requirement.

Deep fakes key points:

Regulation Overview:

• Deployers of AI systems that create or manipulate content (deepfakes) must disclose that such content is artificially generated or manipulated.
• There are exceptions for content that is clearly artistic, creative, satirical, or fictional, suggesting that deepfakes can be permitted under certain conditions.

Surprising Allowance:

• One major takeaway is that deepfakes are generally allowed under the current framework, provided there is a transparency obligation to disclose their nature.
• Concerns are raised about the European Union’s approach to regulating deepfakes, especially given their growing prevalence and sophistication in society.

Challenges with Deepfake Technology:

• Quality and Accessibility: The quality of deepfakes has significantly improved, and tools to create them are now easily accessible to casual users, raising the risk of misuse.
• Minimal Reference Data: The amount of reference data needed to create effective deepfakes has decreased, making them easier to produce.

Arguments for Stricter Regulation:

Argued that deepfakes should generally be prohibited, except for specific legitimate uses, given their potential for misuse.

Disappointment that the AI Act does not more aggressively regulate deepfakes despite the known risks.

Potential Advantages of Deepfakes:

• Creative Expression: Deepfakes can facilitate artistic and creative endeavours, such as memes and digital art.
• Education and Training: They could enhance educational experiences by featuring historical figures or simulations for training purposes.
• Entertainment: Deepfakes may allow actors to license their likeness or voices for various productions.
• Investigative Uses: Law enforcement could potentially use deepfakes to assist in criminal investigations.

Disadvantages and Risks:

• Misinformation and Manipulation: The use of deepfakes can lead to widespread misinformation, scams, and fraud, particularly affecting banking and personal security.
• Non-consensual Explicit Content: Deepfakes can be misused to create nonconsensual explicit materials, raising significant ethical concerns.

Regulatory Gaps:

• There is frustration over the EU’s approach to regulating deepfakes, as the current provisions may not effectively address the potential for harm.
• Why have more direct and comprehensive regulations not been established, especially given the known issues related to deepfakes in social media and other platforms?

19. General purpose AI models

1. Lack of Classification Rules:

Theres a surprising absence of specific classification rules for general-purpose AI in Title Five of the AI Act, highlighting that only AI with systemic risk is classified.

2. Definition of General Purpose AI:

According to Title One, a general-purpose AI model is defined as an AI model that:

• Is trained on a large amount of data using self-supervision at scale.
• Demonstrates significant generality and can competently perform a wide range of distinct tasks.
• Can be integrated into various downstream systems or applications.

3. Generality and Capability:

Familiar to AI tools like ChatGPT, which can handle diverse tasks such as translation, writing, diagnostics, and role-playing in job interviews, illustrating the model’s versatility.

4. Foundational Models:

There is a distinction between generative AI like ChatGPT and foundational models. Foundational models serve as a base for building applications, rather than being the final product themselves.

5. Self-Supervision in Learning:

The concept of self-supervised learning is discussed, distinguishing it from supervised and unsupervised learning:

• Supervised Learning: Involves training with labeled data where a target feature is present (e. g. , predicting loan repayment).
• Unsupervised Learning: Focuses on finding patterns in data without a specific target feature (e. g. , clustering customer data).
• Self-Supervised Learning: Involves creating tasks where the model learns from its predictions based on the structure of the data itself, like predicting masked words in a sentence.

6. Examples of Self-Supervised Learning:

Examples illustrate how self-supervised learning works:

• Predicting masked words in text.
• Predicting the next frame in a video sequence.
• Filling in missing portions of images.

7. Generative Models:

Models like ChatGPT, video generators like Sora, and image generators like Midjourney are identified as general-purpose AI models due to their ability to perform a range of tasks, use self-supervised learning, and integrate into various applications.

8. Exceptions for Research and Development:

The definition includes exceptions for AI models used in research, development, or prototyping that are not yet available on the market. Such models do not qualify as general-purpose AI until they are commercially released.

20. Obligations for providers of GPAI:

Article 53 outlines the obligations for providers of general-purpose AI, emphasising transparency, accountability, and compliance with legal standards. Here are the key points:

1. Technical Documentation and Evaluation: Providers must maintain up-to-date technical documentation for their AI models, covering training and testing processes, as well as evaluation results. This ensures transparency and accountability, verifying that the model meets safety and ethical standards.

2. Information Sharing with AI System Providers: Providers need to prepare documentation that can be shared with other AI system providers integrating their general-purpose AI models. This information should clarify the model’s capabilities and limitations, similar to obligations for high-risk AI.

3. Compliance with Copyright and Related Rights: Providers must implement policies that ensure compliance with EU copyright laws, particularly concerning data used in training their AI models.

4. Transparency About Training Data: Providers are required to create and publicly share a detailed summary of the content used for training the AI model, following a specific template from the AI office. This addresses concerns about data transparency, particularly for major companies hesitant to disclose such information.

5. Exceptions for Open-Source Models: Open-source AI models are exempt from certain obligations if they make their models, parameters, architecture, and usage information publicly available.

6. Compliance and Cooperation: Providers must cooperate with authorities and can use established codes of practice to demonstrate compliance until harmonised standards are available.

7. Confidentiality and Data Protection: Any information obtained during compliance processes, including trade secrets, must be treated confidentially in accordance with EU data protection regulations.

This section discusses the classification of general-purpose AI models that may qualify as general-purpose AI with systemic risk, similar to high-risk AI. Here are the key points:

1. Classification Criteria: A general-purpose AI model can be classified as one with systemic risk if it meets any of the following conditions:

• High Impact Capabilities: The model has significant capabilities assessed through technical tools and methodologies, indicated by a fixed threshold. Specifically, if the cumulative computation used for its training exceeds 1.050625×1031 floating point operations, it is presumed to have high impact capabilities. Most contemporary large AI models, like those from OpenAI or other major providers, likely surpass this threshold.

2. Commission Decision: A model may also be classified based on a decision by the European Commission or following a qualified alert from a scientific panel, even if it doesn’t meet the first condition. This decision would consider criteria outlined in Annex 13, which includes:

• The number of parameters in the model;
• The quality or size of the training dataset;
• The amount of computation used for training;
• The input and output modalities (e. g. , text, voice, video); and
• Benchmarks, including the number of registered end users.

3. Broad Interpretation: The EU has considerable discretion to classify various general-purpose AI models as having systemic risk, even if they do not strictly meet the first condition. This broad interpretation raises concerns that many current AI models, particularly those from major providers like OpenAI and Meta, may qualify under this classification.

4. Future Flexibility: The criteria for classification can be adjusted and expanded by the European Commission, allowing for evolving standards and considerations regarding systemic risk.

AI regulatory sandboxes are frameworks established within the European Union to foster innovation in artificial intelligence by allowing for the supervised development, testing, and validation of AI systems before they enter the market. Originating from the fintech sector, these sandboxes are designed to create a controlled environment for providers, particularly those working with high-risk AI.

21. Key Features of AI Regulatory Sandboxes:

1. Purpose and Structure:

Sandboxes serve as a controlled environment for AI providers to test innovative systems. Each EU country will establish an office that acts as a sandbox, staffed by AI experts, data scientists, and legal consultants.

2. Focus on Providers:

The sandboxes are primarily designed for providers of AI systems, such as banks, rather than deployers. Deployers may only need to inform the sandbox about the provider of their AI system.

3. Support for SMEs:

To encourage innovation, there will be exceptions for small and medium-sized enterprises (SMEs), such as reduced paperwork and fast-track approval processes.

4. Testing Approaches:

Providers can either propose real-world experiments for approval or conduct testing within the sandbox. Proposals require extensive documentation, with a waiting period for approval before experiments can proceed. Testing in the sandbox may involve anonymizing customer data or creating synthetic data, as real customer data cannot be used.

5. Database of High-Risk AI Systems:

Sandboxes will maintain a semi-public EU database of high-risk AI systems, accessible to regulatory bodies but with restrictions for the public.

This raises concerns about trade secrets, as competitors may gain insights into proprietary AI applications.

22. Summary of Challenges in Establishing AI Regulatory Sandboxes

1. Talent Shortage:

There is a significant lack of experienced talent for managing AI sandboxes compared to fintech sandboxes. While the finance sector has a large talent pool, AI expertise is limited, making it challenging to recruit qualified individuals.

2. Broad Definitions:

The broad definition of AI may lead to an overwhelming number of applications and documentation submissions to the sandboxes. The potential for too many use cases could strain the regulatory bodies’ capacity to handle the workload efficiently.

3. Steep Learning Curve:

Both industries and regulators will face a steep learning curve as they adapt to the new regulatory landscape. There is a need for clear expectations from companies, and regulators must learn to implement regulations effectively, which can lead to inefficiencies.

4. Ecosystem Impact:

The focus on providers might lead companies to outsource AI development rather than maintaining in-house capabilities, potentially harming the AI ecosystem in the EU. Companies may prefer to rely on external vendors to avoid extensive regulatory burdens.

5. Time Constraints:

There is insufficient time to establish these sandboxes before the enforcement of regulations. The timeline from the approval of the act to the implementation of prohibited practices is tight, raising concerns about the readiness of the sandboxes.

6. Need for Experiments:

The concept of conducting real-world experiments within the sandbox framework may face resistance. Many industries in the EU traditionally rely on historical data rather than experimentation, making it difficult to adapt to a new expectation of conducting regulated experiments with AI.

23. Section 2: Requirements for HighRisk AI Systems (Summary)

Article 8: Compliance with Requirements

High-risk AI systems must meet specified requirements, reflecting the latest AI technological standards and their intended purpose. Providers of such systems must ensure compliance with all relevant EU laws and may streamline compliance efforts with existing regulatory requirements to minimize redundancies.

Article 9: Risk Management System

Providers must implement a continuous, iterative risk management system for highrisk AI systems throughout the system’s lifecycle. Key steps include:

• Risk Identification and Analysis: Identifying foreseeable risks to health, safety, or fundamental rights.
• Risk Estimation and Evaluation: Assessing risks from intended or reasonably misused purposes.
• Post-Market Monitoring: Evaluating new risks based on operational data.
• Targeted Risk Management: Implementing measures to eliminate or reduce these risks, including design adjustments, mitigative actions, and user training. Testing must ensure risk management aligns with intended functions, and real-world testing may be required. Providers must also consider potential impacts on vulnerable populations, like minors.

Article 10: Data and Data Governance

Data used for high-risk AI must meet specific quality standards, covering:

• Data Integrity and Representativeness: Ensuring data is complete, representative, and appropriate for the AI’s purpose.
• Bias Prevention: Measures to detect and mitigate potential biases are required to prevent harmful impacts on safety or fundamental rights.
• Special Data Use: Personal data processing is permitted for bias correction under strict security measures, including pseudonymisation and deletion upon reaching the data’s purpose limit.

Article 11: Technical Documentation

Comprehensive technical documentation must be prepared before market release, proving compliance with EU requirements and updated throughout the lifecycle. Simplified forms are allowed for small and medium enterprises (SMEs), with the aim to streamline their regulatory burden.

Article 12: Record-Keeping

High-risk AI systems must automatically record relevant events (logs) to support traceability, allowing monitoring of usage and maintenance and aiding in identifying issues post-deployment.

Article 13: Transparency and Provision of Information

Systems must have transparency features to help deployers understand and interpret AI outputs. Information provided should cover the system’s purpose, limitations, accuracy, and maintenance requirements, helping users to responsibly deploy the AI system.

Article 14: Human Oversight

Systems must facilitate human oversight, especially where AI outputs may affect health, safety, or rights. Users should be able to understand and monitor AI functioning, avoid “automation bias,” and intervene as needed. In certain cases, two independent verifications of AI outputs are required for critical decisions.

Article 15: Accuracy, Robustness, and Cybersecurity

High-risk AI systems must be designed for consistent accuracy and resilience against errors, environmental changes, and unauthorised modifications. Cybersecurity measures must address vulnerabilities specific to AI, including protections against data and model manipulation and attacks exploiting AI weaknesses.

24. Summary of Article 50: Transparency Obligations for Providers and Deployers of Certain AI Systems

1. Informing Users about AI Interaction:

AI system providers must ensure that users interacting with AI are informed they are engaging with an AI system unless it’s apparent to a reasonable person. Exceptions are granted for legally authorised systems used in criminal justice activities, provided they safeguard third-party rights.

2. Labelling Synthetic Content:

Providers of AI systems that generate synthetic content (text, image, audio, or video) must label outputs as AI-generated or manipulated in a machinereadable way. Solutions should be effective and interoperable within technological limits and standards. Exceptions include minor content edits, functions that do not significantly alter input data, and lawful uses in criminal justice.

3. Transparency in Emotion and Biometric Recognition Systems:

Deployers using emotion recognition or biometric categorisation must notify individuals about the system’s operation, adhering to applicable data protection regulations. Exceptions apply for legally permitted systems in criminal justice.

4. Disclosure of Deepfake Content:

AI-generated or manipulated media, such as deepfakes, must be disclosed as such by deployers, except in legally authorised criminal justice contexts. In cases of artistic, satirical, or fictional works, disclosure should respect the content’s creative integrity. Text created for public information requires disclosure unless human-reviewed or editorially controlled.

5. Accessibility of Information:

Information from the previous sections must be clearly accessible and provided at the initial interaction with the AI, following accessibility standards.

6. Non-Prejudice to Other Transparency Laws:

These requirements supplement, not override, other transparency obligations in EU or national laws on AI.

7. Support for Compliance through Codes of Practice:

The AI Office will promote codes of practice for effective compliance with transparency and labelling standards, and the EU Commission may approve or, if needed, specify rules through implementing acts.

25. Summary of Chapter V: GeneralPurpose AI Models

Section 1: Classification Rules

Article 51 – Classification as Systemic Risk:

General-purpose AI models (GPAI) are classified as having “systemic risk” if they demonstrate high-impact capabilities based on established benchmarks or if the European Commission designates them as such. Models are presumed highimpact if their training used over 10^25 floating point operations. The Commission can adjust thresholds based on technological progress.

Article 52 – Notification and Designation Process:

Providers of GPAI that meet high-impact criteria must notify the Commission promptly, presenting any reasons they believe their model does not pose systemic risk. If the Commission disagrees, the model will be classified as systemic risk. The Commission can also designate a model as systemic risk on its initiative or in response to alerts from scientific panels. Providers may request reassessment six months after designation, provided they submit new, objective reasons. A public list of GPAI models with systemic risk will be maintained, ensuring intellectual property and confidentiality protections.

Section 2: Obligations for Providers of General-Purpose AI Models

Article 53 – Provider Obligations:

Providers must maintain up-to-date technical documentation about model training and testing, and provide resources to help other providers understand and comply with GPAI regulations. They must also follow copyright laws and release a summary of training content. Open-source models are exempt from some requirements unless classified as systemic risk. Providers may use approved codes of practice or harmonised standards to meet these obligations.

Article 54 – Authorised Representatives:

Non-EU providers must appoint an EU-based representative before releasing a GPAI model in the EU. This representative will ensure the provider’s compliance with technical and documentation requirements for ten years. Open-source models are generally exempt unless they present systemic risks.

Section 3: Additional Obligations for Systemic Risk Models

Article 55 – Systemic Risk Compliance:

Providers of systemic-risk GPAI models must conduct advanced evaluation, mitigate EU-level risks, and document any incidents. They must also implement cybersecurity measures for both the model and its infrastructure. Compliance can be demonstrated via codes of practice or harmonised standards. Section 4: Codes of Practice

Article 56 – Development and Monitoring of Codes:

The AI Office will support the development of codes of practice to guide GPAI compliance and will ensure codes cover systemic risk obligations. Stakeholders, including providers and civil society, may participate in drafting. Codes must have clear goals and track performance. The Commission may validate codes, making them EU-wide standards. If codes are not developed by August 2025, the Commission will establish implementation rules to ensure compliance with Articles 53 and 55.

26. Download the FREE eBook