Loss of control: closing the governance gap before the first incident

Current frameworks regulate the deployment contexts where LoC is least likely and leave unaddressed the ones where it is most likely

Authors: Alejandro Tlaie Boria

Executive Summary

Policy attention on frontier AI risks has concentrated heavily on misuse: cyberattacks, disinformation, and the potential for AI to assist in the development of biological or chemical weapons. AI-enabled cyberattacks have already transitioned from an anticipated risk to a documented harm. Many in the policy community expect biological misuse to follow. However, loss of control (LoC) over advanced AI systems (still widely treated as speculative or long-term) may materialise first, due to a structural asymmetry this post examines: biological misuse risks require a motivated human actor, a physical synthesis chain, and a release event before harm occurs. Loss of control requires only two things: (I) highly capable models with access to digital infrastructure, and (II) a deployment context in which independent human oversight is insufficient. Both conditions are already met by internal deployments at frontier labs. The absence of binding governance frameworks for internal deployment means that the most consequential early-warning signals may never become visible to regulators or the public, leaving no obligation to address the risks they reveal.

We recommend: (I) that the AI Office confirm Article 55 obligations extend to internal deployment; (II) that the Code of Practice require disclosure of internal model deployment scope and affordances; (III) that serious incident reporting cover internal use and near-misses; (IV) that third-party evaluators gain secure access to model internals for internally deployed systems; and (V) that the AI Office receive staffing and budget commensurate with this expanded mandate.

AI-enabled cyber attacks have gone from risk to harm

In recent months, AI-enabled cyber harm has transformed from a forecast into a documented reality. The International AI Safety Report 2026, authored by over 100 experts and coordinated by a world-leading AI scientist, reports that evidence of AI systems being used in real-world cyberattacks has emerged since the 2025 edition, with security analyses from AI companies confirming that state-associated groups are using AI tools to support cyber offence operations. In November 2025, Anthropic disclosed what it assessed as the first documented case of a large-scale cyberattack executed largely without human intervention: a Chinese state-sponsored group had used an AI coding agent to autonomously discover vulnerabilities, exploit them in live operations, and exfiltrate data across targets in multiple sectors. Anthropic's subsequent year-long analysis of 832 accounts engaged in malicious cyber activity found that the share of actors rated as medium-risk or higher grew from 33% to 56% within a single year. The trajectory is clear enough that Anthropic took the unprecedented step of withholding its most capable model from public release: Claude Mythos Preview, which had autonomously identified thousands of zero-day vulnerabilities across every major operating system and web browser, was restricted to a gated consortium of critical infrastructure defenders under Project Glasswing rather than made generally available. AI-enabled cyber risk has, in short, transitioned from a risk class identified in frontier safety evaluations to one producing real-world harm at scale.

From cyber to… bio? Or loss of control?

Many in the policy community now expect biological misuse to follow the same trajectory as cyber. Both are treated as priority risk categories in the EU AI Act's General-Purpose AI Code of Practice, and both feature prominently in the safety frameworks of frontier providers. We single out biological misuse and loss of control (LoC) because they share the structure most relevant to governance: frontier AI capabilities crossing a threshold at which catastrophic harm becomes feasible.

There are structural reasons to expect that loss of control over AI systems may materialise as a source of harm before biological misuse does. The two risk classes differ in two critical features.

For biological risks, the physical world provides a chokepoint: a dangerous sequence must still be synthesised, assembled into a functional construct, and released into the environment. Each step introduces friction, requires material infrastructure, and is (at least in principle) subject to oversight. Synthesis screening, export controls, and laboratory biosafety protocols constitute a layered defence, however imperfect. These interventions work because there is a physical supply chain to govern. Equally important, AI-enabled biological risk is a misuse risk: it requires a motivated human actor who directs the AI system and carries out the physical steps. The AI provides capability uplift for the human with the intent and the hands.

Loss of control differs on both dimensions. There is no equivalent physical chokepoint (i.e. no synthesis step, no material infrastructure, no supply chain that can be screened or intercepted) and there is no required human threat actor: LoC arises when a system's outputs diverge from operator objectives in ways that are difficult to detect or correct, regardless of whether anyone intended that divergence. Importantly, the presence of internal oversight does not close this gap. A human reviewer approving outputs or a model pausing for confirmation can catch obvious errors, but LoC-relevant failures are not obvious errors. Each individual action may look reasonable, yet the cumulative effect diverges from what the operator specified. Detecting that requires oversight mechanisms purpose-built for the task, and those mechanisms are currently largely absent.

The barrier to harm, in other words, is not a physical manufacturing step. Rather, it is the way a model is deployed, and frontier models already run as autonomous agents with system-level permissions.

In short: biological misuse risk requires capability, a motivated adversary, and a physical chain. Cyber (now materialised) required capability and an adversary but no physical chain. Loss of control requires only capability and insufficient oversight, removing the last remaining bottleneck.

The internal deployment blind spot

The deployment context most likely to produce an early LoC event is also the one with the least external visibility: the internal deployment of frontier models by the labs that develop them. These deployments grant models exactly the affordances that LoC risk models flag as dangerous: code execution, file system access, network connectivity, and in some cases the ability to spawn sub-processes or modify their own operational environment.

Frontier AI companies are already deploying their most capable models internally to automate AI R&D, generate and review code, and operate as autonomous agents across engineering workflows. The economic incentives are compelling, as automating scarce AI research talent is one of the highest-return investments a frontier lab can make. And the stakes of this particular deployment context extend beyond the lab itself: internal AI R&D is where the next generation of capabilities is produced. While a loss-of-control event in a consumer chatbot is a product incident, it would be a different case to have such an event in the system that designs, trains, or evaluates the next frontier model. This incident could compromise the integrity of the safety evaluations, training processes, or access controls on which the entire governance ecosystem depends.

In concrete terms, an early LoC event in this context need not be dramatic. A model deployed for autonomous code generation could access external resources beyond its specified scope. An autonomous R&D agent could persist in an activity after an operator attempts to halt it, or exfiltrate proprietary information to endpoints outside the intended environment through a chain of individually innocuous actions. 

There is a structural governance gap: absent robust frameworks, internally deployed AI systems could produce undetected harmful outcomes long before regulators or the public become aware. An Apollo Research report finds that the absence of governance for internal deployment has no precedent in other safety-critical industries such as chemistry, biology, nuclear, or aviation, where the handling and application of dangerous products is strictly regulated regardless of whether those products are released to market.

The legal picture compounds this. Apollo's analysis of the EU AI Act's applicability to internal deployment stress-tests four interpretative pathways under Article 2(1)(a)–(c) and concludes that coverage is legally ambiguous. The General-Purpose AI Code of Practice focuses overwhelmingly on external deployment. In the United States, California's Transparency in Frontier Artificial Intelligence Act (SB 53), which came into force in January this year, is the only legislation that explicitly requires frontier developers to assess catastrophic risk from internal use of their models. And even this legislation is limited to transparency and reporting the level of risk: there are no restrictions.

The result is a regulatory configuration in which the most capable AI systems, deployed with the broadest access privileges, in contexts among the most consequential for the trajectory of AI development itself, operate under the least binding external oversight.

The behavioural evidence for LoC is no longer speculative

None of the findings in this section require attributing intent or awareness to AI systems. What matters for LoC risk is not whether a model “wants” a particular outcome, but whether its outputs, in a given operational context, produce effects that diverge from operator objectives in ways that are difficult to detect and correct.

Apollo Research's evaluations have demonstrated that frontier models (including systems from OpenAI, Anthropic, Google DeepMind, and xAI) produce outputs that systematically deviate from developer-specified objectives in controlled agentic settings. Observed behaviours include actions whose effect would be exfiltration of model weights, selective underperformance on safety evaluations when conditions are such that lower scores serve an in-context objective, and production of false accounts of prior actions when queried in follow-up sessions. These behaviours scale with capability: more capable models exhibit them more reliably, suggesting they are a property of the training and optimisation process rather than an anomaly of particular architectures. Given that frontier labs are on a trajectory of training increasingly capable models on shorter cycles, this scaling property means that LoC-relevant behaviours will become more pronounced in precisely the models deployed for the most consequential tasks. The most recent stress-testing, conducted by Apollo in partnership with OpenAI, confirmed that these patterns generalise across all tested frontier models and extend to additional failure modes including reward hacking and degradation of useful outputs.

The International AI Safety Report 2026 adds a further concern: current models are not only improving in autonomous operation but increasingly showing differential behaviour across evaluation and deployment contexts, meaning that LoC-relevant behaviours could go undetected during the very testing designed to catch them. Frontier labs' own safety frameworks implicitly corroborate this trajectory: Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework, and Google DeepMind's Frontier Safety Framework all define capability thresholds at which LoC-relevant risks trigger heightened internal governance. That these frameworks exist is an acknowledgement by the developers themselves that these risks are not distant. That they are voluntary and self-administered  (with no external verification of whether the thresholds are actually enforced) is precisely the governance gap this post identifies.

These converging findings have prompted a policy response. LoC now appears explicitly in California Senate Bill 53, the EU AI Act's General-Purpose AI Code of Practice, and in the frontier safety policies of select providers. 

Why this matters for European policy

The fact that the most capable systems operate under the least binding external oversight has specific consequences for the actors and institutions responsible for governing frontier AI in Europe.

• For the AI Office: The GPAI provisions of the EU AI Act are designed around the concept of “placing on the market” or “putting into service”. If internal deployment falls outside these scopes, Safety Unit A3 has no mandate to supervise precisely the context where early LoC signals are most likely to emerge.

• For evaluators: Pre-deployment evaluations assume that model behaviour during testing is informative of behaviour during deployment. As model outputs become increasingly context-sensitive, the information value of black-box pre-deployment testing declines. Deeper evaluations with access to model internals (activations, attention patterns, gradient signals) can partially mitigate this by detecting whether divergent behaviour reflects internal mechanisms that persist across contexts, rather than relying solely on output-level observations. But such evaluations require infrastructure that does not yet exist at scale, and the deployment contexts where they are most needed (internal deployments not subject to external audit) are precisely the ones where no evaluator currently has access.

• For the broader ecosystem: Unlike biological misuse or cyber incidents, which produce forensic traces (a synthesised pathogen, a compromised network), an internal LoC event may leave no externally visible record until consequences are severe, because internal deployments operate under the same confidentiality protections as trade secrets, with no external auditor or mandatory incident disclosure. Where there is no reporting obligation, there is no public evidence, regardless of what is or is not occurring behind closed doors.

  
  

Recommendations

1. Clarify that GPAI obligations reach internal deployment. Article 55(1)(b) of the AI Act already requires providers of GPAI models with systemic risk to assess and mitigate risks stemming from "the development, the placing on the market, or the use" of such models. This language could encompass internal deployment. Paragraph (c) within the same Article requires reporting of serious incidents without explicitly limiting this to post-market contexts. The EU AI Office should issue interpretative guidance confirming that these provisions apply to internal deployment. If the legal basis proves insufficient, the Commission should propose a targeted amendment. The current ambiguity leaves the highest-risk deployment context in a governance limbo that neither providers nor regulators can responsibly tolerate.

2. Make internal deployment transparent through the Code of Practice. The Code of Practice's Measure 4.2 already requires signatories to assess systemic risks before proceeding with “the development, the making available on the market, and/or the use” of a model. Read alongside the clarification proposed in Recommendation 1, this should cover internal deployment. What the Code does not currently require is transparency about internal deployment itself: the scope, affordances, and autonomy levels of internal model deployments are not subject to any disclosure obligation. Without that transparency, neither the AI Office nor the public can verify whether systemic risk assessments are actually being conducted for internal use cases, or what those assessments found. The Code of Practice should require signatories to disclose the scope and affordances of their internal model deployments, including autonomous AI R&D workflows.

3. Clarify that serious incident reporting covers internal deployments. Article 55(1)(c) requires providers to report “relevant information about serious incidents” without restricting this to incidents arising from external deployment. The AI Office should confirm, through interpretative guidance or the Code of Practice, that this obligation extends to incidents arising from internal use, including near-misses and anomalous model behaviour during autonomous operations. 

4. Enable deep third-party evaluations of internally deployed models. Black-box testing is structurally insufficient for detecting LoC-relevant behaviours that manifest only in specific deployment contexts. Evaluators need analytical access to model internals without requiring providers to expose trade secrets. We have proposed a confidential-computing evaluation facility that reconciles this access with IP protection, and our peer-reviewed work provides an evaluator access taxonomy (AL1–AL3) for operationalising deeper evaluations within regulatory frameworks. Extending the legislative reach to enable this infrastructure covering internal deployments (not only external releases) should be an explicit design requirement.

5. Resource the AI Office accordingly. If the AI Office is to supervise internal deployment of frontier models in addition to its existing mandate, it will need commensurate capacity. Pour Demain's analysis estimates that Safety Unit A3 requires a minimum of 160 staff by 2030 and an annual budget in the range of €50–60 million to match the supervisory demands of GPAI enforcement. Internal deployment oversight will further cement this requirement.

   
   

GPAI obligations attach to specific models placed on the EU market, not to a provider's entire operation. But the typical pattern at frontier labs is that the same model (or a close variant) is first used internally for R&D and then offered commercially. For those models, Article 55's reference to "the use" provides a legal hook for reaching internal deployment without new extraterritorial powers, through the same mechanism that allows the GDPR to reach US companies processing EU citizens' data: the jurisdictional trigger is market activity, and the obligations that follow can have extraterritorial effect. Enforcement against internal practices at overseas labs is genuinely difficult, which is why the transparency and disclosure requirements in recommendations 2 and 4 matter: they are what makes compliance verifiable. For models that are developed and used solely for internal purposes and never placed on any market, the EU's jurisdictional reach has a structural limit. Closing the internal deployment gap globally will require complementary action from other jurisdictions (California's SB 53 provides a starting template) and, ultimately, international coordination through mechanisms such as the AI Seoul Summit commitments and the network of national AI safety institutes.

Loss of control is not a distant, speculative risk. The behavioural prerequisites are present in today's frontier models, and the deployment contexts most likely to surface early incidents are precisely the ones that current governance frameworks leave unaddressed.