Do We Still Have AI Under Control?
The most dangerous moment may not come when artificial intelligence rebels against humans. It could come much sooner – the moment humans begin voluntarily handing over decision-making power to it for matters they no longer understand themselves. Just a few years ago, we were asking what AI was capable of. Today, we should ask ourselves a more uncomfortable question: Do we still know what it’s doing, why it’s doing it, and how to stop it if it starts doing something we don’t want? This isn’t science fiction. In an extraordinarily short time, AI has moved from laboratories into the economy, government, the media, intelligence, cybersecurity, industry, and the military. It increases productivity and the speed of decision-making, but at the same time creates a paradox: the more capable our systems are, the more we need them – and the more we need them, the harder it is to retain the option of not using them. The question, therefore, may not be whether AI will take control of humanity. A much more pressing question is whether humanity is giving up control of its own accord.
From Tool to Actor
A hammer does not decide on its own which nail to drive in, and traditional software generally does not generate its own sequences of steps outside a predefined procedure. With modern AI, this boundary is becoming blurred. A model can analyze its environment, create a plan, use additional tools, communicate with other systems, check results, and continue without further human intervention. With agent-based and multi-agent systems, a qualitatively different situation arises: a human sets the goal and leaves it to the machine to find the path. This is precisely the source of enormous potential – and, at the same time, risk. Control, after all, is not binary. It can fade away gradually: first, we delegate routine tasks; then analysis, decision-making, and process optimization; and finally, we merely formally approve the option generated by the machine. On paper, the “human in the loop” remains; in reality, however, we may end up with a person who merely rubber-stamps the AI’s decisions. The presence of a human in the chain does not, in and of itself, guarantee human control. If the person does not understand the model, does not have time to verify the output, lacks an alternative source of information, or has learned to automatically trust the machine, human oversight may be nothing more than an administrative fiction.
Numbers We Wouldn't Tolerate with Any Other Technology
The underlying analysis, based on a Delphi study of 272 international experts, relies on probabilistic estimates rather than empirically measured frequencies of future disasters. Nevertheless, the magnitude of these estimates is alarming. Assuming current trends continue, experts estimated that by 2030, the probability of a catastrophic outcome would be 22% for dangerous AI capabilities, 21% for weapons and cyberattacks utilizing AI, 18% for the centralization of power, 17% for competitive dynamics and AI discord, 16% for security vulnerabilities, 14% for governance failures, and 13% for disinformation and the loss of a shared reality. Even after implementing pragmatic measures, the risk does not disappear: for dangerous capabilities, weapons, and cyberattacks, the estimate remains at 12%. The conclusion is unpleasant but strategically important: zero risk is not a realistic goal. The purpose of risk management must be to reduce it to a tolerable level and to develop the ability to manage residual risk. In the case of a nuclear power plant, a dam, a commercial aircraft, or a new weapons system, we would not consider the probability of a catastrophic outcome in the range of single-digit or double-digit percentages to be a minor technical problem. With AI, however, the global race for ever-more-capable systems continues at a faster pace than the creation of institutions capable of managing their risks.
A Machine Uprising May Not Be the Real Risk
In public debate, the focus tends to be on superintelligence that spirals out of control. However, AI does not need to become conscious or “hate humanity” to cause extraordinary harm. It is enough for it to be sufficiently capable, to be given a flawed objective, to be connected to a critical system, for a person to trust it too much – or for another person to use it against us. The report distinguishes between misuse, failure, misalignment, structural dynamics, and systemic emergence. Thus, the single term “AI risk” encompasses several distinct mechanisms: a person using AI as a weapon; a system that makes a mistake; a system that correctly optimizes something other than what we actually intended; a race among companies and nations in which no one wants to be the first to slow down; and, finally, the collective behavior of a multitude of autonomous systems that no one has explicitly programmed. Therefore, the greatest danger may not be a single superintelligence. It could be an ecosystem of thousands of interconnected intelligent systems, for whose overall behavior no one will be accountable.
Humans May Be AI's Greatest Weakness
A less dramatic but perhaps more likely scenario for losing control arises from human dependence. If an algorithm consistently provides higher-quality analyses over the long term, the pressure to accept its recommendations grows. The more often it is correct, the less motivation there is to monitor it; the less we monitor it, the more we lose our own ability to perform the task. Delegation leads to a loss of competence, and a loss of competence leads to further delegation. In security, crisis management, and the military, this mechanism is crucial. Imagine a commander for whom AI evaluates thousands of sensor inputs within seconds and recommends a target, while only tens of seconds remain to make a decision. Formally, the commander makes the decision. In reality, however, he may be merely the final link in a process whose analysis he is no longer able to reproduce independently. This is not an argument against military AI. An army that does not use it may lose its decision-making tempo. A security dilemma arises: we cannot afford not to control AI, but at the same time, we may not be able to afford not to use it.
A Race in Which No One Wants to Be the First to Slow Down
AI has become an economic, military, and geopolitical asset. Companies compete for customers and capital; nations compete for technological superiority; militaries compete for faster decision-making cycles; and intelligence agencies compete for an information advantage. Each individual actor has a rational reason to move quickly, but the result may be systemically irrational behavior. A company that slows down due to security testing may lose market share. A nation that restricts the military use of AI may lose its strategic advantage. An army that allows humans more time to make decisions may lose to an adversary that has automated its decision-making process. No one may want uncontrolled AI, yet everyone may collectively be heading in that direction. That is why competitive dynamics and governance failures are “upstream” risks: pressure to deploy quickly weakens testing, oversight, and the willingness to regulate, and increases the likelihood of further failures. Controlling AI is thus not just a technical problem. It is a problem of collective action and strategic stability.
Regulation Alone Won't Save Us
Europe tends to respond to technological risks with legal regulation. While this is necessary, the law alone cannot create an expert capable of auditing the most advanced models, red teams, computing infrastructure, or digital sovereignty, nor can it ensure the ability to detect hostile operations. Nor can it guarantee that a person will reject an algorithm’s recommendation in a crisis situation. That is why defense in depth is key: the security of the model and its development, the security of deployment and operation, resilient infrastructure, governance and regulation, and, finally, societal resilience. The goal is not to assume that a single safeguard will never fail, but to create a system in which the failure of a single safeguard does not lead to a catastrophe. We apply the same principle in nuclear energy, aviation, and military systems. When it comes to AI, there is no reason to be any less demanding.
Kontrola znamená také schopnost vypnout
There is a simple test of true control over technology: can we continue without it? If not, our control is limited. A country whose critical decision-making processes depend on foreign models, the cloud, chips, and data infrastructure may not have true technological sovereignty, even if it owns the end-user applications. A military that cannot perform a key function without AI has no backup. An institution whose employees are no longer able to independently perform a task delegated to an algorithm has not merely created automation – it has created dependency. True control therefore requires not only “human in the loop” but also “machine out of the loop”: a secure degraded mode, backup procedures, segmentation, the ability to isolate the system, and, in critical applications, rapid shutdown. Otherwise, the OFF button exists only in theory.
The Year 2030: The Intelligence of Our Institutions Will Be the Deciding Factor
The underlying analysis considers three possible futures: controlled adaptation, in which security and regulatory capacities keep pace with technology; erosion of control, in which the race continues and institutions fall behind; and a shock in the form of a large-scale cyberattack, a disinformation crisis, or an incident involving dangerous AI capabilities. The difference between these scenarios may not be determined by the intelligence of the models themselves. It may be determined by the intelligence of our governance system. It is therefore crucial to build coordination, audit, and oversight capacities; a secure framework for the use of AI in the defense sector; systematic training; crisis exercises; and, gradually, greater digital sovereignty. This is not a fight against AI. A state that fails to manage its risks will ultimately be unable to safely reap its benefits either.
So, Do We Still Have AI Under Control?
The answer depends on what we mean by “control.” If it refers to the ability to write a prompt, receive a response, and close the application, then yes. However, if control means the state’s ability to know all the places where AI is being used; the military’s ability to utilize it without losing human judgment; the ability of a regulator to oversee systems more complex than the institution overseeing them; the ability to detect a machine-led attack at the speed of machines; the ability to function during a failure of foreign infrastructure; and, above all, the ability to shut down the technology before it causes irreversible consequences – then the answer is less reassuring. AI is not yet beyond our control, but control is no longer a given. The greatest danger may not be the moment when artificial intelligence gains a will of its own. It may be the moment when we realize that we still have a will of our own – but that we have gradually lost the ability to assert it. The AI revolution has a peculiar characteristic: the tool we are supposed to control is simultaneously becoming the tool through which we control an ever-larger part of the world. The question “Do we still have AI under control?” is therefore not just a question for computer scientists. It is a question for the government, the security community, the military, intelligence agencies, regulators, businesses, and society at large. And perhaps we should add one more question: will we even recognize the moment when we begin to lose that control?
