Tom Shea, CEO, OneStream Software.
Recently, the people building the most powerful AI models told us that AI could end humanity if we are not careful. Anthropic’s Dario Amodei called for the industry to slow the development of frontier systems. Sam Altman and Elon Musk backed him.
They are right to raise the alarm on catastrophic risk, but their proposed remedy—slowing down the frontier labs—misses where the most immediate danger lies. The reality is the risk is already here, and organizations everywhere are amplifying it by connecting today’s models to the systems that run their businesses.
Think about an LLM as the frontal lobe of the human brain. By itself, it can reason, but it cannot accomplish much in the physical world.
Now connect that frontal lobe to a nervous system and a body. Through Model Context Protocol (MCP), APIs, agentic frameworks, databases, enterprise applications, operating systems, industrial systems and eventually physical machines, we have already been giving AI models the ability to observe, decide and act, changing the risk equation completely.
The right way to think about AI risk is in three distinct categories—errors of commission, errors of omission and errors of ignorance—and they require a unique control system to minimize that risk.
Errors Of Commission
Errors of commission involve the intentional misuse of AI. Hackers, criminal organizations, hostile governments and other bad actors can use LLMs to dramatically increase their technical capabilities.
This is where the current regulatory conversation runs into a structural problem. Most of the proposals have been designed to govern how frontier labs build and release their models. That is reasonable, but it assumes the danger is contained inside the companies being regulated.
Open-weight models already exist throughout the world. The regulatory debate cannot focus exclusively on controlling the intelligence inside the model. We have to focus, with equal urgency, on what that intelligence is allowed to touch and control.
Errors Of Omission
The second category may ultimately represent an even larger systemic risk: errors of omission, danger created not through malicious intent, but because we failed to build appropriate controls around increasingly autonomous software.
Millions of developers are racing to add agentic capabilities to their products, and MCP is making it increasingly easy to connect LLM reasoning directly to system functionality. That raises an urgent question: Are we prioritizing the speed of agentic access over the control of it?
No automobile manufacturer would connect a self-driving algorithm to steering and braking without extensive testing and human override, yet portions of the software industry are moving toward the digital equivalent, giving probabilistic reasoning systems access to deterministic systems that move money, alter infrastructure and execute transactions.
We do not have to guess what happens when that access outruns the controls around it. In July, Hugging Face disclosed that an autonomous AI agent had breached part of its production infrastructure, carrying out roughly 17,600 unauthorized actions before it was caught. OpenAI later confirmed the agent was one of its own internal research models, a system built and closely held by the company most invested in AI safety. It was simply given autonomy and access it was not adequately bound to hold.
That is the pattern that should concern us most, not a model becoming more capable, but a capable model being handed authority faster than the controls around it could keep pace.
Errors Of Ignorance
For most of the history of computing, building powerful software required years of technical education, and that requirement created an important natural barrier. AI is rapidly removing that barrier. Someone with very little engineering background can now describe what they want in natural language and have an AI system generate sophisticated applications, integrations and agents.
The problem is these builders do not know what they do not know. The hard part of building consequential systems was never making it work when everything goes right, it is knowing what happens when a process fails midway, a credential is compromised or an agent takes an action no one anticipated. AI-generated software expands the population capable of creating powerful systems without expanding the population capable of understanding how those systems fail.
The software being generated is increasingly capable of acting on its own. A poorly built website inconveniences its users. A poorly built autonomous agent connected through an API can move money, expose data or alter infrastructure without anyone realizing it was ever at risk.
We will soon have vastly more software, built much faster, by many more people, and that software will hold authority over consequential systems. The productivity gains will be enormous, but so will the blast radius of the mistakes we do not see coming because we did not know to look for them.
I propose a framework that accounts for all three failure modes. A much better approximation:
AI Risk Equals Intelligence Times Access Times Autonomy
Today’s models are already powerful enough to create substantial risk, which is why we need better control architectures around the models already deployed.
Agentic systems need permissions, identities, authorization boundaries, transaction limits, monitoring and independent verification, and, critically, they need architectures in which the AI cannot simply override the systems responsible for supervising it. This is not fundamentally different from what engineers learned in aviation, nuclear power and financial systems: powerful systems require powerful controls.
Where The Real Fight Is
While the frontier lab founders debate how fast to build the next generation of models, that debate does nothing to protect a company that has already given an agent unconstrained access to its financial systems, infrastructure or operations.
The most important AI safety battle of the next several years will not happen inside the AI labs, but inside software companies as engineers make thousands of mundane decisions about APIs, MCP servers, permissions, authentication, agent autonomy and system access. Those decisions will determine whether AI becomes an extraordinarily powerful tool operating inside well-engineered boundaries, or a vast collection of probabilistic systems with excessive authority over the digital and physical infrastructure surrounding us.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?