Gregory Shahnovsky is CEO of Modcon Systems, specializing in process analytics and AI-driven optimization for complex industrial operations.
Artificial intelligence is moving rapidly from the information domain into the operational world. Until now, enterprise AI systems analyzed data, generated content or supported human decisions. That is changing.
AI agents can now access software systems, execute workflows, interact with connected devices and perform actions with consequences outside the digital environment. It is reasonable to expect that AI will soon participate directly in operational decision-making.
This changes the nature of the risk. Once AI can influence physical equipment, industrial processes or critical infrastructure, conventional IT security methods are no longer sufficient on their own. Protecting identities, networks, software and data remains essential, but what happens if AI makes the wrong decision, is manipulated, exceeds its authority or creates a hazardous physical intervention?
That is fundamentally an operational technology and functional-safety problem. High-consequence AI should therefore draw more heavily from established industrial safety engineering.
Industry Has Already Faced This Problem
Industrial automation has never assumed that instruments, controllers or operators will always behave correctly. Process plants and other production facilities use independent layers of protection because failures are expected to occur.
IEC 61508 formalized this approach through Safety Integrity Levels, from SIL 1 to SIL 4. The higher the required SIL, the greater the required risk reduction and the more demanding the design, verification and independence of the safety function.
The lesson for AI is not to copy SIL terminology directly, but to apply the same principle: The greater the possible consequence of failure, the stronger and more independent the protection must be.
Not Every AI Application Carries The Same Risk
An AI tool that summarizes documents or creates advertising material should not be considered in the same way as an AI system capable of changing factory operating conditions, controlling vehicles, altering air traffic operations or approving financial transactions.
Enterprises should classify AI applications according to potential consequence.
Low-risk applications may require conventional cybersecurity, audit trails and human review. Higher-risk applications should require stricter access controls, defined operating boundaries, mandatory approval and comprehensive logging.
Where AI can influence safety-related processes or critical infrastructure, independent validation, deterministic operating limits and automatic isolation should become part of the architecture. For the highest-risk applications, AI should not be allowed to rely on its own judgment to determine whether its behavior remains safe.
AI Needs An Independent Safety Shutdown
In process industries, controllers regulate flow or temperature, while an independent safety instrumented system prevents hazardous conditions. If a critical variable exceeds an allowable limit, the safety function overrides the control system regardless of why the controller made its decision.
AI architecture should follow the same principle. If an AI system attempts an action outside its authority, changes a process variable beyond an approved limit, accesses a prohibited system or shows signs of compromise, a separate layer should be capable of stopping it.
This may involve rejecting a command, removing write access, isolating the AI, transferring control back to the operator or moving equipment into a safe state.
This protection should not depend on asking the same AI system whether its own action is safe. For high-consequence applications, the protection layer should remain effective even if the AI itself has been compromised.
Security Must Also Exist Inside The AI
External protection is essential, but it should not be the only safeguard. AI systems used in consequential applications should operate with a clear hierarchy of priorities.
In industrial optimization, AI may be tasked with reducing energy use, increasing production or improving quality. Those objectives must remain subordinate to safety and security constraints.
The hierarchy should be explicit: safety first, then security and mandatory operating limits, followed by equipment integrity, quality, production and economic optimization.
A technically attractive or economically beneficial action may still be operationally unacceptable. Some limits are not variables to be traded against performance. They are boundaries.
Cybersecurity And Functional Safety Are Converging
Cybersecurity and functional safety have traditionally been managed as separate disciplines, although today they are increasingly considered part of the same operational risk environment. Cybersecurity protects systems against unauthorized access and manipulation, while functional safety protects people, equipment and the environment when systems fail or behave incorrectly.
AI increasingly connects the two. If a cyberattack compromises an AI system with authority over operational equipment, a cybersecurity event can become a process-safety event.
Enterprises therefore need to ask not only whether an AI system can be attacked, but what happens if the attack succeeds. What can it control? Which limits can it change? Which systems can it access? What prevents an incorrect action from propagating into the physical process?
Industrial defense-in-depth principles are directly relevant. Permissions can limit what AI can do. Internal rules can prevent prohibited decisions. Independent monitoring can identify abnormal behavior. Deterministic protection can reject unacceptable actions. Existing safety systems can remain responsible for final process protection.
AI Needs An Engineering Safety Case
Before a high-consequence AI application is deployed, organizations should be able to answer practical questions:
• What is the worst consequence of an incorrect AI decision?
• Which actions can it take without human approval?
• Which constraints can never be overridden?
• What happens if it is compromised?
• Which independent system can detect unacceptable behavior and safely shut the system down?
This type of structured risk assessment is already extensively used in industry through HAZOP reviews, where potential deviations, causes, consequences and safeguards are systematically examined before a process is put into operation. A similar discipline should be applied to high-consequence AI systems.
These are normal questions in industrial safety engineering. They should become normal questions in enterprise AI governance.
The Bottom Line
As AI moves from providing information to making recommendations, issuing commands and eventually making operational decisions, its security architecture will need the same discipline.
For high-consequence applications, safety and security should be separately engineered, independently protected and capable of bringing the system to a safe state when behavior moves outside acceptable limits.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?