Deepak Khosla is Chief Growth Officer & Head of AI Business at Impetus Technologies, Inc.
For many CTOs, the first question around agentic AI has been which model to use or how much to invest in fine-tuning, prompt engineering and infrastructure. Those questions matter, but they are not the full strategy. In my view, models are not the moat. One model may lead today, and another may catch up a few months later.
The real reason we still see so many agentic AI solutions sitting in sandbox environments, or moving into production only as chatbot-oriented use cases, is that more complicated AI solutions are missing something very key—context.
Context is what makes agentic solutions perform better, think better, take actions and repeat actions—and do so in a uniform way. Without it, even the best model will struggle to do real enterprise work.
This is also a cost issue. Without the right context, enterprises may spend far more on memory, inference, tokens and storage to get the same result. A strong context and semantic layer help agents perform better while making cost and FinOps easier to manage.
Agentic AI Is Not A Smarter Chatbot
One misunderstanding I see is that many leaders still think of agentic AI as a smarter chatbot because many use cases so far have been chatbot-oriented. But the right agents, if built correctly, will take action, make a sequence of decisions, operate with some level of agency and work continuously toward an outcome.
Once you move into that kind of environment, the problem is not only the AI solution itself or the power of the model. The problem is context.
The second misunderstanding is that many CTOs believe the constraint is the model. They spend a lot of energy, effort and money on infrastructure, fine-tuning and prompt engineering. Some of that is required. But if enterprise data is still siloed, if the organization is not semantically aware and if the right context is not available when agents need to act, having the best model will not solve the problem.
Morgan Stanley’s work with OpenAI in wealth management is a useful example. The assistant was designed to help financial advisors access Morgan Stanley’s vast intellectual capital, including hundreds of thousands of pages of research and commentary. In other words, the model became more useful because it was grounded in enterprise context.
The third misunderstanding involves building capabilities before controls. CTOs are under pressure to deploy agents, but if agents perform human tasks at scale without the right controls, the capability quickly becomes enterprise risk.
The Four Parts Of The Context Gap
In practical technical terms, I think the context gap consists of four things: the data gap, the semantic gap, the execution gap and the trust gap.
The data gap is information that exists but that agents cannot reach. It may be in legacy systems, ERP platforms, PDFs, spreadsheets, unstructured repositories or systems that were never connected. The enterprise may have the information; the agent simply cannot access it. A loan origination system and a risk model at the same bank may define ‘customer’ differently, and the agent reasoning across both will contradict itself without knowing it.
The semantic gap comes next. Even if all that data sits in a lake or warehouse, the data itself is meaningless unless the agent can interpret it. Without a semantic layer, agents may take 10x or 20x the tokens and time that a proper semantic graph or knowledge graph could provide. An agent that sees a product return as a transaction record cannot read it as a sizing defect, a quality signal or an early markdown trigger.
The execution gap is whether the agent can make the right recommendation with the right constraints, such as supplier agreements, promotional calendars, inventory details, approval thresholds and operating rules. Even with a strong semantic layer, those constraints must be engineered into the solution. An agent that correctly identifies a clearance opportunity but executes without checking the supplier’s minimum advertised price agreement makes a mistake, creating a compliance finding.
The fourth is the trust gap. If an agent makes a decision that nobody can explain, the recommendation may be ignored or nullified by a human in the loop. To scale adoption, organizations need a trust layer that explains why an action was recommended or taken. Trust isn’t a soft requirement. It’s what separates pilots from production.
CTOs Need Context, Execution And Control Layers
For agentic AI to scale, three layers must be in place, and they must be built in the right order.
The first is the context layer. A context fabric that is semantically enriched, connected, governed and representative of enterprise knowledge. This may include knowledge graphs, domain ontologies, vector stores, memory systems or other structures agents can reason over.
The second is the execution layer. This includes the agent runtime, orchestration tools, multi-agent coordination and agent-to-agent trust that allow agents to execute work across workflows and systems.
The third is the control layer: guardrails, human-in-the-loop models, anomaly detection, compliance enforcement and rules. An agent needs to know not only what it needs to do but also what it is not permitted to do.
The Control Plane Makes AI Operational
An enterprise AI control plane is all about governance and observability across the agents that an organization builds.
It needs to manage identity and permissions, such as what data an agent can access, what tools it can use, what actions it can take and when it must call in a human. It also needs to manage context health, including whether the context is accurate, current and complete at the moment of inference and whether the system can detect context drift.
Auditability is essential. Every decision and action should have a decision trail that traces the context, steps, reasoning process and action taken. Cost governance also matters because agents can become expensive if they are not properly controlled. Inference, memory, storage, model and token costs must be governed against the business objective.
Finally, organizations need a human-in-the-loop model, not just a human-in-the-loop solution. They must define where humans intervene, what actions require approval and where deterministic controls should remain in place. The enterprises that will scale agentic AI in the next three years will be differentiated by how well they govern it because governance is what earns the trust of the regulators, risk teams and customers that determine whether AI gets to act at scale.
Legacy Systems Are Institutional Memory
Legacy systems are not technical debt; they are institutional memory. They are repositories of enterprise context, housing decades of business logic, definitions, exceptions and institutional memory. This memory is exactly what an agent needs to act correctly in that enterprise’s specific context.
The challenge is that this context is not AI-accessible. CTOs should work toward AI-ready data foundations while recognizing that well-designed agents can help navigate complexity across systems. The answer is balance. Modernize where needed, unlock context where possible and make legacy knowledge usable without unnecessary disruption.
The goal is to make business and technology processes more AI-native, whether in underwriting, loan origination or supply chain visibility. Agentic AI becomes powerful when the right building blocks, semantic layer and controls are in place.
It will not scale through experimentation alone. It will scale when enterprises build the context, execution and control layers that allow agents to do real work safely, securely and repeatedly.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
