Monica Bajaj is the Senior Vice President of Engineering at Okta.
Engineering leaders are facing an uncomfortable AI reality. Token consumption is soaring, while true business impact often remains unclear. Companies want their teams to embrace AI, but there’s growing pressure to demonstrate a return on that investment. Engineering leaders want to see greater velocity. Security leaders want to ensure that all systems are safe. Finance leaders want to know how dollars are spent and if they are getting the proper return. It’s a three-edged sword, and that tension is building every day.
Part of the problem is that our systems, architectures and workflows were designed for humans. AI agents are the new kid on the block. Humans find workarounds when processes aren’t working. Agents can continue looping, consuming resources without producing meaningful results. We need to rethink how we measure AI’s value, distinguishing between consumption and outcomes.
Token Consumption Is Not Business Value
In my opinion, token consumption measures system activity, not business value. It tells you what you’re spending, not what you’re getting in return. A sharp increase in token volume can reflect inefficient system design, bloated prompts, repetitive trial and error or agents caught in unnecessary loops. Someone might repeatedly prompt an AI tool because the output isn’t useful, consuming more tokens without improving productivity.
That’s why I believe we need to shift our mindset from cost per token to cost per successful outcome. Instead of measuring consumption alone, evaluate the cost per resolved task, quality and defect rates, task completion velocity and caching efficiency. Evaluate whether a use case needs AI or would be better served by traditional automation.
We should always be asking whether we’re getting meaningful outcomes at a reasonable cost.
Introducing The SQUAD Framework
To evaluate AI investments, I’ve developed a framework called SQUAD. It looks at five dimensions engineering leaders should consider together.
Security: Is AI introducing systemic risks or vulnerabilities? We need to measure how many vulnerabilities are detected before code is merged and what proportion of AI-generated code requires security remediation.
Quality And Reliability: Is the codebase getting healthier, or is it degrading because of AI-generated work? Faster code generation doesn’t mean much if we’re introducing more defects. Change failure rates, bug density, code churn and test coverage help us understand the actual impact.
Unit Economics: Does the output we’re getting justify the per-seat and compute costs? Instead of focusing exclusively on token consumption, measure cost per merged pull request, cost per successful task completion and whether we’re using the right models for the right tasks.
Acceleration And Velocity: Are our teams shipping business outcomes faster? Look at pull request review cycles, lead time for changes and whether we’re actually delivering more value rather than simply generating more code and PRs.
Developer Experience: Are these tools helping engineers be more productive or creating additional pain? Consider developer sentiment, review fatigue and time spent correcting AI-generated work.
Evaluate these dimensions together. The substantial improvement in pull request throughput isn’t considered a win if the number of security issues or quality failures has risen. It’s also crucial to look beyond code acceptance rates. What is the gain if the engineer accepts AI-generated code and then spends an hour refactoring it?
SQUAD helps us understand whether AI improves engineering rather than simply accelerating individual activities.
Create Guardrails Without Discouraging Innovation
Experimentation is extremely important. With AI, it’s faster and easier to experiment, fail fast and learn. But we have to balance that freedom with responsible spending.
Build innovation sandboxes separate from production pipelines, giving developers reasonable testing quotas. Give developers reasonable caps or quotas for freeform testing. When someone reaches that limit, don’t automatically shut down the project. Instead, evaluate the prototype’s validity and whether it can graduate to a production-level budget.
Another important practice is what I call model-conscious engineering. People often use the most powerful model for the simplest tasks. But you don’t need a frontier model to generate boilerplate code or format JSON. Use dynamic model routing to match the task’s complexity with the appropriate model, balancing quality, cost and latency.
Architectural hygiene is equally imperative. Caching, better prompt design and limiting the tools available to agents can reduce unnecessary consumption.
We’ve asked teams to adopt AI; now we need to educate them on using it effectively and efficiently.
Regulate The Data Instead Of The Tool
My philosophy is to regulate the data, not the tool. Models and tools are continuously evolving, but organizations can establish consistent policies around their information. I use three categories: green for public information, yellow for internal proprietary information that requires approved enterprise environments and red for restricted or confidential information that shouldn’t be exposed to unauthorized AI systems.
Human accountability is equally vital. When you commit code, you own it. The responsibility doesn’t belong to the agent. Developers must be able to justify their work, including its security, performance and edge cases.
We also need to be enablers, not blockers. If you tell a team of problem solvers they can’t use the sharpest tool on the market, they’ll find a workaround. Providing secure, enterprise-approved access is far more effective than creating unnecessary barriers.
Measure Outcomes Across The Entire Organization
As AI adoption expands into product and design, we need to rethink how value is measured across teams. For engineering, we’re looking at velocity, quality, security and how quickly we’re shipping features.
For product teams, we should measure discovery cycle time, not how many product requirement documents people are generating. AI should help teams validate problems, gather customer insights and reach cross-functional alignment faster.
For design teams, the focus should be on broader exploration, testing different workflows and reducing the back-and-forth before engineering begins.
Engineering, product and design need to work together rather than operating in silos. Otherwise, we risk generating more AI slop that creates additional work downstream (when that happens, the net ROI can actually be negative).
Ultimately, we want to deliver better outcomes at a reasonable cost. With the right framework, guardrails and mindset, we can encourage innovation while guaranteeing our AI investments create meaningful business value.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?

