Yuri Gubin is the Chief Innovation Officer at DataArt.
After about two years of the generative AI (GenAI) wave, the big question is: What is the ROI, and how do we measure it? It’s not just about AI’s capabilities but also the value and risks.
AI has the potential to deliver transformative results, but it comes with challenges. Models, training, inference and data management are expensive. Without a clear plan, costs can spiral quickly. To get the most out of AI while staying within budget, companies must be strategic, mindful of resources and focused on delivering measurable outcomes.
It all starts with two approaches to balance innovation with cost control:
1. Build, then optimize. Prove the hypothesis first. Validate that your customers, end users, employees or systems find value in the new AI capability. Does it improve quality, efficiency or performance? If so, scale the initiative and optimize it. Avoid overengineering something that doesn’t exist yet—focus on customer feedback, satisfaction and measurable value before optimizing.
2. Embrace a “fail fast” mindset. Prototyping is essential, but not every idea will work, and that’s okay. Test small, fail quickly and move on. By keeping projects lightweight, you can focus on what shows potential instead of wasting time on what doesn’t. This lets you innovate continuously without betting everything on one approach.
The Benchmarking Challenge
Most companies aren’t building foundational models but instead create applications using third-party APIs or hosted models. The challenge lies in measuring effectiveness. Whether you’re working on chatbots, knowledge bases or data extraction, it’s critical to benchmark engines and models, testing them for precision, accuracy and context-specific performance.
In the early days of AI, evaluation was manual. Engineers tested, iterated and validated models. Today, frameworks like Ragas provide metrics to benchmark AI solutions, helping measure accuracy and precision in specific contexts. Tools like these have evolved from basic manual tests to more sophisticated automated comparisons.
Understanding Costs: AI Vs. Supporting Infrastructure And Data
For companies relying on third-party models, cost control in inference is vital.
Techniques like token compression reduce token usage while maintaining quality. Retrieval-augmented generation (RAG) and emerging methods like GraphRAG also play a role, but infrastructure costs (e.g., storage, databases and network traffic) can often outweigh model-related expenses.
Poorly designed infrastructure can lead to excessively high costs, with single queries sometimes reaching $700 to $800, having dozens of vector DBs each only 10% utilized or data warehouses that are cost-prohibitive.
In other words, it’s crucial to evaluate total ownership costs, including non-AI-specific expenses.
Starting To Measure ROI
Without understanding the costs, it is impossible to measure any ROI.
The best practices for understanding the cost of AI include FinOps, cost attribution and tagging to help identify bottlenecks and control costs. Organizations can also pinpoint inefficiencies and better understand ROI by measuring expenses at the application, service or prompt level. This clarity enables decisions based on financial returns and capability value.
Remember that AI development, like other development, is an iterative process. The moment you release an AI application, you’ll likely already see the scope for the next version. User feedback will inform retraining, prompt adjustments and fine-tuning. New technologies, like GraphRAG or actionable AI, continue to emerge, underscoring the need for flexibility and scalability in AI frameworks.
However, it is very possible that end users will not notice a difference if your application swaps one model with another. It is important to tie AI interactions with outcomes like new transactions, purchases, requests served, etc., to gauge ROI.
The Bottom Line: Innovate With Purpose
AI has the power to reshape industries, but it’s not a free pass to experiment without limits. The most successful companies strike a balance between bold ideas and practical resource management. They focus on what matters, stay mindful of costs and use feedback to refine their approach.
By prioritizing value and efficiency, businesses can harness the full potential of AI without losing sight of their bottom line.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
