As AI workloads and accelerated applications grow in sophistication and complexity, businesses and developers need better tools to assess their infrastructure’s ability to handle the demands of both training and inference more efficiently. To that end, Nvidia has been working on a set of performance testing tools, called DGX Cloud Benchmark Recipes, that are designed to help organizations evaluate how their hardware and cloud infrastructure perform when running the most advanced AI models available today. Our team at HotTech had a chance to kick the tires on a few of these recipes recently, and found the data they can capture to be extremely insightful.
Nvidia’s toolkit also offers a database and calculator of performance results for GPU-compute workloads on various configurations, including the number of Nvidia H100 GPUs and cloud service providers, while the recipes allow businesses to run realistic performance evaluations on their own infrastructure. The results can help guide decisions on whether to invest in more powerful hardware, cloud provider service levels, or tweak configurations to better meet machine learning demands. These tools also take a holistic approach that incorporates network technologies for optimal throughput.
What Are DGX Cloud AI Benchmarking Recipes?
Nvidia DGX Cloud Benchmarking Recipes are a set of pre-configured containers and scripts that users can download and run on their own infrastructure. These containers are optimized for testing the performance of various AI models under different configurations, making them very valuable for companies looking to benchmark systems, whether on prem or in the cloud, before committing to larger-scale AI workloads or infrastructure deployments.
In addition to offering static performance data, time to train and efficiency calculated from its database, Nvidia has recipes readily available for download that let businesses run real-world tests on their own hardware or cloud infrastructure, helping them understand the performance impact of different configurations. The recipes include benchmarks for training models like Meta’s Llama 3.1 and Nvidia’s own Llama 3.1 branch, called Nemotron, across several cloud providers (AWS, Google Cloud, and Azure), with options for adjusting factors like model size, GPU usage, and precision. The database is broad enough to cover popular AI models, but it is primarily designed for testing large-scale pre-training tasks, rather than inference on smaller models. The benchmarking process also allows for flexibility. Users can tailor the tests to their specific infrastructure by adjusting parameters such as the number of GPUs and the size of the model being trained.
The default hardware configuration in Nvidia’s database of results uses the company’s high-end H100 80GB GPUs, but it is designed to be adaptable. Although currently, it does not include consumer or prosumer-grade GPUs (e.g., RTX A4000 or RTX 50) or the company’s latest Blackwell GPU family, these options could be added in the future.
Running the DGX Cloud Benchmarking Recipes is straightforward, assuming a few prerequisites are met. The process is well-documented, with clear instructions on setting up, running the benchmarks, and interpreting the results. Once a benchmark is completed, users can review the performance data, which includes key metrics like training time, GPU usage, and throughput. This allows businesses to make data-driven decisions about which configurations deliver the best performance and efficiency for their AI workloads. This could also go a long way in helping companies maintain green initiatives in terms of meeting power consumption and efficiency budgets.
DGX Cloud Benchmarking Recipes Market Impact And Potential For AI Efficiency
While the DGX Cloud Benchmarking Recipes offer valuable insights, there are a few areas where Nvidia’s tools could be expanded. First, benchmarking recipes are currently focused primarily on pre-training large models, not on real-time inference performance. Inference tasks, such as token generation or running smaller AI models, are equally important in many business applications. Expanding the toolset to include more detailed inference benchmarks would provide a fuller picture of how different hardware configurations handle these real-time demands. Additionally, by expanding the recipe selection to include lower-end or even higher-end GPUs (like Blackwell or even competitive offerings), Nvidia could cater to a broader audience, particularly businesses that don’t require the massive compute power of a Hopper H100 80GB cluster for every workload.
Regardless, Nvidia’s new DGX Cloud Benchmarking Recipes look like a very helpful resource for evaluating the performance of AI compute infrastructure, before making major investment decisions. They offer a practical way to understand how different configurations—whether cloud-based or on-premises—handle complex AI workloads. This is especially valuable for organizations exploring which cloud provider best meets their needs, or if the company is looking for new ways to optimize existing infrastructure.
As AI’s role in business and our everyday lives continues to grow, tools like this will become essential for guiding infrastructure decisions, balancing performance versus cost and power consumption, and optimizing AI applications to meet real-world demands. As Nvidia expands these recipes to include more inference-focused benchmarks and potentially expands its reference data with a wider range of GPU options, these tools could become even more indispensable to businesses and developers of all sizes.
