Nishanth Prakash, Principal Member of Technical Staff, AI/ML Engineer, Oracle.
AI is rapidly moving beyond software that generates text, images and code. A new generation of AI systems is beginning to interact with the physical world through robots capable of interpreting instructions, understanding their surroundings and translating high-level goals into actions.
For decades, robots have been extremely capable within carefully controlled environments. Industrial robots can repeatedly weld, assemble or move components with extraordinary precision. But these systems are typically programmed around predefined tasks, known objects and structured environments.
Large language models (LLMs) and, increasingly, multimodal models are beginning to change this paradigm. Instead of programming every possible action individually, developers can use language as a higher-level interface between humans and machines.
This is where an important architectural shift is beginning to emerge: Robots are moving from executing predefined instructions toward interpreting human intent.
Why Traditional Robotics Alone Is Not Enough
Traditional robotic systems are built around components such as perception, motion planning, localization and control. These technologies remain essential because robots ultimately need precise and predictable mechanisms for interacting with the physical world.
The challenge emerges at a higher level. Consider a robot asked to perform a task such as “Clear the table and put the dishes in the sink.” A traditional system may require the task to be decomposed beforehand into several explicitly programmed steps: identify dishes, determine grasp positions, navigate toward the sink, avoid obstacles and place each object in the appropriate location. A language-enabled system can potentially interpret the broader goal and help determine the sequence of actions necessary to complete it.
In this architecture, the language model does not replace the robotic controller. Instead, it becomes part of the reasoning layer surrounding it. The distinction is important. Motors, collision avoidance systems and real-time controllers still need deterministic behavior. Language models can instead help provide context, planning and interpretation.
From Language Models To Vision-Language-Action Models
Language alone, however, is not sufficient for robotics. A robot needs to understand not only what a person says but also what exists in its environment. This has contributed to the development of vision-language models that combine natural-language understanding with visual perception. The next evolution is increasingly referred to as vision-language-action (VLA) models.
These systems attempt to connect three capabilities:
1. Understanding human instructions
2. Interpreting visual information from the environment
3. Translating that understanding into robotic actions
Google DeepMind’s RT-2, for example, demonstrated how a vision-language model could be adapted to generate robotic actions, allowing knowledge learned from large-scale vision and language data to help robots respond to previously unseen scenarios.
Other robotics platforms are moving in a similar direction. Nvidia’s GR00T platform combines multimodal inputs, robotics foundation models, simulation and runtime infrastructure for general-purpose humanoid robotics.
The broader pattern is becoming clear: Generative AI is evolving from generating digital content toward helping machines interpret and act within physical environments.
How Physical Environments Increase The Complexity
Moving from generating words to generating actions introduces an entirely different class of engineering problems. A chatbot can generate an incorrect answer and allow the user to correct it. A robot operates in an environment where an incorrect action can damage equipment, drop an object or potentially create safety risks.
Robotic AI systems therefore need to reason within a much tighter operational loop. They must continuously process:
• Camera and sensor inputs
• Spatial information
• Object locations
• Human instructions
• Robot state
• Environmental changes
• Physical constraints
The system may also need to modify its plan while executing it. Imagine a warehouse robot instructed to retrieve a package from a shelf. While navigating toward the package, another worker may place an obstacle in its path. The robot must detect the change, update its understanding of the environment and determine another safe route.
The challenge is no longer simply generating an intelligent response. It is maintaining an accurate connection between reasoning and a constantly changing physical world.
Robots Need More Than Language Models
This is also why simply attaching an LLM to a robot is unlikely to create a general-purpose robotic system. Language models are optimized for reasoning over information represented through tokens. Robots operate in a world governed by geometry, friction, force, timing and physical uncertainty. Modern robotic architectures are therefore beginning to separate different forms of intelligence.
Figure’s Helix architecture, for example, combines a vision-language system for higher-level understanding with a faster visuomotor policy responsible for real-time robotic control. The higher-level component interprets context and goals, while the faster system converts those representations into continuous physical actions.
This type of separation makes sense because different parts of robotics operate at very different timescales. A language model might determine: “Pick up the blue container and place it on the second shelf.”
A robotic controller still needs to determine the exact trajectory of the arm, control the gripper, adjust to sensor feedback and avoid collisions potentially hundreds of times per second. The future of robotics may therefore involve multiple specialized models working together rather than one model controlling everything.
Why Grounding And Safety Matter
As AI becomes capable of initiating physical actions, grounding becomes increasingly important. The system must understand the difference between what it believes is happening and what its sensors actually observe. For example, if a model expects a container to be present but the robot’s camera no longer detects it, the robot should not continue executing the original action blindly. It should stop, reassess the environment and potentially request additional information.
This requires safeguards around AI decision-making. Production robotic systems will likely need mechanisms such as:
• Action validation
• Physical safety boundaries
• Confidence thresholds
• Permission controls
• Emergency stops
• Continuous sensor verification
• Human intervention for uncertain situations
These concepts resemble the guardrails emerging around autonomous software agents, but the consequences are different. When AI moves from calling APIs to controlling machines, reliability becomes a physical requirement rather than simply a software quality metric.
The opportunity is not simply to put an LLM inside a robot but to combine language reasoning with perception, real-time control and appropriate safety mechanisms. As these layers become more tightly integrated, robotics can increasingly move from executing predefined instructions toward understanding human intent and translating it into reliable physical action.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?