Advancing Robot Learning, Perception, and Manipulation with Latest NVIDIA Isaac Release

At CES 2025, NVIDIA announced key updates to NVIDIA Isaac, a platform of accelerated libraries, application frameworks, and AI models that accelerate the development of AI robots.

NVIDIA Isaac streamlines the development of robotic systems from simulation to real-world deployment. In this post, we discuss all the new advances in NVIDIA Isaac:

Isaac Sim
Isaac Lab
Isaac Manipulator
Isaac Perceptor
What’s new in Isaac Sim 4.5
NVIDIA Isaac Sim is a reference application built on NVIDIA Omniverse that enables you to develop, simulate, and test AI-driven robots in physically based virtual environments.

Coming at the end of January, the new Isaac Sim 4.5 will offer a number of significant changes, including the following:

A reference application template
Improved URDF import and setup
Improved physics simulation and modeling
New joint visualization tool
Simulation accuracy and statistics
NVIDIA Cosmos world foundation model
Reference application template
Isaac Sim has been redesigned as a customizable reference application. A minimal template for faster startup and a full template with complete functionality along with all dependencies. This enables you to tailor the application to your specific needs, whether for headless applications or full Isaac Sim experiences.

Improved URDF import and setup
Significant improvements have been made to the URDF importer. The User Interface has been simplified to provide a more streamlined process and align with how other formats are imported. You can now individually configure joint drives, making the robot ready to use immediately after import. To assist with joint drive configuration, a natural frequency-based tuning option is provided.

Improved physics simulation and modeling
Isaac Sim 4.5 features significant advancements in physics modeling and simulation. You can define and configure various joint types between robot components, setting parameters like stiffness and damping to fine-tune joint behavior.

New joint visualization tool
A new joint visualization tool enables you to inspect the physics properties of selected prims, including their position, rotation, linear and angular velocities, and accelerations. You can now review and optimize various scene parameters such as deformable surfaces or memory usage, before running simulations.

Figure 1. Various attributes of selected objects and their association with the other objects

Simulation accuracy and statistics

Simulation accuracy is greatly improved with a new implementation of full momentum conservation for rigid bodies and articulations. 

You can now also visualize the simulation statistics of objects and scenes that can either interact with each other or are completely independent. You can review different parameters from deformable surfaces to overall memory used in the buffer. This provides a way to troubleshoot and optimize your scene prior to running the simulation. 

NVIDIA Cosmos world foundation model
Also announced at CES, the NVIDIA Cosmos world foundation model platform can be used to generate massive amounts of controllable synthetic data to train perception robots when paired with Isaac Sim.

In Isaac Sim, you compose SimReady 3D scenes by unifying diverse data inputs including CAD, lidar-to-point cloud scans, and generated 3D objects from AI models such as Edify 3D. Then, you compose and stage the scenario to reflect the specific task the robot must perform, and render images or videos.

Cosmos can ingest images and videos and output photoreal video clips to then retrain policy models.

What’s new in Isaac Lab 2.0
NVIDIA Isaac Lab is an open-source unified framework for robot learning to train robot policies. Isaac Lab is built on top of NVIDIA Isaac Sim, helping developers and researchers more efficiently build intelligent, adaptable robots with robust, perception-enabled, simulation-trained policies.

A new 2.0 version of Isaac Lab will be made available at the end of the month, including performance and usability improvements including:

Tiled rendering: Up to a 1.2x boost in tiled rendering speed, which combines outputs from simultaneous simulations into a single, large image rather than processing numerous smaller images from individual cameras.
Quality of life improvements: Simplified installation process using Python package managers. Isaac Lab will also be available as a container, enabling the movement of workloads across systems without underlying dependencies.
Previewed at CES, humanoid robot developers can also take advantage of the NVIDIA Isaac GR00T Blueprint for building custom data pipelines for generating vast amounts of synthetic trajectory data from just a small number of human demonstrations. The GR00T blueprint is currently in invite-only early access. Join the NVIDIA Humanoid Developer Program when it becomes widely available in beta.

What’s new in Isaac Manipulator
NVIDIA Isaac Manipulator, built on ROS 2, is a collection of NVIDIA CUDA-accelerated libraries, AI models, and reference workflows. It enables you to build AI-enabled robot arms, or manipulators, that can perceive, understand, and interact with their environments.

Isaac Manipulator now includes new end-to-end reference workflows for pick-and-place and object-following, enabling you to quickly get started on fundamental industrial robot arm tasks:

Object-following: Shows the robot gripper’s ability to maintain a consistent position relative to a moving object, while maneuvering around obstacles.
Pick-and-place: Shows how a robot can pick up an object and release it in a predetermined region while avoiding obstacles (Figure 3).

These reference workflows are now supported in Isaac Sim, enabling rapid testing without the need for physical hardware setup.

Developer support and other enhancements include the following:

Performance improvements to FoundationPose
Updates to nvblox for manipulator use cases
A tutorial for robot hand-eye calibration
An Isaac Sim-based tool for setting and simulating custom grasps for a gripper and object pair
What’s new in Isaac Perceptor
NVIDIA Isaac Perceptor, built on ROS 2, is a collection of NVIDIA CUDA-accelerated libraries, AI models, and reference workflows for the development of autonomous mobile robots (AMRs). It enables AMRs to perceive, localize, and operate in unstructured environments such as warehouses or factories.

Isaac Perceptor encompasses CUDA-accelerated libraries such as nvblox for 3D scene reconstruction and cuVSLAM for stereo-visual-inertial SLAM (simultaneous localization and mapping), which you can integrate into existing AMR workflows.

Isaac Perceptor’s latest updates bring significant improvements to AMR’s environmental awareness and operational efficiency in dynamic settings such as warehouses. Key new features and improvements include:

New end-to-end visual SLAM reference workflow
New examples on running nvblox with multiple cameras for 3D scene reconstruction with people detection and dynamic scene elements
Improved 3D scene reconstruction by running Isaac Perceptor on multiple RGB-D camera
These updates significantly improve 3D scene reconstruction, leading to higher accuracy and robustness in 3D scene capture and mapping performance in real-world scenarios and complex, dynamic environments.

NVIDIA ecosystem partners such as Orbbec, LIPS (Realsense), StereoLabs (Zed) offer compatible cameras. Developer support also includes the Mapping and Localization with Isaac Perceptor tutorial for offline mapping capabilities with the cuVGL and cuVSLAM libraries using Nova sensors.

Ecosystem adoption
Multiple industry partners have announced integrations of NVIDIA Isaac into their platforms and solutions:

Boston Dynamics is using Isaac Lab and NVIDIA Jetson AGX Orin to enable simulated policies to be directly deployed for inference, simplifying the deployment process.

To train their GR-1 and GR-2 humanoid robots, the Fourier team turned to NVIDIA Isaac Gym (now deprecated) for reinforcement learning. They are currently porting their workflows to NVIDIA Isaac Lab.
Foxglove has developed an extension in Isaac Sim that enables the real-time visualization of robotics simulation data directly in Foxglove.

Main Street Autonomy’s Calibration Anywhere software automates sensor calibration improving sensor fusion for robotics using Isaac Perceptor.

Miso Robotics automates kitchen tasks using advanced robotics, Isaac Manipulator, and NVIDIA Isaac ROS, enhancing efficiency, consistency, and customer satisfaction in commercial kitchens.
RGo Robotics and NVIDIA are transforming mobile robotics with advanced AI and perception technologies using Isaac Perceptor and Isaac ROS.

Scaled Foundations, an NVIDIA inception member, has developed General Robot Intelligence Development (GRID), an advanced cloud-based platform that accelerates robot AI solution development. GRID seamlessly integrates Isaac Sim and Isaac Lab technologies, offering an end-to-end platform for robotics developers and researchers to train, simulate, and deploy their robotics applications. For more information, see NVIDIA Isaac Sim on GRID.

Virtual Incision is using NVIDIA platforms involving Holoscan, IGX, Sensor Bridge, Isaac Sim. They are exploring Cosmos to train, simulate, and test, and also exploring inference AI features for the next generation of assisted robotic surgery devices.

NVIDIA Inception member and deep tech startup Wandelbots is building custom robot simulations with their operating system Wandelbots NOVA, which is seamlessly integrated with Isaac Sim.