NVIDIA RTX Neural Rendering Introduces Next Era of AI-Powered Graphics Innovation

NVIDIA today unveiled next-generation hardware for gamers, creators, and developers—the GeForce RTX 50 Series desktop and laptop GPUs. Alongside these GPUs, NVIDIA introduced NVIDIA RTX Kit, a suite of neural rendering technologies to ray trace games with AI, render scenes with immense geometry, and create game characters with lifelike visuals.

RTX Kit enhances geometry, textures, materials, and lighting, providing greater visual fidelity and performance for complex open worlds while reducing artifacts, instability, and VRAM usage. Developers can integrate individual—or all—RTX Kit SDKs directly into their rendering pipeline. 

RTX Neural Shaders bring AI to programmable shaders

25 years ago NVIDIA introduced GeForce and programmable shaders, unlocking two decades of innovation in graphics technology, from pixel shading to compute shading to real-time ray tracing.  Alongside the new GeForce RTX 50 Series GPUs powered by NVIDIA Blackwell architecture, NVIDIA is introducing RTX Neural Shaders, which bring small neural networks into programmable shaders, enabling a new era of graphics innovation.

The applications of neural shading are vast, including radiance caching, texture compression, materials, radiance fields, and more.  

The RTX Neural Shaders SDK enables developers to train their game data and shader code on an RTX AI PC and accelerate their neural representations and model weights with NVIDIA Tensor Cores at runtime. During training, neural game data is compared to the output of the traditional data and is refined over multiple cycles. Developers can simplify the training process with Slang, a shading language that splits large, complex functions into smaller pieces that are easier to handle.

This breakthrough technology is used for three applications: RTX Neural Texture Compression, RTX Neural Materials, and Neural Radiance Cache (NRC).

• RTX Neural Texture Compression uses AI to compress thousands of textures in less than a minute. Their neural representations are stored or accessed in real time or loaded directly into memory without further modification. The neurally compressed textures save up to 7x more VRAM or system memory than traditional block compressed textures at the same visual quality.
• RTX Neural Materials uses AI to compress complex shader code typically reserved for offline materials and built with multiple layers such as porcelain and silk. The material processing is up to 5x faster, making it possible to render film-quality assets at game-ready frame rates.
• RTX Neural Radiance Cache uses AI to learn multi-bounce indirect lighting to infer an infinite amount of bounces after the initial one to two bounces from path traced rays. This offers better path traced indirect lighting and performance versus path traced lighting without a radiance cache. NRC is now available through the RTX Global Illumination SDK, and will be available soon through RTX Remix and Portal with RTX.

Developers can get started with the RTX Neural Shaders and RTX Neural Texture Compression SDKs at the end of the month through NVIDIA RTX Kit. Sign up to be notified of availability. For DirectX developers, Microsoft will be adding Cooperative Vector support to DirectX in the near future. This will enable neural shaders to take advantage of accelerated performance with RTX Tensor Cores.

“Microsoft is excited to partner with NVIDIA to catalyze this next generation of graphics programming by bringing industry-wide support for neural shading technology,” said Bryan Langley, partner group program manager of Silicon, Media, and Graphics at Microsoft. “

DirectX will soon support Cooperative Vectors, which will unlock the power of Tensor Cores on NVIDIA GeForce RTX hardware and enable game developers to fully accelerate neural shaders on Windows.” 

Crossing the uncanny valley of rendering digital humans

Rendering human faces is one of the most challenging tasks in real-time graphics, as people are naturally conditioned to notice even the smallest deviations from reality. RTX Neural Faces offers an innovative, new approach to improve face quality using generative AI.

Instead of brute force rendering, Neural Faces takes a simple rasterized face plus 3D pose data as input and uses a real-time generative AI model to infer a more natural face. The generated face is trained from thousands of offline generated images of that face at every angle, under different lighting, emotion, and occlusion conditions.

The training pipeline can use real photographs or AI generated images, with variants created using a diffusion model.  The trained model is then NVIDIA TensorRT optimized to infer the face in real time. RTX Neural Faces represent a first step in a journey to redefine real-time graphics with generative AI.

RTX Neural Faces is complemented by the RTX Character Rendering SDK for hair and skin. Getting hair and skin on game characters to look and simulate accurately is challenging. Cutting edge strand-based hair techniques require up to 30 triangles per strand of hair and 4 million triangles for an entire groom, which makes ray tracing BVH builds expensive.

Linear-Swept Spheres (LSS) is a new GeForce RTX 50 Series GPU-accelerated primitive that reduces the amount of geometry necessary to render strands of hair and uses spheres instead of triangles to get a more accurate fit for hair shapes. LSS makes it possible to do ray-traced hair with better performance and a smaller memory footprint. 

Traditional rendering methods don’t accurately simulate how light interacts with human skin, which can result in a plastic-like look. Subsurface Scattering (SSS) simulates how light penetrates beneath the surface of translucent materials and scatters internally, creating a softer, more natural appearance.

Developers can get started with both RTX Skin and Hair algorithms through RTX Character Rendering SDK, coming at the end of the month. RTX Skin will also be available in RTX Remix. Sign up to be notified of availability. 

Create infinitely complex open worlds

The number of triangles used to create games has exponentially increased over the past 30 years. With the introduction of the Unreal Engine 5 Nanite geometry system, developers can build open worlds filled with hundreds of millions of triangles.

However, as ray traced game scenes explode in geometric complexity, the cost to build the bounding volume hierarchy (BVH) each frame for various levels of detail (LOD) grows exponentially, making it impossible to achieve real-time frame rates. RTX Mega Geometry accelerates BVH building, making it possible to ray trace up to 100x more triangles than today’s standard.

RTX Mega Geometry intelligently updates clusters of triangles in batches on the GPU, reducing CPU overhead and increasing performance and image quality in ray traced scenes.

RTX Mega Geometry is coming soon to the NVIDIA RTX Branch of Unreal Engine (NvRTX), so developers can use Nanite and fully ray trace every triangle in their projects. For developers using custom engines, RTX Mega Geometry will be available at the end of the month as an SDK to RTX Kit. Sign up to be notified of availability.

Maximize performance and responsiveness of next-generation worlds

NVIDIA DLSS is a suite of neural rendering technologies that uses AI to boost FPS, reduce latency, and improve image quality. The latest breakthrough, DLSS 4, brings new Multi Frame Generation and Transformer-powered Super Resolution and Ray Reconstruction, accelerated by GeForce RTX 50 Series GPUs and fifth-generation Tensor Cores.

DLSS Multi Frame Generation boosts FPS by generating up to three extra frames for each frame that is rendered. These frames are properly paced using improved hardware-based flip metering on GeForce RTX 50 Series.

DLSS 4 also introduces a new transformer-based architecture for DLSS Super Resolution, DLSS Ray Reconstruction and Deep Learning Anti-Aliasing, which uses self-attention and can much more easily identify the longer-range patterns across a much larger pixel window for improved image quality vs traditional convolutional neural networks (CNNs).

DLSS 4 SDK will be available at the end of the month to all developers through NVIDIA Streamline. Streamline simplifies integration of the latest NVIDIA technologies into applications and games. The new transformer models are backwards compatible with existing Super Resolution and Ray Reconstruction integrations. For Unreal Engine 5 developers, DLSS 4 is coming at the end of the month to NvRTX. Sign up to receive DLSS updates. 

NVIDIA Reflex is a latency reduction technology that synchronizes the CPU and GPU for ultimate responsiveness, providing faster target acquisition, quicker reaction times, and improved aim precision in latency-sensitive games. New Reflex 2 introduces Frame Warp, which further reduces latency by updating the rendered game frame based on the latest mouse input right before it is sent to the display.

A predictive inpainting technique uses color and depth data from prior frames to fill holes left by the warp. As a result, Reflex Frame Warp reduces latency up to 75% compared to native rendering. Reflex 2 SDK is coming soon to GeForce RTX 50 Series GPUs, with support added for other GeForce RTX GPUs in a future update. Sign up to receive Reflex updates. 

Redefining game AI with ACE autonomous game characters

The term AI has been used in games for decades. These non-playable characters (NPCs) traditionally follow strict rules designed to mimic intelligence, adhere to a guided story, and provide a scripted interaction with the player. However, with the rise of intelligent language models, game AI is primed for a truly intelligent overhaul.

NVIDIA is redefining game AI with the introduction of NVIDIA ACE autonomous game characters.  Powered by generative AI, NVIDIA ACE enables living, dynamic game worlds with companions that comprehend and support player goals, and enemies that adapt dynamically to player tactics.

Enabling these autonomous characters are new ACE small language models (SLMs) capable of planning at human-like frequencies required for realistic decision making, and multimodal SLMs for vision and audio that allow AI characters to hear audio cues and perceive their environment.

Replicating human decision-making requires game characters to interact naturally with players, perceive their environment, reason through potential options and strategically plan and execute their next move within hundreds of milliseconds.

Innovative distillation techniques and advancements in GeForce RTX 50 Series GPUs enable multiple AI models to run simultaneously on a GPU, giving autonomy to game characters while maximizing the image quality and performance of the game. New and NVIDIA ACE models coming soon include the following.

Models for perception

• NemoAudio 4B Instruct takes audio and text in and produces text out and is capable of describing a soundscape in a gaming environment.
• Cosmos Nemotron 4B 128K Instruct takes images and text in and produces text out and is capable of simple spatial understanding. 
• Parakeet CTC XXL 1.1B Multilingual transcribes multilingual audio to text.
• NV-OCR quickly recognizes characters and text on screen and transcribes into plain text.

Models for cognition

• Mistral Nemo Minitron 8B 128K Instruct is state-of-the-art for instruction following capabilities and allows game characters to accurately roleplay.
• Mistral Nemo Minitron 4B 128K Instruct reduces accuracy for less VRAM consumption.
• Mistral Nemo Minitron 2B 128K Instruct is our smallest version, which uses 1.5 GB of VRAM.

Models for action

• A² Flow is a zero-shot text-to-speech model that supports several languages.

Perception, cognition, and actions are stored in an optimized embedding model that game characters continuously recall to make their next decision. Implementing these models within your agentic framework will drive human-like decision making in real-time gameplay scenarios. 

KRAFTON’s PUBG Ally demonstrates a Co-Playable Character (CPC) that strategically plans and executes clever tactics based on real-time interactions, dynamically working with a human player to ensure survival. KRAFTON’s InZOI, their upcoming life simulation game, demonstrates Smart Zoi: characters that adjust their behavior through actions they take throughout the day. 

Wemade’s MIR5, their upcoming MMORPG, delivers an intelligent raid boss that learns from past battles and adapts its tactics based on player behavior, creating more engaging experiences every run. NetEase’s NARAKA: BLADEPOINT MOBILE PC VERSION has an intelligent AI companion that carries out actions based on commands from the player.

Audio-driven AI facial animation will soon get a big update, making characters more expressive. NVIDIA announced a new Audio2Face real-time diffusion-based architecture that delivers more accurate and detailed lip-sync, improved dynamic and lively emotions, and natural nonverbal responses. 

The latest NVIDIA ACE on-device models for agentic workflows are coming soon in early access. Get started on your PC deployment with NVIDIA In-Game Inferencing (IGI) SDK, which preconfigures the PC with the necessary AI models, engines, and dependencies. It orchestrates in-process AI inference for C++ games and applications and supports all major inference backends across different hardware accelerators.