Rendering Is Becoming a Reconstruction Stack

Last August, I wrote that the final pixel would not always come directly from the renderer. My reasoning was that geometry, camera state and scene structure would remain reliable ground truth, while more of the final appearance would move into learned reconstruction.

CES 2025 makes that direction easier to see.

NVIDIA's DLSS 4 can generate up to three additional frames for every traditionally rendered frame. Super Resolution and Ray Reconstruction are moving to transformer models. RTX Neural Shaders bring small neural networks into shader execution for materials, textures and lighting. At the same time, RTX Mega Geometry is designed to ray trace far more explicit geometry.

That combination matters more to me than any single feature. The future graphics pipeline does not look like neural rendering replacing conventional rendering. It looks like conventional rendering producing increasingly rich signals that learned systems reconstruct into the image we finally see.

One rendered frame is no longer the displayed frame

DLSS has been moving in this direction for years, but DLSS 4 makes the split unusually clear.

Multi Frame Generation can create up to three additional frames from one rendered frame. NVIDIA says that when Multi Frame Generation, Ray Reconstruction and Super Resolution work together, the system can generate 15 out of every 16 displayed pixels rather than shade each one through traditional rendering.

That does not mean the renderer did 1/16 of the work. The generated pixels depend on information produced by the rendering pipeline: motion vectors, depth, ray-traced samples and previous frames. The expensive physical calculation is becoming a sparse set of constraints from which a larger visual result can be reconstructed.

If I render internally at a lower resolution, reconstruct it to the target resolution, replace traditional ray denoisers with a learned model and then synthesise intermediate frames, the display output is the result of a stack. No single stage owns the final image.

The renderer provides evidence. Reconstruction converts that evidence into presentation.

Reconstruction is moving deeper into the pipeline

DLSS 4 is not only adding more generated frames.

NVIDIA is replacing the previous CNN architecture used by Super Resolution, Ray Reconstruction and DLAA with a vision transformer. The new model uses self-attention across pixels and multiple frames to decide which information matters for reconstruction. NVIDIA says this improves temporal stability, moving detail and ghosting behaviour.

Ray Reconstruction replaces hand-designed denoising stages with one learned system. Super Resolution reconstructs a higher-resolution result from lower-resolution inputs. Frame Generation constructs images that were never conventionally rendered at all.

These are becoming cooperating stages inside frame production rather than isolated effects after a finished frame.

The renderer therefore has a new optimisation problem: decide which signals deserve expensive deterministic computation and which parts of the result can be inferred from those signals.

A shader can now contain inference

The more interesting CES announcement may be RTX Neural Shaders.

NVIDIA is proposing small neural networks that execute inside programmable shaders using Tensor Cores. Initial uses include neural texture compression, neural materials and neural radiance caching. RTX Neural Materials compresses complex layered material logic into learned representations, while Neural Radiance Cache learns multi-bounce indirect lighting after an initial set of path-traced bounces.

Microsoft announced planned Cooperative Vector support for DirectX on the same day. Its purpose is to let AI matrix-vector operations run inside shader stages, allowing a small neural network to execute in something like a pixel shader without turning the whole GPU over to a separate ML workload. Microsoft says it is working with AMD, Intel, NVIDIA and Qualcomm on cross-vendor support.

For decades, shader code has been a deterministic program written by a graphics programmer. We are now moving toward shaders that can contain learned functions trained from data.

A material may still have physically meaningful inputs. Lighting may still begin with ray-traced samples. But part of the mapping from those inputs to appearance can increasingly be learned instead of written line by line.

Geometry is not disappearing

This is where I differ from the simplest version of the "AI will replace rendering" prediction.

At the same CES, NVIDIA announced RTX Mega Geometry, designed to ray trace up to 100 times more triangles than current approaches by changing how acceleration structures are updated.

That is almost the opposite of removing geometry.

As reconstruction becomes stronger, explicit geometry can become more valuable because it gives neural stages better constraints. Depth, motion, visibility, surface orientation and camera projection are difficult things to invent reliably across time. A 3D scene already knows them.

This is why I still expect geometry to remain production ground truth.

In Blender, I want the object to have the correct dimensions. In product visualisation, I want the camera and silhouette to be exact. In a game, collision, animation, visibility and interaction all need deterministic spatial state.

What can change is how much appearance has to be authored and calculated explicitly after that state is known.

Rendering starts to look like signal production

The phrase I keep coming back to is signal quality.

A traditional renderer is judged mainly by the image it produces. A reconstruction stack should be judged by both the image and the intermediate signals feeding later stages.

Bad motion vectors can break frame generation. Unstable depth or disocclusion data can create temporal artifacts. Too few useful ray samples can leave reconstruction with weak evidence. Geometry errors can become more visible when a learned system confidently reconstructs around the wrong structure.

For a technical artist or graphics developer, debugging therefore becomes more layered. The geometry may be wrong, the motion data may be wrong, sampling may have missed information, the reconstruction stage may have inferred the wrong detail, or temporal history may be carrying an error from earlier frames.

That is a different production mindset from treating the renderer as one black box.

What I would change in a production pipeline

For 3D production, I would invest more heavily in clean structural data rather than trying to bake every last visual detail into the source asset.

Geometry, UVs, material segmentation, camera metadata, motion and physically meaningful surface properties remain reusable signals. Learned appearance techniques can change much faster than those assets.

I would keep neural stages replaceable.

If Super Resolution, Ray Reconstruction or neural material models improve every hardware generation, I do not want scene truth tied to one reconstruction method. The asset pipeline should preserve high-quality source data and allow the final appearance layer to evolve.

For game development, rendering settings may start to look more like budget allocation. Instead of asking only how many rays or samples a frame can afford, engines may increasingly decide how much compute goes to geometry, sparse physical sampling and neural inference.

For product visualisation, the same idea could eventually matter for more than speed. Learned reconstruction may handle visual complexity that is expensive to describe manually while geometry preserves the product itself.

My prediction

I think the renderer is becoming one part of a reconstruction stack.

The deterministic part of the pipeline will continue to own the things that need to remain true: geometry, camera state, motion, visibility and a useful amount of physical light transport.

The learned part will take a growing share of the work required to turn those signals into the final image: denoising, super resolution, temporal reconstruction, material response, indirect lighting and generated frames.

I do not expect this boundary to settle quickly. Some learned techniques will fail under motion, unusual materials or production edge cases. Offline rendering and real-time graphics will move at different speeds.

But the direction now looks less like a single neural renderer replacing the graphics pipeline and more like neural inference spreading through it.

That is close to what I expected last year.

The renderer still defines the scene.

Increasingly, the final image is reconstructed from what the renderer knows.

Sources

1. NVIDIA, NVIDIA DLSS 4 Introduces Multi Frame Generation & Enhancements For All DLSS Technologies, 6 January 2025.
2. NVIDIA Developer, NVIDIA RTX Neural Rendering Introduces Next Era of AI-Powered Graphics Innovation, 6 January 2025.
3. Microsoft DirectX Developer Blog, Enabling Neural Rendering in DirectX: Cooperative Vector Support Coming Soon, 6 January 2025.
4. NVIDIA Research, Real-Time Neural Appearance Models, presented at SIGGRAPH 2024.
5. NVIDIA Newsroom, NVIDIA Blackwell GeForce RTX 50 Series Opens New World of AI Computer Graphics, 6 January 2025.