The Final Pixel Won't Come From the Renderer

For most of computer graphics, the renderer has been the final authority. Geometry, materials, lights and cameras go in. Pixels come out. Better rendering has usually meant tracing more rays, sampling more accurately, or finding faster approximations to the same physical process.

I think that boundary is starting to move.

My bet is that geometry and scene structure will remain the reliable ground truth, while the final photorealistic appearance will increasingly be reconstructed by neural systems. The renderer will still matter, but it may stop being the component that determines every visible pixel directly.

Current tools do not do this end to end. The evidence is split across denoising, upscaling, neural materials and scene reconstruction, but the direction is becoming easier to see.

The renderer is already giving up pixels

DLSS is the clearest production example.

Real-time rendering works inside a fixed compute budget. When native-resolution path tracing is too expensive, developers reduce samples or resolution and reconstruct the missing information.

With DLSS 3.5 Ray Reconstruction, NVIDIA replaces several hand-tuned denoisers with a neural network trained to reconstruct higher-quality ray-traced images from sparse samples and temporal data. NVIDIA describes the network as learning lighting patterns from offline-rendered reference images and using them to infer better pixels during real-time rendering.

That changes the role of the renderer. It produces physically meaningful evidence: geometry, motion, sampled rays, depth and lighting information. The neural stage decides how that incomplete evidence should become a clean final image.

Super Resolution pushes the same idea further. Frame Generation goes further again: some displayed frames are synthesised rather than rendered conventionally.

The important point is not faster rendering. The final image is becoming a reconstruction problem.

Rendering has always contained reconstruction

Renderers have always reconstructed to some degree. Path tracing estimates an integral from finite samples; denoisers estimate a converged result; temporal methods reuse prior frames; upscalers infer detail. Neural methods expand how much of that reconstruction can be learned rather than hand-designed.

DLSS 3.5 is interesting because NVIDIA is not placing the neural model after a finished render as a cosmetic effect. Ray Reconstruction replaces part of the ray-tracing pipeline itself. The network sits between sparse physical samples and the image we see.

Once that works reliably, it is reasonable to ask how much more of the appearance pipeline can move into learned representations.

Materials are starting to become learned representations

Material systems are another useful signal.

NVIDIA's Real-Time Neural Appearance Models work shows that complex layered materials can be baked into compact learned representations and evaluated with neural decoders inside a real-time path tracer. The researchers report that these neural material shaders can be more than an order of magnitude faster to evaluate than the original non-neural layered materials while retaining complex appearance.

This is not diffusion-style generative rendering. The material remains constrained by scene, view and light transport, but part of its appearance no longer needs a conventional shader graph evaluated node by node.

The same pattern appears in neural texture compression. NVIDIA's 2023 work stores material textures in a learned representation and decodes them on demand, reporting four times the linear resolution in its examples while using less memory than conventional block-compressed textures.

From a production perspective, this matters because textures and shaders have always been part of the source description of appearance. If those representations become learned, the boundary between asset and renderer becomes less rigid.

Scene representation is moving too

3D Gaussian Splatting is a different branch of the same story.

The 2023 work by Kerbl and colleagues represents a captured scene using optimised 3D Gaussians and renders new viewpoints in real time. It does not use a conventional polygon-and-material pipeline to reproduce appearance. The representation itself stores spatially organised appearance information learned from images.

For product visualisation, games and VFX, I do not expect geometry to disappear. A mesh gives us topology, scale, contact, occlusion and editable structure. Cameras, animation and lights provide more controllable constraints. I expect that explicit scene structure to become more valuable as a control signal, even if the photorealistic surface of the final frame becomes less rigidly represented.

That distinction is the core of my prediction.

Geometry as truth, appearance as reconstruction

When I work in Blender or a real-time engine, I care about two different kinds of correctness.

The first is structural correctness: the product is the right shape, proportions hold, the camera matches, the animation contacts the floor, and the object casts its shadow from the right location.

The second is perceptual fidelity: leather that reads as leather, brushed metal that breaks up light correctly, and scratches, dust, fingerprints and subsurface effects that hold up at the final viewing distance.

Traditional rendering solves both through the same pipeline. We build detailed geometry, texture maps, procedural materials, lights and simulation until the renderer can calculate the final appearance.

I suspect these two responsibilities will separate.

Geometry, animation and coarse physical simulation can remain explicit because they are useful for control. The neural layer can increasingly handle the expensive tail of visual realism: fine material response, high-frequency detail, reconstruction from sparse rays and perhaps eventually parts of lighting appearance that are expensive to sample directly.

The renderer then becomes less like a camera producing the finished photograph and more like a measurement system producing trusted signals for another stage.

Why I would not replace the renderer with a generator

There is an obvious bad version of this idea: render a rough frame, send it through an image generator and accept whatever looks photorealistic.

I would not trust that pipeline. A logo can move, a seam can disappear, or a reflection can imply geometry that is not present. Across animation frames, those mistakes become temporal instability.

A useful neural renderer needs tighter conditioning than a prompt and a low-resolution image.

Depth, normals, motion vectors, object IDs, material properties, lighting samples and camera state are far more useful constraints. They give the neural stage freedom where appearance is expensive to compute while preserving the parts of the scene that production needs to control.

This is why I see hybrid rendering as the likely path. Deterministic graphics establish the scene. Learned systems reconstruct increasingly large parts of its appearance.

What this could change in production

If this direction holds, artists may spend less time manufacturing detail purely so a renderer can reproduce it pixel by pixel.

For product work, I would still build accurate geometry, cameras, animation and controlled lighting. But some microscopic appearance detail could move into learned reconstruction.

Game engines could make the same trade: decide what must be physically sampled and what can be reconstructed reliably. Polygon budgets, texture memory and shader complexity would remain, but signal quality and temporal stability would become larger parts of optimisation.

My prediction

Over the next few years, I expect the phrase "rendered frame" to become less literal.

A frame may contain geometry processed conventionally, a limited set of ray-traced samples, temporally reused information, neural material evaluation, reconstructed lighting, synthesised detail and pixels that were never directly shaded at native resolution.

The mix will depend on the application. Games need temporal stability and predictable latency; product visualisation needs shape and material fidelity; film can spend more compute but tolerates fewer artifacts.

I do not expect physics-based rendering to become obsolete. I expect it to move upstream.

It can provide reference data, training targets and hard constraints. Neural systems can then spend their capacity on the parts of image formation where brute-force sampling is expensive or where learned priors give a better trade between quality and compute.

That is the direction I would design for now: keep geometry explicit, keep production state deterministic, and assume that the layer producing the final photorealistic appearance will become increasingly learned.

The renderer will still tell us what the scene is.

It may no longer be the thing that paints every final pixel.

Sources

1. NVIDIA, NVIDIA DLSS 3.5: Enhancing Ray Tracing With AI, 22 August 2023.
2. Zeltner et al. / NVIDIA Research, Real-Time Neural Appearance Models, presented at SIGGRAPH 2024.
3. Vaidyanathan et al. / NVIDIA Research, Random-Access Neural Compression of Material Textures, SIGGRAPH 2023.
4. Kerbl et al., 3D Gaussian Splatting for Real-Time Radiance Field Rendering, SIGGRAPH 2023.
5. NVIDIA Research, AI and Simulation Advancements at SIGGRAPH 2024, 12 July 2024.