Nvidia clarifies DLSS 5 infers on 2D screencaps plus motion vectors and does not use existing geometry, textures or lighting – NotebookCheck.net News

Immediately after the public backlash prompted by the DLSS 5 announcement, Nvidia provided additional info regarding the underlying systems that the upcoming technology is based on, yet most of the statements were still unclear and up to interpretation. Youtuber Daniel Owen directly contacted Nvidia’s Jacob Freeman in order to clarify all the carefully crafted verbiage from the official statements and actually spell out what DLSS 5 actually does. Nvidia chose to answer via email rather than arrange a live conversation.

1. Owen starts out with the burning question. What is DLSS 5 actually doing behind the scenes? The guys over at Digital Foundry already hinted at this in their extended analysis, so Owen went ahead and asked if DLSS 5 is just taking a 2D frame (a screencap of what is displayed on the screen) combined with motion vectors and then outputs the result. 

   The Nvidia official confirmed this plain and simple. So DLSS 5 is turning the screen output into its input that then runs through a generative AI model to “enhance” it. The final output is rendered on top of the game engine.
    

2. The next question is intended to clarify if DLSS 5 can in any way analyze 3D geometry and depth or it is just reinterpreting the 2D image as best as it can to make it more realistic based on inference. 

   “DLSS 5 is trained end to end to understand complex scene semantics such as characters, hair, fabric and translucent skin along with environmental lighting conditions like front-lit, back-lit or overcast – all by analyzing a single frame.” 

   Here, Nvidia seems to entirely dodge the 3D geometry part, but its answer suggests that DLSS 5 has nothing to do with 3D geometry.
    

3. If the underlying geometry and textures remain unchanged as Nvidia claims in its official primer, then why do some examples exhibit modified textures, albeit subtly, but still seemingly different compared to the original render? Owen includes an example with a male character from Starfield that appears to have more hair on his left temple after the DLSS 5 treatment. 

   Nvidia dodged this question completely and stated that the underlying geometry is indeed unchanged and what they have presented is only a very early preview of the tech. The youtuber then ventures to say that the geometry may not be changed, but the final output that appears on the screen can very well be whatever DLSS 5 deems correct through inference. Also, if this is an early version, Nvidia should take all the feedback and improve what the technology can ultimately do.
    

4. In the official primer, Nvidia claims that DLSS 5 enhances PBR properties on materials. Owen wants to clarify if DLSS 5 is actually taking the game developer PBR specifications included in the game engine (material properties, roughness, normal maps etc.) as inputs or DLSS 5 is only inferring all the PBR properties. 

   Nvidia answers that DLSS 5 only takes rendered frame and motion vectors as inputs, while materials are inferred from the rendered frame. Owen argues that this clarification qualifies the official statements regarding PBR property enhancement to be misleading.
    

5. This question is quite loaded as it deals with the preservation of artistic intent. Owen essentially asks if game developers can control anything beyond color grading other than turning intensity of the DLSS 5 feature down or completely masking it on specific objects / characters. He gives an example with the opening scene of RE: Requiem, where Grace originally appears to not wear makeup, but the DLSS 5 render shows her wearing makeup. In other words, can developers somehow instruct DLSS 5 to render Grace without wearing any makeup? 

   Nvidia gives a lengthy explanation that is taken directly from the primer, but does not provide an answer regarding the possibility to instruct DLSS 5 to render something in a specific way. As it stands right now, it seems like the early version cannot actually preserve artistic intent, but one may be optimistic and say that could change given enough feedback.
    
6. Is DLSS 5 really limited to screen space so it is not aware of off-screen light sources or reflections or ambient occlusion and shadows? 

   Nvidia confirms that DLSS 5 is indeed limited to screen space, meaning that it does not even understand ray tracing. Then why would gamers even activate ray tracing in games if DLSS 5 does not take those complex calculations that require so much processing power into account? Are we supposed to understand that DLSS 5 can replace ray tracing altogether, and if so, does it do it in a more efficient way that requires less processing power? Using two RTX 5090 cards clearly does not help demonstrate efficiency.
    

7. The last question is not directly related to DLSS 5 but Owen asks if the introduction of DirectML to DirectX for hardware agnostic acceleration of ML tasks in games could mean that proprietary technologies like DLSS 5 may at some point become open source to work on any hardware brand. 

   Nvidia says it has nothing to announce in this regard right now so it is not completely denying the possibility, somehow leaving the door slightly open for a glimmer of hope.

Related Articles

Bogdan Solca – Senior Tech Writer – 2406 articles published on Notebookcheck since 2017

I first stepped into the wondrous IT&C world when I was around seven years old. I was instantly fascinated by computerized graphics, whether they were from games or 3D applications like 3D Max. I’m also an avid reader of science fiction, an astrophysics aficionado, and a crypto geek. I started writing PC-related articles for Softpedia and a few blogs back in 2006. I joined the Notebookcheck team in the summer of 2017 and am currently a senior tech writer mostly covering processor, GPU, and laptop news.