[QUESTION] Is OpenGL Renderer Differentiable?

[rrzhang139]

Hello, I am trying to write a script that simulates a model's state then renders it into a frame using wp.sim.render.SimRendererOpenGL's get_pixels.

        for frame in range(self.num_frames):
            with wp.ScopedTimer("simulate"):
                for i in range(self.sim_substeps):
                    self.states[frame * self.sim_substeps + i].clear_forces()
                    
                    wp.launch(
                        kernel=eval_external_particle_forces,
                        dim=self.model.particle_count,
                        inputs=[self.particle_activations],
                        outputs=[self.states[frame * self.sim_substeps + i].particle_f]
                    )
                    self.integrator.simulate(
                        self.model, 
                        self.states[frame * self.sim_substeps + i], 
                        self.states[frame * self.sim_substeps + i + 1], 
                        self.sim_dt
                    )
                    self.sim_time += self.sim_dt

                with wp.ScopedTimer("render", active=self.train):
                    self.renderer.begin_frame(self.render_time)
                    self.renderer.render(self.states[frame * self.sim_substeps])
                    self.save_frame(frame)
                    self.renderer.get_pixels(
                        target_image=self.pixels,
                        split_up_tiles=False,
                        mode="rgb" if self.channels == 3 else "depth"
                    )
                    frame_copy = wp.clone(self.pixels)
                    self.pred_frames.append(frame_copy)
                    self.renderer.end_frame()
                    self.render_time += self.frame_dt

However, my gradients are zero suspecting that the gradients may be cut off. Do you know if this renderer is differentiable. And if not are there some examples that work (e.g diffray)?

[ucacaxm]

Hi,

The sim renderer (warp/sim/render.py) is based on
warp/render/render_opengl.py
which is an OpenGL renderer using the GL pipeline of the GPU. It is not differentiable.

There is a discussion here about a diff renderer:
#151

My little contribution...
Best

[daedalus5]

Hi yes if your model is a single mesh, example_diffray should give you an idea of how to include (primitive) differentiable rendering in your graph. We don't have an official Warp module for this, but it should be easy to copy what the example is doing.