Due to an internal limitation of librviz, the ViewController takes ownership of the camera's transform. It is not possible to directly set the camera transform without also disabling the view controller. It is perhaps possible to create a custom view controller instead to allow this functionality.