Render of NERF being Trained

[felixgeen]

Fox being trained video

I want to render a video with a camera path that shows the nerf being trained progressively over each frame (Like the fox in the video above). I have managed to get somewhere by deleting the "testbed.load_snapshot" function, using the "train_and_render" function and trying to cycle over each frame with "testbed.train(spp)" but im not able to get it to work, it wont render anything.

import os, sys, shutil
import argparse
from tqdm import tqdm
import common
import pyngp as ngp # noqa
import numpy as np

def train_and_render(resolution, numframes, scene, name, bb, spp, fps, exposure=0):
    # Initialize a Testbed object and load the snapshot and camera path from the scene
    testbed = ngp.Testbed(ngp.TestbedMode.Nerf)
    testbed.load_camera_path(os.path.join(scene, "base_cam.json"))

    # If a bounding box is specified, set the render_aabb attribute of the Testbed object
    if bb:
        bbs = bb.split(",")
        testbed.render_aabb = ngp.BoundingBox([bbs[0], bbs[1], bbs[2]], [bbs[3], bbs[4], bbs[5]])

    # Create a directory to store the rendered frames
    directory_name = "render/jpg/{}".format(name)
    if not os.path.exists(directory_name):
        os.makedirs(directory_name)    

    # Train the neural graphics primitives
    for i in tqdm(list(range(min(numframes, numframes+1))), unit="frames", desc=f"Training"):
        testbed.train(spp)

    # Render the scene
    for i in tqdm(list(range(min(numframes, numframes+1))), unit="frames", desc=f"Rendering"):
        # Render the scene and save the rendered frame as a JPEG image
        frame = testbed.render(resolution[0], resolution[1], spp, True, float(i)/numframes, float(i + 1)/numframes, fps, shutter_fraction=0.5)
        common.write_image(f"render/jpg/{name}/{name}_rgb_{i:04d}.jpg", np.clip(frame * 2**exposure, 0.0, 1.0), quality=100)

    # Use ffmpeg to convert the rendered frames into a video file
    os.system(f"ffmpeg -r {fps} -i render/jpg/{name}/%04d.jpg -c:v libx264 -pix_fmt yuv420p render/mp4/{name}.mp4")
def parse_args():
	parser = argparse.ArgumentParser(description="render neural graphics primitives testbed, see documentation for how to")
	parser.add_argument("--scene", "--training_data", default="", help="The scene to load. Can be the scene's name or a full path to the training data.")
	parser.add_argument("--width", "--screenshot_w", type=int, default=1920, help="Resolution width of the render video")
	parser.add_argument("--height", "--screenshot_h", type=int, default=1080, help="Resolution height of the render video")
	parser.add_argument("--n_seconds", type=int, default=1, help="Number of steps to train for before quitting.")
	parser.add_argument("--fps", type=int, default=60, help="number of fps")
	parser.add_argument("--bounding_box", type=str, default="", help="Bounding box info, x,y,z min, x,y,z max. Ensure that min greater than max")
	parser.add_argument("--render_name", type=str, default="", help="name of the result video")
    
	args = parser.parse_args()
	return args

if __name__ == "__main__":
	args = parse_args()	

	train_and_render([args.width, args.height], args.n_seconds*args.fps, args.scene, args.render_name, args.bounding_box, spp=8, fps=args.fps)

Any help much appreciated.

[Tom94]

Hi, several things.

• You need to load a dataset and set testbed.shall_train = True before training
• You need to interleave training with frame rendering, else you'll just do all the training ahead of time, leaving you with a regular video of a finished model
• The testbed.train(...) function actually takes the batch size as input and does a single training step (docs are wrong, sorry, will fix in an update). So calling it as testbed.train(testbed.training_batch_size) should give best results. You'll then need to perform multiple such calls to adjust how much training is done per frame of the video that you render.

[felixgeen]

Thank you Tom! I'll give it a go but it already feels a bit too advanced for my basic python skills.