About dataset update interval

[pansanity666]

Hi,

A quick question is that if the total iteration is 30k, and the update interval is 2.5k, only (30/2.5)=12 images will be edited, which seems to be usually smaller than the dataset size. Is there something wrong with my understanding?

Best,

[StephenYangjz]

Hi @pansanity666, sry this seems to be a separate question: I wonder if you are able to get the most up to date code working? for me it runs without calling ip2p and doing the editing : /

[pansanity666]

Hi @pansanity666, sry this seems to be a separate question: I wonder if you are able to get the most up to date code working? for me it runs without calling ip2p and doing the editing : /

I haven't tried to run this repo yet.

[StephenYangjz]

@pansanity666 Gotcha, if you do it would be greatly appreciated if you can lmk hows it going cuz im having this issue: #10

[cvachha]

Hi @pansanity666, the full dataset of training images is edited/updated every 2.5k iterations. The pipeline in igs2gs_pipeline.py first edits all dataset images with Instruct-Pix2Pix, then trains the splat for 2.5k iterations, then repeats the process.

[pansanity666]

Hi @pansanity666, the full dataset of training images is edited/updated every 2.5k iterations. The pipeline in igs2gs_pipeline.py first edits all dataset images with Instruct-Pix2Pix, then trains the splat for 2.5k iterations, then repeats the process.

Hi,

I understand your explanation but still kind of confused by the implementation code:

1. It seems that you just edit 1 image every config.gs_steps here

   instruct-gs2gs/igs2gs/igs2gs_pipeline.py

   Lines 115 to 132 in 5890bab

   	idx = self.curr_edit_idx
   	camera, data = self.datamanager.next_train_idx(idx)
   	model_outputs = self.model(camera)
   	metrics_dict = self.model.get_metrics_dict(model_outputs, data)
   	original_image = self.datamanager.original_cached_train[idx]["image"].unsqueeze(dim=0).permute(0, 3, 1, 2)
   	rendered_image = model_outputs["rgb"].detach().unsqueeze(dim=0).permute(0, 3, 1, 2)
   	edited_image = self.ip2p.edit_image(
   	self.text_embedding.to(self.ip2p_device),
   	rendered_image.to(self.ip2p_device),
   	original_image.to(self.ip2p_device),
   	guidance_scale=self.config.guidance_scale,
   	image_guidance_scale=self.config.image_guidance_scale,
   	diffusion_steps=self.config.diffusion_steps,
   	lower_bound=self.config.lower_bound,
   	upper_bound=self.config.upper_bound,
   	)

2. Editing the whole dataset might be very time-consuming, making the training very slow?

Best,

[cvachha]

Hi, it iterates through all training images and edits one image at a time (also which ends up training 1 iteration per edited image), so the first [num of images] iterations are just for editing each individual image and training after replacing the original image with the edited one.

Once the full dataset has been updated (and it has trained for [num of images] iterations), we train for 2.5k (gs_steps) iterations on all the edited images, which applies the edit over the entire splat.

Since the edits are progressive (editing over the previously edited images after each dataset update), we generally recommend 2-3 dataset updates (training for about 5k-7.5k iterations). Editing the full dataset is a little time consuming (few seconds per training image) but applying the edits on the splat (training for 2.5k) is pretty quick.