Training improvement

[binhphamthanh]

Hello everyone,
I'm working on a single speaker model training from scratch using piper in colab environment.
My approach is to use multiple small datasets (200 sentences/train).
Then I fine-tune the last trained checkpoint with another small dataset.
So far, everything is going well.

However, I have some concerns:

1. I'm setting the maximum 2000 epochs/train, but it seems too much. I don't see much difference between the testing results of 1000 and 2000 checkpoints. Can you please suggest some tips to avoid wasting training time?

2. Is it possible to do multiple training sessions, and then merge the checkpoints?
   Something like this:

import torch
# Load checkpoints
checkpoint1 = torch.load("checkpoint_dataset1.ckpt")
checkpoint2 = torch.load("checkpoint_dataset2.ckpt")
# Merge checkpoints
state_dict = checkpoint1["state_dict"]
state_dict.update(checkpoint2["state_dict"])
# Save merged checkpoints
torch.save(state_dict, "checkpoints.ckpt")

Appreciate your help!