Scaling musika?

[youssefabdelm]

Hey Marco, thank you so much for blowing us all away! Such impressive work.

I've tried fine-tuning and got some interesting results but I feel like I'm going to get even more mind blowing results if I can scale it.

I have some questions about scaling musika:

1. I want to train a larger version of musika on some more powerful hardware on the cloud (e.g. A100, or RTXA6000). What do you think is the largest number of parameters I can scale to that can still work in terms of inference on a free Colab? For example, can I 5x or 10x the number of parameters on the existing model before pretraining?
2. Also out of curiosity, how big do you think a model would have to be to be indistinguishable from real music?
3. Do you think it would be worth it to train a scaled version of the model without also scaling or re-training the autoencoders (I know you mentioned in another issue that you plan to release this eventually. Would you recommend I wait til that happens?)
4. In the readme you mentioned "You can also increase the number of parameters (capacity) of the model with the --base_channels argument (--base_channels 128 is the default)." How many more parameters would --base_channels 192 have? / How can I calculate final number of parameters and whether that will fit in memory for a free Colab?
5. Do you support multi-GPU training?

I plan to train from scratch, and then fine-tune. For the fine-tuning, how many iterations do you recommend? I've often found when fine-tuning I quickly reach a plateau after 20-30 epochs and am not sure if I should continue training the next day (on Colab) or wait til I've done 250 epochs.