Dataset design ideas

[avihu111]

Some thoughts about fuse dataset structure.

the DatasetDefault class is not super intuitive in my opinion. few issues:

1. In most datasets, you simply load them, and do not need to specify sample ids. In this it is mandatory
2. The feature allow_uncached_sample_morphing is pretty complex and exposes too much under-the-hood IMO.
   allow_uncached_sample_morphing: when enabled, allows an Op, to return None, or to return multiple samples (in a list)

In contrast, the class DatasetWrapSeqToDict fits the "spirit" of the library (take PyTorch modules and wrap them in a NDict for simpler extendibility) - but most examples do not use it.

IMO, the fuse dataset should expect to wrap a PyTorch dataset and expose some add functionality above it (naming the dataset outputs, possibly caching), rather trying to replace it.

[avihu111]

Another comment - is it reasonable to assume that all the labels can be stored in memory?
This could save some runtime when initializing the batch sampler, which re-loads every sample from the hard drive to get its label.

[mosheraboh]

Thanks @avihu111 !
I would schedule a meeting with everyone who is interested to brainstorm.

Meanwhile,
It looks like DatasetWrapToDict is what you are looking for, and your suggestion is to drop DatasetDefault. Am I right?
If so, I guess we would need to go over the benefits of DatasetDefault and discuss if those are worth the overhead.
It would be great if you can come up with an example of when using DatasetDefault requires significant overhead.

[YoelShoshan]

Sharing my perspective on this:

About "allow_uncached_sample_morphing" - what alternative do we have?
removing that optional feature completely isn't good IMHO, as it's useful for debugging sample-morphing cases, or for running such cases when caching is disabled.
We can possibly improve the comment and add "advanced" or "only for debugging" or something like that, if you think it will be less intimidating for newcomers