Transitioning to Julian datasets and data loaders

[darsnack]

I wanted to add to this the general discussion from Zulip (linked below). It's best we transfer concrete suggestions to here so it's available for everyone to contribute. The plan has evolved into two "milestones" — short-term and long-term. I'll summarize both below.

Short-term:

• Implement iterable and map-like datasets à la PyTorch
• Implement a DataLoader interface mimicking PyTorch
• The goal here is to take PyTorch implementations as-is to get something up and running, since this is a desperately needed functionality in the ecosystem
• Being tracked here: Implement Dataloader interface #5

Long-term:

• Transfer iterable dataset interface to Base iterator interface [1]
• Transfer the map-like dataset interface to Base indexable collections [1]
• The DataLoader interface doesn't necessarily need to change, but long term we should make sure that concrete implementations of the interface take advantage of things like Base.Random and Distributions for sampling

Relevant Zulip topics:

• https://julialang.zulipchat.com/#narrow/stream/237432-ml-ecosystem-coordination/topic/Datasets
• https://julialang.zulipchat.com/#narrow/stream/237432-ml-ecosystem-coordination/topic/DataLoaders

---

[1]: the Base interfaces may not be perfect for our needs, so we might need to build off of them

[darsnack]

cc @opus111 @SomTambe

[lorenzoh]

I think the LearnBase.jl is being established pretty nicely, with DataLoaders.jl and FastAI.jl fully embracing it.

As an update:

• FastAI.jl also includes all datasets from the fastai dataset repository and prototype data containers for files and tables which will be moved to MLDatasets.jl when stable (see this PR.
• Additionally, there are useful data container transformations that, once stable, will be merged into MLDataPattern.jl.

[darsnack]

I think most of the original issue has effectively been completed by adopting JuliaML/LearnBase.jl. In addition to the dataset transfer and data container transformations above, there are some short-term objectives that need to be met:

• Updating/refreshing LearnBase.jl's interface so that future development is easier: Refactor ObsDim out of LearnBase and update related interfaces JuliaML/LearnBase.jl#44
  • Updating MLDataPattern to reflect these changes: Updated to reflect new LearnBase interface for getobs and ObsDim JuliaML/MLDataPattern.jl#50
  • Updating MLLabelUtils to reflect these changes: Refactor MLLabelUtils to reflect obsdim changes in LearnBase.jl JuliaML/MLLabelUtils.jl#44

[lorenzoh]

Yeah, I didn't want to close this, just to give an update on what's been happening 👍

[darsnack]

Closing this since we either have completed the sub-issue as part of the LearnBase.jl -> MLUtils.jl transition, or we have a tracking issue on MLUtils.jl / MLDatasets.jl itself.