New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

exploring tiledb's dataloaders #38

Open

jkanche opened this issue Aug 4, 2024 · 1 comment

Assignees

Member

jkanche commented Aug 4, 2024

I ran into Tiledb's ML repo (https://github.com/TileDB-Inc/TileDB-ML/blob/master/tiledb/ml/readers/pytorch.py), which seems to implement a pytorch based dataloader. It looks like we can speed up our dataloaders without setting threads to 1.

Mostly exploration and then figuring out if we can adopt the same logic to our implementations.

jkanche assigned tony-kuo

Collaborator

tony-kuo commented Aug 12, 2024

Setting context to "spawn" will remove the need to setting threads to 1.

I'm going to refactor the dataloader to reduce the number of database accesses to one per mini-batch
I'm going to move to a sample based epoch rather than multi-set. I think it makes more sense.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment