Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Parallel Processing with Dask [RFC] #128

Open
tallamjr opened this issue May 8, 2019 · 1 comment
Open

[FEATURE] Parallel Processing with Dask [RFC] #128

tallamjr opened this issue May 8, 2019 · 1 comment
Assignees
@tallamjr
Labels
feature To add a new feature, new standalone files. (High level)

Comments

@tallamjr
Copy link
Collaborator

tallamjr commented May 8, 2019

As there seems to be a preference to incorporate more pandas functionality into core snmachine, here I propose a discussion for bringing dask (https://docs.dask.org/en/latest/) into the codebase for dealing with data > size of RAM but keeping to a familiar API similar to pandas

If this feature is desired, it can be discussed now and developed on later, post public release
@tallamjr tallamjr added the feature To add a new feature, new standalone files. (High level) label May 8, 2019
@tallamjr tallamjr self-assigned this May 8, 2019
@tallamjr
Copy link
Collaborator Author

Posting this potentially useful resource for reference later when this comes to be developed further.

It goes through the steps to run on a SLURM system, much like hypatia

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tallamjr tallamjr

Labels
feature To add a new feature, new standalone files. (High level)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tallamjr