Worker optimization #5

bobluppes · 2021-04-25T12:47:37Z

Adds a separate version of the dask worker class that optimizes the incoming task graph for incoming tasks.

…orker node

…stom_re2 op

github-actions

Lintly has detected code quality issues in this pull request.

Issues resolved in commit 81e90c3

github-actions

Lintly has detected code quality issues in this pull request.

github-actions

Lintly has detected code quality issues in this pull request.

…mode

github-actions

Lintly has detected code quality issues in this pull request.

github-actions

Lintly has detected code quality issues in this pull request.

github-actions

Lintly has detected code quality issues in this pull request.

github-actions

Lintly has detected code quality issues in this pull request.

github-actions

Lintly has detected code quality issues in this pull request.

github-actions

Lintly has detected code quality issues in this pull request.

github-actions

Lintly has detected code quality issues in this pull request.

github-actions

Lintly has detected code quality issues in this pull request.

… helpers.py

github-actions

Lintly has detected code quality issues in this pull request.

github-actions

Lintly has detected code quality issues in this pull request.

github-actions

Lintly has detected code quality issues in this pull request.

bobluppes added 15 commits April 13, 2021 15:10

Added dask distributed to required packages

7089124

Fixed syntax error in setup.py

916412d

Added jupyter server proxy to required packages

eb33ed6

Added jupyter notebooks for scheduler and worker plugins

ffdb453

Removed depricated workers.ipynb

245ee28

Added dask distributed worker space to .gitignore

89ad246

Added some cells for testing the connection between a scheduler and w…

2da94fe

…orker node

Ignore all dask-worker-space directories in project

80a9a94

Added tornado and flake8 to required packages

fba0e9f

Basic setup for an accelerated worker in dask distributed

847a18b

accelerated_worker.py substitutes re2 operator

a246de1

Fixed bug in which accelerated worker did not get any arguments at cu…

03d1414

…stom_re2 op

Added config for Flake8

057d7b8

Added data_generator dir to Flake8 exclude since it is a submodule

aae071a

Added workflow to check flake8 on PR

a68b999

github-actions bot previously requested changes Apr 25, 2021

View reviewed changes

Fixed Flake8 warnings

81e90c3

bobluppes added 8 commits April 25, 2021 15:27

Added linter action badge to README.md

bc449fe

Added simple benchmark for multiple accelerated workers to main.py

ca2f51b

Added tabulate to plots.ipynb

38b049e

Changed CMakeLists.txt ABI to 1

f76ab60

Streamlined testing on AWS

bf72776

Added bokeh to setup.py

ba3bbca

Added option to spin up vanilla workers from start_worker.py

e94e7b3

Added batch size benchmark

9278880

github-actions bot suggested changes May 5, 2021

View reviewed changes

Specify scheduler address on cli for main.py

e69d0f7

github-actions bot suggested changes May 5, 2021

View reviewed changes

Fixed lintly E503

08c0880

Start scheduler explicitly such that cluster is not running in local …

f70761e

…mode

github-actions bot suggested changes May 8, 2021

View reviewed changes

Cleaned main.py and extracted scheduler specific methods to helpers.py

785c917

github-actions bot suggested changes May 8, 2021

View reviewed changes

Remove benchmark dry run

af00024

github-actions bot suggested changes May 8, 2021

View reviewed changes

Updated benchmark config

d69da99

github-actions bot suggested changes May 8, 2021

View reviewed changes

Fixed keyerror in main.py

5be4ab6

github-actions bot suggested changes May 8, 2021

View reviewed changes

Dont overwrite data dict and set repeats to 1

622f23e

github-actions bot suggested changes May 8, 2021

View reviewed changes

Set repeats to 5

23ad475

github-actions bot suggested changes May 8, 2021

View reviewed changes

Made plots for both cluster benchmarks in plots-worker.ipynb

75abf66

github-actions bot suggested changes May 8, 2021

View reviewed changes

bobluppes added 4 commits May 8, 2021 23:01

Added newline to end of utils.py

c4dfc8f

Added option to start an RE2 accelerated worker

40e8384

Cleaned up main.py and extracted functionalities to benchmarks.py and…

28183cf

… helpers.py

Added method to warm workers, effectively scattering input parquet files

c9b6159

github-actions bot suggested changes May 8, 2021

View reviewed changes

6 repeats and dont consider first 3 runs in benchmark

e8f1b8d

github-actions bot suggested changes May 9, 2021

View reviewed changes

Warm workers in main loop and do consider first 3 runs

1c05e77

github-actions bot suggested changes May 9, 2021

View reviewed changes

bobluppes added 3 commits May 9, 2021 16:55

Use underscore variable for dry run result in benchmarks.py

8cf1608

Newest plots in plots-worker.ipynb

d3cfeba

Open changes in plots-worker.ipynb

702d9a2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Worker optimization #5

Worker optimization #5

bobluppes commented Apr 25, 2021

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

Worker optimization #5

Are you sure you want to change the base?

Worker optimization #5

Conversation

bobluppes commented Apr 25, 2021

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment