Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Concurrent loading of ensemble #77

Closed
wants to merge 24 commits into from
Closed

WIP: Concurrent loading of ensemble #77

Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
f7abe42
Concurrent loading of ensemble
wouterjdb Dec 9, 2019
06910ed
Removed some spaces
wouterjdb Dec 10, 2019
37cbc42
Remove double code, changed camel case
wouterjdb Dec 10, 2019
eea4314
black
wouterjdb Dec 10, 2019
d16f525
Leaner writing of functional calls
wouterjdb Dec 10, 2019
24492f9
Small change in documentation
wouterjdb Dec 10, 2019
ac061e8
Added concurrent to load_file()
wouterjdb Dec 10, 2019
983e927
Fix static realidxregexp and autodiscovery
wouterjdb Dec 10, 2019
5010057
Added index, minor documentation change
wouterjdb Dec 10, 2019
f2565fb
Limit concurrent.futures import to ProcessPollExecutor
wouterjdb Dec 10, 2019
184cea7
Minor documentation changes
wouterjdb Dec 10, 2019
2b49f8b
Fixed missing positional argument names
wouterjdb Dec 10, 2019
58d55cf
Changed argument creation for concurrent
wouterjdb Dec 10, 2019
08fac79
Fix for concurrent not updating self
wouterjdb Dec 10, 2019
3904d21
Concurrent to load_smry
wouterjdb Dec 10, 2019
28e9870
Alternative way to test Python version
wouterjdb Dec 11, 2019
345ce26
removing caching for parallel
wouterjdb Dec 11, 2019
7ac23bb
Removed unused code, black
wouterjdb Dec 11, 2019
57a3ccb
Minor code fixes
wouterjdb Dec 11, 2019
cc8df36
Fix Python2.7 cach removal
wouterjdb Dec 11, 2019
32e9ec8
Added pylint disable W0212
wouterjdb Dec 11, 2019
8e76bfd
rm wrapper for ScratchRealization.__init__
berland Dec 11, 2019
e0ac3e7
Minor code fixes + black
wouterjdb Dec 12, 2019
5909079
Changed i to idx because of pylint complaining
wouterjdb Dec 12, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 6 additions & 23 deletions src/fmu/ensemble/ensemble.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,23 +213,6 @@ def _shortcut2path(keys, shortpath):
# calling function handle further errors.
return shortpath

@staticmethod
def _init_scratch_realization(realdir, realidxregexp, autodiscovery):
"""Wrapper function used for concurrent loading

Args:
realdir (str): directory to realization
realidxregexp (str): Regular expression or None
autodiscovery (boolean): whether files can be attempted
auto-discovered

Returns:
realization (ScratchRealization): Initialized ScratchRealization
"""
realization = ScratchRealization(
realdir, realidxregexp=realidxregexp, autodiscovery=autodiscovery
)
return realdir, realization

def _check_loading_of_realization(self, realization, realdir, realidxregexp):
"""Helper function for checking and logging the results of loading a
Expand Down Expand Up @@ -295,12 +278,12 @@ def add_realizations(self, paths, realidxregexp=None, autodiscovery=True):
if USE_CONCURRENT:
args = [(realdir, realidxregexp, autodiscovery) for realdir in globbedpaths]
with ProcessPoolExecutor() as executor:
for realdir, realization in executor.map(
self._init_scratch_realization, *zip(*args)
):
count += self._check_loading_of_realization(
realization, realdir, realidxregexp
)
realfutures = [executor.submit(ScratchRealization, realdir, realidxregexp=realidxregexp,
autodiscovery=autodiscovery) for realdir in globbedpaths]

for realfuture in realfutures:
count += self._check_loading_of_realization(
realfuture.result(), "dummydirfornow", None)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you use dummy arguments?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated those lines in the meanwhile by the way - let me know if that was not what you had in mind.

And I see that for the other usage there is not this additional wrapper (there is one, but it is different.). Though, we might consider changing that to the submit() method as well - just to use the same thing.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dummydir was just for laziness. Need to find a pretty solution, but it is only used for logging.

else:
for realdir in globbedpaths:
realization = ScratchRealization(
Expand Down