Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for multithreading. #205

Open
Drvanon opened this issue Aug 16, 2023 · 1 comment
Open

Support for multithreading. #205

Drvanon opened this issue Aug 16, 2023 · 1 comment

Comments

@Drvanon
Copy link

Drvanon commented Aug 16, 2023

Chains offer an amazing opportunity for parallalelization, since unless a call to "thru" is encountered (or a function accepts the whole input), all calls can be parallelized. Right now, when I execute the following:

>>>  py_(range(5)).map(time.sleep).for_each(lambda _: datetime.now()).value()
[datetime.datetime(2023, 8, 16, 13, 22, 11, 875152),
 datetime.datetime(2023, 8, 16, 13, 22, 11, 875152),
 datetime.datetime(2023, 8, 16, 13, 22, 11, 875152),
 datetime.datetime(2023, 8, 16, 13, 22, 11, 875152),
 datetime.datetime(2023, 8, 16, 13, 22, 11, 875152)]

Where as I believe that the following output would also be quite possible:

>>>  py_(range(5)).map(time.sleep).for_each(lambda _: datetime.now()).value()
[datetime.datetime(2023, 8, 16, 13, 22, 11, 875152),
 datetime.datetime(2023, 8, 16, 13, 22, 12, 875152),
 datetime.datetime(2023, 8, 16, 13, 22, 13, 875152),
 datetime.datetime(2023, 8, 16, 13, 22, 14, 875152),
 datetime.datetime(2023, 8, 16, 13, 22, 15, 875152)]
@Drvanon
Copy link
Author

Drvanon commented Aug 17, 2023

I was thinking about this earlier today. Though possible (and significantly benificial) to identify chain sections that do not depend on each other and perform those parts in parallel, that might be very challenging. What might be simpler is to provide a parallel API for the map functions, where the pydash.collections.itermap function is replaced with a threaded_map function.

Something maybe like:

import multiprocessing
pool = multiprocessing.Pool()

def pooled_iter_map(collection, iteratee):
    return pool.map(collection, iteratee)

def pooled_map(collection, iteratee):
    return list(pooled_iter_map(collection, iteratee))

def pooled_flat_map(collection, iteratee=None):
    return pyd.flatten(pooled_iter_map(collection, iteratee=iteratee))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant