Skip to content
This repository has been archived by the owner on May 27, 2024. It is now read-only.

Parallelize tests #16

Closed
grst opened this issue Dec 1, 2023 · 1 comment
Closed

Parallelize tests #16

grst opened this issue Dec 1, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@grst
Copy link
Contributor

grst commented Dec 1, 2023

Some of the methods are embarrassingly parallel, e.g. statsmodels, wilcoxon test.

I suggest to use the following snippet from scirpy:

  • using joblib it is more robust than multiprocessing
  • joblib natively supports the dask backend, so you get out-of-machine support for free.

https://github.com/scverse/scirpy/blob/443e59e6245b917e87972f87df350ae4f429d011/src/scirpy/util/__init__.py#L567-L579

def _parallelize_with_joblib(delayed_objects, *, total=None, **kwargs):
    """Wrapper around joblib.Parallel that shows a progressbar if the backend supports it.

    Progressbar solution from https://stackoverflow.com/a/76726101/2340703
    """
    try:
        return tqdm(Parallel(return_as="generator", **kwargs)(delayed_objects), total=total)
    except ValueError:
        logging.info(
            "Backend doesn't support return_as='generator'. No progress bar will be shown. "
            "Consider setting verbosity in joblib.parallel_config"
        )
        return Parallel(return_as="list", **kwargs)(delayed_objects)

https://github.com/scverse/scirpy/blob/443e59e6245b917e87972f87df350ae4f429d011/src/scirpy/ir_dist/metrics.py#L231-L233

block_results = _parallelize_with_joblib(
         (joblib.delayed(self._compute_block)(*block) for block in blocks), total=len(blocks), n_jobs=self.n_jobs
)
@grst grst added the enhancement New feature or request label Dec 1, 2023
@grst grst changed the title Parallelize StatsmodelsDE Parallelize tests Feb 23, 2024
@grst
Copy link
Contributor Author

grst commented May 27, 2024

Migrated to scverse/pertpy#613

@grst grst closed this as completed May 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant