Tests fail due to torch-scatter

[DocMinus]

When running the pytest I get a number of Fs. Most likely due to the install? Had to do some manual hands-on, the out-of-box pip install didn't work.
I am running on WSL2 Unbuntu 22 (with Nvidia GPU).
What I did differently after the first pip install due to errors after bailing out due to torch_scatter:

mamba install -c pyg -c conda-forge torch-scatter
pip install "syntheseus[all]"
pip install typing-extensions==4.6.0 (since 4.5.0 not combatible)

the first run of tests conked out due to graphviz?
Thus I manually installed that via
sudo apt install graphviz
and rerun
pip install "syntheseus[all]"
which reinstalled typing_extenstions 4.5.0.
Running pytests then led to this

syntheseus/tests/cli/test_cli.py ..F...F..F...F..F...F..F...F.                                                                                                                                                                                                                                                        [  5%]
syntheseus/tests/cli/test_eval_single_step.py .............................                                                                                                                                                                                                                                           [ 11%]
syntheseus/tests/cli/test_search.py .                                                                                                                                                                                                                                                                                 [ 11%]
syntheseus/tests/interface/test_bag.py .                                                                                                                                                                                                                                                                              [ 11%]
syntheseus/tests/interface/test_models.py ......                                                                                                                                                                                                                                                                      [ 12%]
syntheseus/tests/interface/test_molecule.py .........                                                                                                                                                                                                                                                                 [ 14%]
syntheseus/tests/interface/test_reaction.py .....                                                                                                                                                                                                                                                                     [ 15%]
syntheseus/tests/interface/test_toy_models.py ..                                                                                                                                                                                                                                                                      [ 15%]
syntheseus/tests/reaction_prediction/chem/test_utils.py ...                                                                                                                                                                                                                                                           [ 16%]
syntheseus/tests/reaction_prediction/data/test_dataset.py ..............                                                                                                                                                                                                                                              [ 19%]
syntheseus/tests/reaction_prediction/inference/test_models.py ..F.......F.....F.......F...                                                                                                                                                                                                                            [ 24%]
syntheseus/tests/reaction_prediction/utils/test_misc.py ....                                                                                                                                                                                                                                                          [ 25%]
syntheseus/tests/reaction_prediction/utils/test_parallel.py ..                                                                                                                                                                                                                                                        [ 25%]
syntheseus/tests/search/algorithms/test_best_first.py .............................F.......                                                                                                                                                                                                                           [ 32%]
syntheseus/tests/search/algorithms/test_breadth_first.py ............................................................................                                                                                                                                                                                 [ 47%]
syntheseus/tests/search/algorithms/test_mcts.py .............................................                                                                                                                                                                                                                         [ 55%]
syntheseus/tests/search/algorithms/test_pdvn.py .......................................                                                                                                                                                                                                                               [ 63%]
syntheseus/tests/search/algorithms/test_random.py ..........................ss..............................ss....                                                                                                                                                                                                    [ 75%]
syntheseus/tests/search/analysis/test_diversity.py ..................                                                                                                                                                                                                                                                 [ 79%]
syntheseus/tests/search/analysis/test_route_extraction.py .......................                                                                                                                                                                                                                                     [ 83%]
syntheseus/tests/search/graph/test_andor.py ...............                                                                                                                                                                                                                                                           [ 86%]
syntheseus/tests/search/graph/test_message_passing.py .......                                                                                                                                                                                                                                                         [ 87%]
syntheseus/tests/search/graph/test_molset.py ...........                                                                                                                                                                                                                                                              [ 90%]
syntheseus/tests/search/graph/test_route.py .                                                                                                                                                                                                                                                                         [ 90%]
syntheseus/tests/search/graph/test_standardization.py ..                                                                                                                                                                                                                                                              [ 90%]
syntheseus/tests/search/node_evaluation/test_common.py .....................................                                                                                                                                                                                                                          [ 97%]
syntheseus/tests/search/test_mol_inventory.py ...                                                                                                                                                                                                                                                                     [ 98%]
syntheseus/tests/search/test_visualization.py ........./home/a/mambaforge/envs/syntheseus-full/lib/python3.9/site-packages/coverage/control.py:892: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")

WARNING: Failed to generate report: No data to report.

/home/a/mambaforge/envs/syntheseus-full/lib/python3.9/site-packages/pytest_cov/plugin.py:329: CovReportWarning: Failed to generate report: No data to report.

and way too much text to copy/paste here. many, not all, seem to refere to torch_scatter where errors like are shown:

OSError: /home/a/mambaforge/envs/syntheseus-full/lib/python3.9/site-packages/torch_scatter/_version_cuda.so: undefined symbol: _ZN3c106detail14torchCheckFailEPKcS2_jRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

another recurring error:

    def _call_impl(self, *args, **kwargs):
        forward_call = (self._slow_forward if torch._C._get_tracing_state() else self.forward)
        # If we don't have any hooks, we want to skip the rest of the logic in
        # this function, and just call forward.
        if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
                or _global_backward_pre_hooks or _global_backward_hooks
                or _global_forward_hooks or _global_forward_pre_hooks):
>           return forward_call(*args, **kwargs)
E           TypeError: forward() got an unexpected keyword argument 'is_causal'

../mambaforge/envs/syntheseus-full/lib/python3.9/site-packages/torch/nn/modules/module.py:1527: TypeError

any ideas?

[AustinT]

Hello @DocMinus, it looks like your issues are related to the installation of torch_scatter. This package seems to always cause problems (I've had it fail before myself). I just tried a fresh re-install with python 3.10 and the install of torch scatter failed for me.

I'm not sure there is an easy solution to this. However, torch scatter is only required by a small subset of the single-step models (retro-knn, maybe some other indirectly). Unless you want to use that specific model you don't actually need to install torch scatter. For example, I ran

pip install "syntheseus[chemformer,graph2edits,local-retro,megan,mhn-react]"

on a linux system (not WSL though) and all the tests passed for me (admittedly, the CLI tests are skipped if not all the single-step models are installed). Could that help get things working for you?

[AustinT]

Separate to above, @kmaziarz does "pip install syntheseus[all]" work for you on WSL (I no longer have a windows computer to test it on). If torch-scatter is causing problems, should we change the dependencies in any way to make most tests run without it?

[DocMinus]

thanks. I will try some of the above suggestions. I am guessing I don't have to bother testing this on Mac Silicone?

[AustinT]

I am guessing I don't have to bother testing this on Mac Silicone?

Yeah, no need I think

[kmaziarz]

Thanks for reporting this @DocMinus, I confirm I can reproduce the issue locally in WSL (but not on CI, so it does seem to be somewhat environment dependent).

I'm looking into what the best fix should be, but it may be a few days before that propagates all the way to syntheseus as released via PyPI. Workaround for now (which does work for me and makes the tests pass) is:

• Create the environment from environment_full.yml afresh.
• Run pip install "torch-scatter @ https://data.pyg.org/whl/torch-1.10.0%2Bcu113/torch_scatter-2.0.9-cp39-cp39-linux_x86_64.whl".
• Clone this repo and run pip install ".[all]".

I think the sudo apt install graphviz may still be needed, as this is outside of the conda/pip environment. But you've done this already, so likely no need to repeat. The graphviz PyPI package should get installed as part of pip install ".[all]".

[DocMinus]

@kmaziarz Thanks. Since my ubuntu/conda install took forever trying to resolve dependencies for the create environment, I installed a fresh Ubuntu (24) with WSL, installed conda python 3.9 and graphviz.
followed your suggestions, only got one error, which I ignored:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
sqlalchemy 2.0.34 requires typing-extensions>=4.6.0, but you have typing-extensions 4.5.0 which is incompatible.

the testing looks much better, warnings mainly regarding depreciation:

collected 520 items

syntheseus/tests/cli/test_cli.py .............................                                                   [  5%]
syntheseus/tests/cli/test_eval_single_step.py .............................                                      [ 11%]
syntheseus/tests/cli/test_search.py .                                                                            [ 11%]
syntheseus/tests/interface/test_bag.py .                                                                         [ 11%]
syntheseus/tests/interface/test_models.py ......                                                                 [ 12%]
syntheseus/tests/interface/test_molecule.py .........                                                            [ 14%]
syntheseus/tests/interface/test_reaction.py .....                                                                [ 15%]
syntheseus/tests/interface/test_toy_models.py ..                                                                 [ 15%]
syntheseus/tests/reaction_prediction/chem/test_utils.py ...                                                      [ 16%]
syntheseus/tests/reaction_prediction/data/test_dataset.py ..............                                         [ 19%]
syntheseus/tests/reaction_prediction/inference/test_models.py ............................                       [ 24%]
syntheseus/tests/reaction_prediction/utils/test_misc.py ....                                                     [ 25%]
syntheseus/tests/reaction_prediction/utils/test_parallel.py ..                                                   [ 25%]
syntheseus/tests/search/algorithms/test_best_first.py .....................................                      [ 32%]
syntheseus/tests/search/algorithms/test_breadth_first.py ....................................................... [ 43%]
.....................                                                                                            [ 47%]
syntheseus/tests/search/algorithms/test_mcts.py .............................................                    [ 55%]
syntheseus/tests/search/algorithms/test_pdvn.py .......................................                          [ 63%]
syntheseus/tests/search/algorithms/test_random.py ..........................ss..............................ss.. [ 75%]
..                                                                                                               [ 75%]
syntheseus/tests/search/analysis/test_diversity.py ..................                                            [ 79%]
syntheseus/tests/search/analysis/test_route_extraction.py .......................                                [ 83%]
syntheseus/tests/search/graph/test_andor.py ...............                                                      [ 86%]
syntheseus/tests/search/graph/test_message_passing.py .......                                                    [ 87%]
syntheseus/tests/search/graph/test_molset.py ...........                                                         [ 90%]
syntheseus/tests/search/graph/test_route.py .                                                                    [ 90%]
syntheseus/tests/search/graph/test_standardization.py ..                                                         [ 90%]
syntheseus/tests/search/node_evaluation/test_common.py .....................................                     [ 97%]
syntheseus/tests/search/test_mol_inventory.py ...                                                                [ 98%]
syntheseus/tests/search/test_visualization.py ........./home/a/miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/coverage/control.py:892: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")

WARNING: Failed to generate report: No data to report.

/home/a/miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/pytest_cov/plugin.py:329: CovReportWarning: Failed to generate report: No data to report.

  self.cov_controller.finish()
                                                                                                                 [100%]

=================================================== warnings summary ===================================================
../miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/torch/torch_version.py:3
  /home/a/miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/torch/torch_version.py:3: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging  # type: ignore[attr-defined]

syntheseus/tests/cli/test_cli.py::test_cli_invalid
syntheseus/tests/cli/test_cli.py::test_cli_invalid
  /home/a/miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/dgl/backend/pytorch/tensor.py:16: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if LooseVersion(th.__version__) < LooseVersion("1.9.0"):

syntheseus/tests/cli/test_cli.py::test_cli_invalid
syntheseus/tests/cli/test_cli.py::test_cli_invalid
  /home/a/miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/dgl/backend/pytorch/tensor.py:340: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if LooseVersion(th.__version__) >= LooseVersion("1.10.0"):

syntheseus/tests/cli/test_cli.py::test_cli_invalid
  /home/a/miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/dgl/dataloading/dataloader.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PYTORCH_VER = LooseVersion(torch.__version__)

syntheseus/tests/cli/test_cli.py::test_cli_invalid
  /home/a/miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/dgl/_dataloading/pytorch/dataloader.py:23: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PYTORCH_VER = LooseVersion(th.__version__)

syntheseus/tests/cli/test_cli.py::test_cli_invalid
  /home/a/miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/dgl/_dataloading/pytorch/dataloader.py:24: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PYTORCH_16 = PYTORCH_VER >= LooseVersion("1.6.0")

syntheseus/tests/cli/test_cli.py::test_cli_invalid
  /home/a/miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/dgl/_dataloading/pytorch/dataloader.py:25: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PYTORCH_17 = PYTORCH_VER >= LooseVersion("1.7.0")

syntheseus/tests/cli/test_cli.py::test_cli_eval_single_step[BackwardModelClass.Chemformer]
  /home/a/miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/lightning_fabric/plugins/environments/xla.py:18: DeprecationWarning: `ModuleAvailableCache` is a special case of `RequirementCache`. Please use `RequirementCache(module=...)` instead.
    from lightning_fabric.accelerators.tpu import _XLA_AVAILABLE, TPUAccelerator

syntheseus/tests/cli/test_cli.py::test_cli_eval_single_step[BackwardModelClass.Chemformer]
  /home/a/miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/lightning_fabric/__init__.py:29: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('lightning_fabric')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    __import__("pkg_resources").declare_namespace(__name__)

syntheseus/tests/cli/test_cli.py::test_cli_eval_single_step[BackwardModelClass.Chemformer]
  /home/a/miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/pytorch_lightning/__init__.py:45: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pytorch_lightning')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    __import__("pkg_resources").declare_namespace(__name__)

syntheseus/tests/cli/test_cli.py::test_cli_eval_single_step[BackwardModelClass.RetroKNN]
  /home/a/miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if LooseVersion(numpy.__version__) >= "1.19":

syntheseus/tests/cli/test_cli.py::test_cli_eval_single_step[BackwardModelClass.RetroKNN]
  /home/a/miniconda3/envs/syntheseus-full/lib/python3.9/site-packages/setuptools/_distutils/version.py:337: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    other = LooseVersion(other)

syntheseus/tests/cli/test_search.py::test_resume_search
  /home/a/syntheseus/syntheseus/search/algorithms/base.py:115: UserWarning: The reaction model does not use caching, which may result in unnecessary duplicate calls with the same input molecule.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

----------- coverage: platform linux, python 3.9.7-final-0 -----------

=============================== 516 passed, 4 skipped, 15 warnings in 197.88s (0:03:17) ================================