Dependency install is inconsistent, does not resolve all constraints

[joncppl]

The strategy of installing the dependencies of each package sequentially can lead to not all packages' dependent versions being satisified.

Take for example the simple case where my workspace contains two projects, that each depend on (just for sake of example) numpy. One package explicitly requests version 1.19 and the other version 1.20.

Running python3 -m colcon_poetry_ros.dependencies.install ... will appear to install both versions. However, after the install only version 1.19 is in site-packages.

In this scenario surely there should be a warning about unresolvable version constraints.

A second example; project1 requests numpy 1.19 and project2 requests numpy <=1.20. There is a version of numpy that satisfies both of these constraints (1.19). However... guess which version ends up in site-packages. It's 1.20

[velovix]

You're absolutely right and this touches on a tension between ROS workspaces and Poetry. I think ideally the solution here would be to give each package its own virtual environment. The main challenge here would be to ensure that the virtual environment is entered when a node in that package is started.

[apockill]

I wonder if there's a fusion between the ideas behind pipx and poetry that could be reached- that is, the way pipx lets you run an executable which it then executes with libraries in a python venv.

If we could figure out how to make the node executables execute the same way pipx runs executables, this might be a clean solution.

[joncppl]

I've been thinking about the per-package venv solution before considering poetry. (Our current pip-based solution resolves the constraints across an entire workspace, which has caused some headache when different packages truly did want different versions of dependencies)

The main challenge here would be to ensure that the virtual environment is entered when a node in that package is started.

A solution I thought about is to have the dependency installer modify or wrap the "entry point" scripts, overriding the python executable and/or PYTHONPATH.

It's less clear to me how to deal with library-like packages and cascading dependencies though

[apockill]

@joncppl Are you familiar with pipx by any chance?

A solution I thought about is to have the dependency installer modify or wrap the "entry point" scripts, overriding the python executable and/or PYTHONPATH.

Pipx seems to do nearly exactly that. The idea with pipx is you can do pipx install {some python program} and it will create a venv, and put the executables in your path, which when launched will use the python libraries in their own custom venv.. For example, if you do pipx install tdirstat you can then run tdirstat, and it will pull dependencies only from its own isolated venv.

Here's what the tdirstat executable ends up looking like:

#!/home/alex/.local/pipx/venvs/tdirstat/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from tdirstat import tdirstat
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(tdirstat())

The key I think is the inclusion of the #!/home/alex/.local/pipx/venvs/tdirstat/bin/python.

Perhaps if colcon-poetry-ros created a venv for each ros package, and then prepended a shebang like the above one pointing to that specific packages executable scripts, everything might just work when the launchfile goes to run the various executables.

[joncppl]

I've had some success prototyping this, but I did it outside of the context of colcon.

One of the considerations I wanted to make is to be able to install python dependencies in a sourceless environment (eg. post install, src and build are deleted but install is left). allowing dependency installation to be a post-install operation for distributed packages.

• colcon build as normal, however each poetry package needs to install it's pyproject.toml to share
• use ament_index_python to list packages (This works in my case, but since colcon doesn't necessarily require ros and ament, I'm not sure how you feel about it. Placing an additional marker or using the presence of pyproject.toml in share could substitute the ament_index_python functionality)
• make temporary folders for each poetry project. I just copy pyproject.toml and poetry.lock here
• I use package.xml as the source for dependencies to other poetry-ros packages (could use poetry path dependencies, but ros workspaces are normally invariant the the relative path between packages and this achieves that)
• modify the temporary pyproject.toml to have path references to the local package.xml dependencies. the path referenced is the temporary directory of the references package.
• run poetry install for each temporary directory
• use poetry env info -p to retrieve the venv path
• for each script in the packages' scripts, insert a line which prepends the venv path to sys.path before importing the script function

This is working in a minimal demo workspace with some token dependencies.

The concern I have left is how to expose the venv to a developer for: cli/repl and for IDE completion. A simple command could wrap poetry env info -p or poetry shell, assuming that the temporary directory still exists or is recreated and poetry uses the same venv if it is recreated in the same directory+project_name (which I haven't determined yet, but believe to be the case)