No support for rosdep rules that specify explicit python package version

[danielcranston]

Alternative title: Not possible to restrict python packages to a specific version

Hi, thanks for making this awesome tool!

It would be really convenient if we could create rules that target python packages at specific versions.

Background

Imagine you have setup a custom rosdep source like this:

$ cat /etc/ros/rosdep/sources.list.d/10-custom.list
yaml file:///etc/ros/rosdep/custom-pkgs.yaml

$ cat /etc/ros/rosdep/custom-pkgs.yaml 
torch-1.4.0-pip:
  ubuntu:
    pip:
      packages: [torch==1.4.0]

My understanding of rosdep is that the above ==1.4.0 is "illegal", and for a given rule (set of packages, in this case just torch), the idea is to offload the choice of package version to the package manager in question: apt, pip, npm etc.

For e.g. apt, this makes sense since apt packages are usually frozen for each ROS distro, and you can apt-mark hold or edit /etc/apt/preferences.d to restrict packages to certain versions.

For pip otoh, it's not as simple because

• By default the official PyPi repository is used, where new package versions can appear at any time
• AFAIK there is no equivalent of /etc/apt/preferences.d for pip: the only way to pin a package is to specify it explicitly at install time or in some requirements.txt (which would only relate to a ROS package, not globally!)
  • Seems there's a notion of constraints.txt, but I could not get rosdep install to respect them
• requirements.txt is not really tied into the rosdep install workflow

So apart from hosting and maintaining your own pip repository, it seems very tricky to restrict python package versions.

Rosdep rules specifying explicit package versions

One way around this would be to allow rosdep rules/keys (like the custom torch-1.4.0-pip above) to specify explicit versions. I imagine you guys would consider the rule value torch==1.4.0 to be unintended usage and "illegal", but I'm not sure since it does almost work out of the box:

• rosdep resolve seems happy
• rosdep install successfully installs at the specified version, but the post-install check for success (based on pip show) fails.
  • System dependencies have not been satisfied: pip torch==1.4.0 (returns error code 1)
• likewise, rosdep check *package* also fails for the same reason as above

(For clarity, the issue is that rosdep invokes pip show torch==1.4.0, and "torch==1.4.0" is obviously not the name of an existing package)

One way to overcome these issues is shown here: danielcranston@f11ed44. Essentially, we change pip_detect to consider the possibility of receiving a "explicitly versioned package" (including ==), and pass only the package name to pip show.

Note

As the commit message states, this is obviously brittle and "best effort", and not an attempt at full version support. I still think it's a worthwhile addition though, since

1. it's a one-line change that provides a solution to the main problem
2. it should be fully backwards compatible
3. rosdep is already kind of "best effort" wrt versions (see example below)

Example of "best effort" of rosdep (expand me)

Say one of your packages specifies these two rosdep rules:

• python3-numpy, that installs numpy version 1.17.4 via apt
• python3-trimesh-pip, that installs trimesh via pip (thus the latest version by default)

Since a couple of weeks or months back, the latest version of trimesh requires a numpy version larger than 1.17.4, so numpy is upgraded to a higher version.

So in conclusion, though the user most likely wanted/expected numpy 1.17.4, due to the "best effort"ness of rosdep this didn't happen.

Questions

I'd be very grateful if you could answer the following questions:

1. Is it written down anywhere what stance rosdep takes wrt package versions?
2. Am I correct in thinking that requirements.txt is not tied into the rosdep install workflow?
3. Is there an intended way to restrict python packages installed via rosdep to specific versions?
4. Has anyone had any success using constraints.txt together with rosdep?
5. Would you be oppose to a PR adding danielcranston@f11ed44?

Sorry for the long exposition, and thanks for reading!

[130s]

Personally I'd like to have such a capability, but it seems complicated. Related discussions #803 (and probably many more places).

[130s]

More conversation in #325 (not just about Python pkg).