Don't bundle compiled Python into debs

[dankamongmen]

This is necessary to consider #532 done. We shouldn't be installing compiled pyc files. This is supposed to be done in the postinst, since the package maker doesn't know which version of python is installed on the target machine. Upgrades to python invalidate the cache.

This is currently leading to the package-installs-python-bytecode lintian error firing off everywhere, and is honestly worth fixing.

[tfoote]

We'd be happy to improve the process for installing python files. Links to documentation on how this is recommended to be done in debian packages would be great. Or if you have time a PR against the bloom installation rules templates that would suppress installing the pyc files and trigger the appropriate postinstall hooks would be even better.

[dankamongmen]

I'm looking into how to do this now, and expect to submit a patch. Thanks for reminding me about this!

[dankamongmen]

So currently the errors look like this:

E: ros-melodic-greenzie-msgs: package-installs-python-bytecode opt/ros/melodic/lib/python2.7/dist-packages/greenzie_msgs/msg/_CalibrateDeadzoneGoal.pyc

ideally, the debian/rules file generated by bloom shouldn't be copying (or generating, but not copying is sufficient) these into the resulting deb. we're already using pybuild, which is good. Looking at https://wiki.debian.org/Python/FAQ and https://debian.org/doc/manuals/debmake-doc, I think it might be as simple as adding some -Xs to the dh_auto_install invocation. Testing this out now.

[dankamongmen]

Yeah, that appears to be sufficient. We're still building the bytecode, but we remove it from the packaging area following dh_auto_install.

I just saw #199 , which covers this same issue. Not sure where that ever went, but we've long since converted to pybuild (what was blocking that bug). Anyway, putting a PR up now.

[dankamongmen]

Hrmmm before we move on this, I want to verify that the new packages are generating the pyc files at install time. I.e. they ought have the following in their postinsts, or something similar:

#!/bin/sh
set -e

# Automatically added by dh_python3:
if which py3compile >/dev/null 2>&1; then
	py3compile -p pitivi /usr/lib/x86_64-linux-gnu/pitivi/python/pitivi -V 3.7
fi
if which pypy3compile >/dev/null 2>&1; then
	pypy3compile -p pitivi /usr/lib/x86_64-linux-gnu/pitivi/python/pitivi -V 3.7 || true
fi

# End automatically added section

[dankamongmen]

Similarly, there should be a pyclean invocation or its non-union equivalent in the prerm,

[dankamongmen]

OK, I'm not seeing any evidence of a postinst or prerm in these debs, so I'm gonna need to figure that out. It looks like only ament_python-type packages are using pybuild:

[schwarzgerat](0) $ grep -r pybuild bloom/generators/
bloom/generators/debian/templates/ament_python/rules.em:	dh $@@ -v --buildsystem=pybuild --with python3
[schwarzgerat](0) $ 

similarly, it is the only one with the necessary build-dep on python. It's evident, though, that many packages include python without using these rules. So, a complete solution could be something like:

(1) for ament_python, rely on pybuild inserting the proper pycompile and pyclean calls
(2) for the other three generators (ament_cmake, cmake, catkin), add an explicit postinst and prerm. these ought look for any py files installed by the package, and if any are present, invoke pycompile/pyclean. this will possibly error out if python is not listed as a depend in the package's control file, but in this case we're flushing out a bug in said file, so that seems a good thing (you shouldn't be installing python without a python dep).

Thoughts @tfoote ?

[tfoote]

Those two approaches sound reasonable. For the second, it could require a switch of some sort that the user who's mixing python and other elements enables instead of trying to do it automatically. Though that's more feasible for the newer packaging methods than something like catkin with a much larger install base.

[dankamongmen]

Great, I'll put together an update today. Ideally, this would be accompanied by large-scale changes to existing ROS debian control files. Is there any mechanism in place to support such a thing? I'm guessing not, that most ROS packages are living on their own and would need independent patches? It's no great loss if so.

[tfoote]

A mass deployment would require rerunning bloom on all the repositories. But I don't think we'd want to do that as we'd want maintainers to be able to QA the results not just bulk deploy this. Once we resolve this as new packages get released they'll have the cleaner builds. As this is a warnings cleanup it doesn't warrant a mass maintainer effort to triage. Depending on how testing goes and if we see risks of regressions we may also want to conditionally deploy it for new distros only so as not to break already released packages.

[realtime-neil]

Note that, since version 5.0.44, debhelper supports the use of DH_ALWAYS_EXCLUDE to keep unwanted filenames out of a package. Here's a debian/rules snippet that seems to work:

export DH_ALWAYS_EXCLUDE=.pyc:.pyo

It's still a hack, albeit a smaller hack than those four find invocations. It also doesn't solve the issue of getting the maintainer scripts to automagically invoke py*compile and/or pyclean.