Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vcpkg support for Bloom #541

Open
wants to merge 35 commits into
base: master
Choose a base branch
from

Conversation

lennonwoo
Copy link

@lennonwoo lennonwoo commented Aug 10, 2019

Abstract

This PR add VcpkgGenerator, RosVcpkgGenerator, and vcpkg's local generator, we can use these generators to expand the templates in vcpkg template folder and get vcpkg's CONTROL, portfiles.cmake; Chocolatey's pkg-name.nuspace, build_dependencies.config. And I validate the generated files in Windows platform.

And notice that this PR use the refactor work in the PR, and you can review code started from the commit message starts with[vcpkg] if you have checked that PR before.

Motivation

Now Bloom could generate Debian & RPM's meta content for ros_buildfarm to compile the ROS package and pack binary files then distribute it. Though there are some works target on other platform, like Arch support: 1, 2, these work haven't made use of the power of ros_buildfarm, and the users in other platforms need to compile ROS themselves, or download the official pre-built ROS archive, it takes lots of time. We all hope that ROS could be used on other platforms easier.

On the other side, the vcpkg is a cross-platform C/C++ package manager, if we could use vcpkgto compile the ROS package, we may utilize it to handle ROS's pre-built process in the platforms vcpkg support. The following part is the details of my lab with my mentors: @cottsay @nuclearsandwich, and with the help of @seanyen @traversaro.

Details

Generator part

The generator is used for analyzing the meta-data in ROS packages' package.xml, and resolve the package's dependencies in its target platform. But the rosdistro work for Windows is still WIP. So we need to do some hack now to expand the template files. What I do including:

  1. modify the distribution file rosdistro to support Win10, as you can see in commits:
  2. modify rosdep package to support Windows in this commit
  3. modify rosdistro python package to change the DEFAULT_DISTRO_URL in this commit
  4. There is some package still haven't ported to Windows in rosdistro, so we need to install them as described in ROS2 Windows build documentation. Then we can hack the Bloom code to ignore the unresolvable package via this commit's in common.py.

For now, we need these hacks to test the vcpkg generators.

Template part

The templates in vcpkg folder include four-part:

  1. CONTROL.em: this is used for vcpkg's dependencies analyze, but we don't add the dependencies in CONTROL.em since our ROS package have extra Python dependencies, so the vcpkg C/C++ package manager couldn't handle it, let Chocolatey do this part job.
  2. portsfile.cmake.em: this is used for vcpkg building process. In general, we download the certain tag archive from ROS package's releasing repository, then use cmake or python setup.py to install the binary files to vcpkg's <CURRENT_PACKAGES_DIR> folder. Then we can pack the files.
  3. template.nuspec.em: this is used to collect the meta-data from ROS package's pacakge.xml. Then NuGet will analyze the dependencies(aka execute depends) in this file, and install the dependencies package for us.
  4. build_dependencies.em: since NuGet cannot specify build depends and execute depends, we need an extra build dependencies files for pre-install dependencies before build package.

For example, you can see the expanded template

Validation

Now we can use vcpkg to build the ROS2 packages for ROS2 ports and export the built binary files to Chocolatey packages. The releasing files include ros_base, rviz Chocolatey packages, and its dependencies. You can download via https://github.com/lennonwoo/vcpkg/releases/tag/v0.11

If you have done the ROS2 build prerequisites. You can download the Chocolatey packages archive and uncompress it to your local device. Then use command
choco install ros-dashing-ros-base ros-dashing-demo-nodes-py ros-dashing-demo-nodes-cpp ros-dashing-demo-nodes-cpp-native -y -s <Your uncompress location>
to install the ros_base meta-package and simple demo. The packages will be installed to C:\opt\ros\dashing, and then you can do a simple test

Build details

I use Bloom to generate 90 meta-packages and its 240 sub-packages' vcpkg port files, as you can see in my forked repostory Then I use a build.py executed in administrator mode to

  • Install the package dependencies via choco install build_dependencies.config
  • Built the package via install.bat <pkg-name>
  • Export the binary files as Chocolatey package.

We also need to change something in vcpkg for building the ROS2 packages successfully, and you can see the detail commits

  1. Remove SHA512 check when download GitHub archive. Since we cannot add SHA512 to ports file when we expand templates in Bloom's vcpkg generation.
  2. Disable postbuildlint since some package in ROS2 can't pass postbuildlint check.
  3. Use cmd_execute instead of cmd_execute_clean, because we need to inherit the environment variable when building packages.
  4. The vcpkg disable using Window register to find cmake packages, we need re-enable it.
  5. The vcpkg build x32-windows default, we need a VCPKG_DEFAULT_TRIPLET environment variable, which value should be "x64-windows"

For Ros2's package, there are somethings we need to patch:

  • The ros_workspace's PYTHONPATH don't fit in Windows, so patch ros_workspace added
  • rcl releasing test folder doesn't contain resources directory, so patch rcl added
  • The patch file formatted with LF line endings in Windows should patch with --bianry, so patches rviz-ogre, uncrustify added
  • The ament package setup.bat still broken, so patch ament_package added

That's all, thank you!

lennonwoo and others added 30 commits July 3, 2019 19:17
Notice that the `get_package_data` method will return a list.
so it's clear that the logic `detect_branches` method in ReleaseGenerator and RosReleaseGenerator is same.
In this commit, I reuse this function and put it into ReleaseGenerator's `handle_arguments`
method so that we can safely delete the duplicate part in RosReleaseGenerator.
We need to put the `missing_dep_resolver` in class as staticmethod
in case successor of Generator want to reuse this function.
And I also add os_name, os_versino, ros_distro if oneday the resolver
need more dependency information.
Firstly put the common place/process template files to common.py.
Then, since every pacakge_system has its prefer way of formatting
description and depends, so pass them as function.
Next, rewrite get_sub_hook in DebianGenerator for debian specific
substitutions attribute, and consider generate_cmd will use the hook,
I make the hook as staticmethod method.
At last, fix up the api change influence.
The major different part of RPM and Debian's generator as follow:
1. rpm don't have native option in local generate_cmd related command
2. rpm and debian's Date, changelogs, and License format are different
3. rpm need NoArch substitution
4. rpm has its description and depends specification
And I also update the utils function like __process_template_folder,
__place_template_folder, and place_tempalte_folder, the newest code in
debian should also fit rpm generator's logic
Also fix typo here(unvalid => invalid)
This commit add the missed rosdistro when call super's get_subs_hook
in rosdebian or rosrpm. The release_history should be passed as **kwargs
format since it have default value already.
And I also combine two line in get_release_tag method since it's not
exceed 120 lines.
Neither Fedora or RHEL 7+ use this value. If another distribution were
to be supported for RPM generation in Bloom that can take advantage of
this value, we should find a better way to add it instead of hard-
coding it into the template.

Signed-off-by: Scott K Logan <[email protected]>
Supported in RPM 4.11 and newer:
https://rpm.org/user_doc/autosetup.html

Signed-off-by: Scott K Logan <[email protected]>
...rather than assuming that the make executable is 'make'.

Signed-off-by: Scott K Logan <[email protected]>
Also move terminating '..' to the end to make it easier to patch cmake
arguments and less likely that the make arguments accidentally get
added to the end of the cmake arguments.

Signed-off-by: Scott K Logan <[email protected]>
There are two classes of issues here:
1. ROS packages often provide libraries which are also provided by the
   operating system. If an operating system declares a dependency on
   that library, we don't want the package manager to install the ROS
   package instead of the system package.
2. Many ROS packages don't install libraries in a way that the
   'provides' portion of dependency generation can detect, so when
   another ROS package takes a dependency on that library, the
   automatic dependency can't be met and the downstream package
   cannot be installed.

More info on RPM dependency generation:
https://rpm.org/user_doc/dependency_generators.html

Signed-off-by: Scott K Logan <[email protected]>
The BRP Python bytecompiler will always use the sytem's default Python
interpreter, which may not be the interpreter we're targeting. Safest
option is to disable the automagic byte compilation altogether.

Note that this doesn't mean that python files won't ever be compiled,
it just means that the catch-all policy at the end of the process won't
attempt to compile anything which hasn't already been compiled.

Signed-off-by: Scott K Logan <[email protected]>
This change shouldn't modify the behavior in Fedora, where all current
releases define 'cmake3' to be the same as 'cmake'. In RHEL 7, where
cmake 2 is the default, we need to use the 'cmake3' macro to use the
supplamental 'cmake3' executable instead of the system default.

All current ROS releases except Indigo have a *minimum* cmake
requirement of 3.

Signed-off-by: Scott K Logan <[email protected]>
…e#540)

The SCM plugin for mock will create an archive of the sources if either
this marker file is present in the root of the project or the
`write_tar` option is specified in mock's configuration.

The way bloom works, we always want to archive the sources, but this
isn't the default configuration of the SCM plugin. Writing this marker
file means one less configuration is necessary when consuming the
release repo to build the RPMs with mock.

Signed-off-by: Scott K Logan <[email protected]>
Since this method won't use self's arrtibutes, change it in case someone
need `tag name` during another staticmethod in the future.
Since some packages may have python, ruby related dependencies, not just
pure C/C++ dependencies, so I leave the Build-Depends to empty, hope we
can handle the dependencies outside vcpkg. And I think we could use Chocolatey
to handle the ROS dependencies installation
- We need set VCPKG_BUILD_TYPE as `release` otherwise vcpkg will build
  `debug` & `release` version together
- We will download Bloom released pacakge from repository via `vcpkg_from_<source_sites>`
- We don't set `CMAKE_INSTALL_PREFIX` to OPTIONS, since vcpkg will set
  it to <CURRENT_PACKAGES_DIR>, and that's enough for us to pack.
- We can't use `CMAKE_PREFIX_PATH` anymore since vcpkg will change it,
  use `AMENT_PREFIX_PATH` instead
- check python did exists, otherwise complain
- follow vcpkg's style, set install path to <CURRENT_PACKAGES_DIR>.
- make the install directory first, otherwise the cmake will complain
- this four templates is same
- we decide the nuspec's metadata during Bloom's generation
- we only handle files in `tools` directory, so the built binary files
  should be compressed and move to tools directory
- we decide the packages' build dependencies in Bloom's generation
- the build_dependencies.config is used for installing packages
  dependencies via `choco install build_dependencies.config`,
  because vcpkg could install all ROS packages' dependencies.
The changes adapter from old generator including:
- copy the format_description from DebianGenerator, this formatted
  description will be put in nuspec metadata's description part
- format depends I modify format_depends function from DebianGenerator,
  change the versions dict adapter NuGet style, as you can see in:
  https://docs.microsoft.com/en-us/nuget/reference/package-versioning#version-ranges-and-wildcards
- notice that `prepare_arguments` & `handle_arguments` call the parent's
  method in end of the function, which follow the Bloom's style,
- the commit style in `generate_package` is similar to RPMGenerator

The special changes in this commit
- we don't handle release history in `generate_package`, since NuGet or
  Vcpkg don't require it
this generate cmd is similar to debian or rpm's generate_cmd
this generator follow the style of old ros generators

But notice that I call VcpkgGenerator.get_subs_hook in the end of
RosVcpkgGenerator's get_subs_hook, this way follow the Bloom style.
If we don't do this way, the appended BuildDepends and Depends won't be
updated to VcpkgGenerator.get_subs_hook, and will cause depend losed problem.
@nuclearsandwich nuclearsandwich self-assigned this Apr 23, 2021
@PhotoTeeborChoka
Copy link

This looks reasonably mature, are there any plans to merge this PR anytime soon?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants