Allow gathering of dependencies for not released packages

[KenYN]

We have our project containing lots of packages each with their own package.xml specifying dependencies on local and global packages, that is we will see things like:

<depend>roscpp</depend>
<depend>foobar_interface</depend>
<depend>std_msgs</depend>
<depend>my_private_msgs</depend>
<depend>roslib</depend>

Now, we want to take a snapshot of the current ROS Core files that our project requires, such as roscpp, std_msgs and roslib and their dependencies in this example, so we can build a minimal ROS for our code to run on top of. However, looking at the verbose output, I can see that the unreleased_repo_names variable contains all our local packages and excludes them from the package search early on.

What we would like would be if these names were searched, but then excluded from the final output.

[dirk-thomas]

Please provide more information what you are trying to achieve and what you have been doing / trying until now.

[KenYN]

OK, try again:

First, the use case is creating as small as possible a Docker image for our application.

If I run:

 rosinstall_generator --rosdistro kinetic --from-path my_project/src --deps --wet-only \
         --exclude opencv3 cv_bridge rosbag rosout socketcan_bridge \
                   socketcan_interface geneus genlisp gennodejs \
         --tar

Despite having about 60 or 70 packages with many system dependencies underneath my_project/src I get just this partial list:

- tar:
    local-name: catkin
    uri: https://github.com/ros-gbp/catkin-release/archive/release/kinetic/catkin/0.7.14-0.tar.gz
    version: catkin-release-release-kinetic-catkin-0.7.14-0
- tar:
    local-name: gencpp
    uri: https://github.com/ros-gbp/gencpp-release/archive/release/kinetic/gencpp/0.6.0-0.tar.gz
    version: gencpp-release-release-kinetic-gencpp-0.6.0-0
- tar:
    local-name: genmsg
    uri: https://github.com/ros-gbp/genmsg-release/archive/release/kinetic/genmsg/0.5.11-0.tar.gz
    version: genmsg-release-release-kinetic-genmsg-0.5.11-0
- tar:
    local-name: genpy
    uri: https://github.com/ros-gbp/genpy-release/archive/release/kinetic/genpy/0.6.7-0.tar.gz
    version: genpy-release-release-kinetic-genpy-0.6.7-0
- tar:
    local-name: message_generation
    uri: https://github.com/ros-gbp/message_generation-release/archive/release/kinetic/message_generation/0.4.0-0.tar.gz
    version: message_generation-release-release-kinetic-message_generation-0.4.0-0
- tar:
    local-name: message_runtime
    uri: https://github.com/ros-gbp/message_runtime-release/archive/release/kinetic/message_runtime/0.4.12-0.tar.gz
    version: message_runtime-release-release-kinetic-message_runtime-0.4.12-0
- tar:
    local-name: ros_canopen/can_msgs
    uri: https://github.com/ros-industrial-release/ros_canopen-release/archive/release/kinetic/can_msgs/0.7.9-0.tar.gz
    version: ros_canopen-release-release-kinetic-can_msgs-0.7.9-0
- tar:
    local-name: roscpp_core/cpp_common
    uri: https://github.com/ros-gbp/roscpp_core-release/archive/release/kinetic/cpp_common/0.6.11-0.tar.gz
    version: roscpp_core-release-release-kinetic-cpp_common-0.6.11-0
- tar:
    local-name: roscpp_core/roscpp_serialization
    uri: https://github.com/ros-gbp/roscpp_core-release/archive/release/kinetic/roscpp_serialization/0.6.11-0.tar.gz
    version: roscpp_core-release-release-kinetic-roscpp_serialization-0.6.11-0
- tar:
    local-name: roscpp_core/roscpp_traits
    uri: https://github.com/ros-gbp/roscpp_core-release/archive/release/kinetic/roscpp_traits/0.6.11-0.tar.gz
    version: roscpp_core-release-release-kinetic-roscpp_traits-0.6.11-0
- tar:
    local-name: roscpp_core/rostime
    uri: https://github.com/ros-gbp/roscpp_core-release/archive/release/kinetic/rostime/0.6.11-0.tar.gz
    version: roscpp_core-release-release-kinetic-rostime-0.6.11-0
- tar:
    local-name: std_msgs
    uri: https://github.com/ros-gbp/std_msgs-release/archive/release/kinetic/std_msgs/0.5.11-0.tar.gz
    version: std_msgs-release-release-kinetic-std_msgs-0.5.11-0

My own packages do not have git URIs so I of course don't expect them to be listed, but they have lots of dependencies I'd like to see. The packages above are just those picked up from local customised copies of cv_bridge, rosbag, etc as they already have git URIs, so get processed.

To workaround, what I have done is add a little bit of Python preprocessing:

    rp = rospkg.RosPack(['my_project/src'])

    # Get all first-level dependencies as system dependencies are not handled well
    first_level_deps = set()
    for package in rp.list():
        first_level_deps |= set(rp.get_depends(package, implicit=False))

I then pass first_level_deps to rosinstall_generator as the list of pkgfile, and I get the desired result of all the system packages that my project uses. Doing this gives me round about 100 system packages that allows me to perform a minimal build.

[dirk-thomas]

Assuming your own custom packages are not released they will need to be checked out in the workspace. If any of your packages depends on other unreleased packages you will need to add them to your workspace manually.

rosinstall_generator will provide you with all recursive dependencies (released ROS packages). That list of dependencies is going to be complete (beside you choosing to exclude some packages explicitly). [I assume you are building these from source too - avoiding to use the Debian packages - to allow skipping some of the packages like geneus.]

After fetching the ROS packages you still need to invoke rosdep to install non-ROS dependencies from Debian packages.

[dirk-thomas]

Assuming the previous comments answered your question I am going to close the ticket. Please feel free to continue commenting.

[KenYN]

Your comment does answer the question "What does rosinstall_generator do?", but my question was "Can it also scan the package.xml in unreleased packages for released packages?I would like to see my Python snippet or similar as an option forrosinstall_generator`, so let's keep this closed (or I'll repost on ROS Answers to sound out others?), but instead I'll maybe submit a pull request and we can discuss from there?

[dirk-thomas]

Can it also scan the package.xml in unreleased packages for released packages?

Yes, it can - using the command line option --from-path you can point to a directory tree with arbitrary packages. And if you only care about its dependencies you can pass --deps --deps-only.

[goekce]

Yes, it can - using the command line option --from-path you can point to a directory tree with arbitrary packages. And if you only care about its dependencies you can pass --deps --deps-only.

Does not seem to work :( See #25 (comment)