test_security tests failing with Connext on OS X

[ivanpauno]

Those tests have been failing for a long time:

• https://ci.ros2.org/view/nightly/job/nightly_osx_release/1599/#showFailuresLink (last nightly, failing in lore)
• https://ci.ros2.org/view/nightly/job/nightly_osx_release/1598/#showFailuresLink (failing in mini3)
• https://ci.ros2.org/view/nightly/job/nightly_osx_release/1509/#showFailuresLink (failing in lore)
• https://ci.ros2.org/view/nightly/job/nightly_osx_release/1508/#showFailuresLink (passing in mini3)
• https://ci.ros2.org/view/nightly/job/nightly_osx_release/1506/#showFailuresLink (failing in lore)

We should double check if connext security plugins and openssl are installed correctly in these machines.

[hidmic]

@cottsay do you think you'll be able to take a quick look at lore and mini3?

[mikaelarguedas]

We should double check if connext security plugins and openssl are installed correctly in these machines.

@ivanpauno looks like the issue is that the mac jobs used openssl 1.0.x by default untilo mojave and now use openssl 1.1.1 (which in itself is good 👍). The mac machines should provide the path to RTI's OpenSSL via the RTI_OPENSSL_BIN and RTI_OPENSSL_LIBS environment variables (like it's done on Linux and Windows).
It will also require this change ros2/system_tests#409

(upgrading OpenSSL to 1.1.1f would be good too to have the latest available for foxy)

[cottsay]

machine	openssl
mini1**	1.1.1d
mini2	1.0.2t
mini3	1.0.2s
lore	1.0.2s

** out of circulation for maintenance.

Eloquent and Foxy both call for 1.0.2r and all 4 of the machines all meet that requirement. I'm hesitant to update to a version that exceeds our documented minimum requirements in an effort to un-break something. If the minimum requirement needs to change, we should have that discussion before we update our test machines.

Has there been a larger discussion about what to do about REP 2000 and the EOL openssl requirements?

[cottsay]

Ah, I think I found the problem. The OPENSSL_ROOT_DIR environment variable isn't set:

machine	OPENSSL_ROOT_DIR
mini1**	/usr/local/opt/openssl
mini2	/usr/local/opt/openssl
mini3	NOT SET
lore	NOT SET

** out of circulation for maintenance.

Here are the docs: https://index.ros.org/doc/ros2/Installation/Eloquent/macOS-Development-Setup/#install-prerequisites

[mikaelarguedas]

I believe we are conflating different things here.

All the machines do already have openssl 1.1.1* installed, this is actually the reason the security tests fail on all machines.

They also all seem able to find openssl (1.1.1*) for compiling, if searching for "Found OpenSSL:" in the logs:
Found OpenSSL: /usr/local/opt/[email protected]/lib/libcrypto.dylib (found version "1.1.1d")
So OPENSSL_ROOT_DIR doesnt seem to be a requirement or an issue here.

What is causing CI to fail is the fact that it is running the RTI Connext security tests using openssl1.1.1 but the version of Connext used by ROS2 is old and supports only OpenSSL 1.0.2.

The way used to solve this on other platforms is to define the RTI_OPENSSL_BIN and RTI_OPENSSL_LIBS environment variables to point to the version of OpenSSL provided by Connext.

[cottsay]

Okay, I think I understand. So were we using the system's openssl before?

I tried a job with RTI_OPENSSL_{BIN,LIBS} set to direct Connext to the still-installed openssl 1.0.2 from homebrew, but I got the same failures: https://ci.ros2.org/job/test_ci_osx/301

[mikaelarguedas]

So were we using the system's openssl before?

Yes. On previous versions of MacOS il looks like homebrew pointed to openssl1.0.x by default. But now points to openssl 1.1.

I tried a job with RTI_OPENSSL_{BIN,LIBS} set to direct Connext to the still-installed openssl 1.0.2 from homebrew, but I got the same failures: https://ci.ros2.org/job/test_ci_osx/301

DId you run it with the PR linked above ros2/system_tests#409 ?

[cottsay]

Did you run it with the PR linked above ros2/system_tests#409?

I did not, but I made a new one that did, and I'm seeing the same results:

[mikaelarguedas]

Looks like you set RTI_OPENSSL_LIB instead of RTI_OPENSSL_LIBS

[cottsay]

Looks like you set RTI_OPENSSL_LIB instead of RTI_OPENSSL_LIBS

🤦 I scheduled a new one. Good catch...

[cottsay]

That worked! All of the tests passed:

Let's move the discussion over to ros2/system_tests#409 then, since that will clearly close this issue.

[mikaelarguedas]

🎉

[kyrofa]

Let's move the discussion over to ros2/system_tests#409 then, since that will clearly close this issue.

Just want to poke this issue: that PR was merged, but I believe it didn't solve all the macOS issues for Connext, correct?

[mikaelarguedas]

It ended up being merged without solving any issue but doing all the code changes needed for the fix to be reduced to "CI machine setup fixes": ros2/system_tests#409 (comment)

The green CIs here were running on machines with RTI's openssl installed and the RTI_OPENSSL* environment variables set to point to it. This is currently not done consistently on the runners so the tests still fail on CI.

ros2/ci#436 is trying to integrate that environment setup in the CI job configuration but has been stalling recently

[clalancette]

We aren't using this repository anymore for buildfarm issues, so I'm going to archive it. Thus I'm closing out this issue. If you continue to have problems, please report another bug against https://github.com/ros2/ros2. Thank you.