Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Circle uses java-library-oss template #596

Closed
wants to merge 4 commits into from
Closed

Circle uses java-library-oss template #596

wants to merge 4 commits into from

Conversation

schlosna
Copy link
Contributor

@schlosna schlosna commented Jan 6, 2022

Before this PR

Circle CI config was unmanaged, blocking many excavators using JDK 11+ gradle plugins

After this PR

==COMMIT_MSG==
Circle uses java-library-oss template
==COMMIT_MSG==

Possible downsides?

@policy-bot policy-bot bot requested a review from jkozlowski January 6, 2022 18:02
@carterkozak
Copy link
Contributor

I don't think this image has /usr/lib/libcrypto.so. we could fall back to the result of something like strings /etc/ld.so.cache | grep '/libcrypto.so$', using whichever version is available from the container, but I don't know the history behind the /usr/lib/libcrypto.so path.

@schlosna schlosna force-pushed the ds/circle branch 4 times, most recently from c7064d9 to 9225634 Compare January 6, 2022 20:01
sudo ln -s /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 /usr/lib/ssl/libcrypto.so
sudo ln -s /usr/lib/x86_64-linux-gnu/libssl.so.1.1 /usr/lib/ssl/libssl.so
sudo ln -s /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 /usr/lib/libcrypto.so
sudo ln -s /usr/lib/x86_64-linux-gnu/libssl.so.1.1 /usr/lib/libssl.so
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this move to the unit-test: section?

@schlosna
Copy link
Contributor Author

schlosna commented Jan 6, 2022

I'm mainly interested in unblocking the excavators and ensuring things can build on recent JDK 11+ and gradle tooling while I had some cycles. Looks like it may be easier to hold off on this until #588 merges

I don't think this image has /usr/lib/libcrypto.so. we could fall back to the result of something like strings /etc/ld.so.cache | grep '/libcrypto.so$', using whichever version is available from the container, but I don't know the history behind the /usr/lib/libcrypto.so path.

Yeah, these are currently using @ellisjoe 's ellisjoe/ibm-java-openssl:java-8 container images which already have openssl installed per https://github.com/ellisjoe/openjdk-java-openssl/blob/develop/Dockerfile , but the CircleCI openjdk convenience images do not currently have openssl installed.

@carterkozak
Copy link
Contributor

Still have investigation in progress before #588 can be merged, but it currently results in a great deal of noise until something like #592 is released.

@schlosna schlosna force-pushed the ds/circle branch 2 times, most recently from 8abf624 to 88bcdfb Compare January 6, 2022 20:28
Copy link
Contributor Author

@schlosna schlosna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@carterkozak builds are green now, but would you prefer I hold off on this until after #592 & #588 merge?

Comment on lines +90 to +97
- run:
name: link_openssl
command: |
sudo apt-get update -q && sudo apt-get install -qy libssl-dev
sudo ln -s /lib/x86_64-linux-gnu/libcrypt.so /usr/lib/libcrypto.so
sudo ln -s /lib/x86_64-linux-gnu/libcrypt.so /usr/lib64/libcrypto.so
echo "Linked openssl /usr/lib/libcrypto.so"
ls -al /lib/libcrypto* /usr/lib/libcrypto* /usr/lib64/libcrypto* /usr/lib/x86_64-linux-gnu/libcrypto*
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the section we'll need to figure out how we want to fold into circle template so that it doesn't immediately get removed by excavator though unit tests should block that from merging, but that also means the excavator won't be updating everything else including CI images.

For future reference, see https://issues.apache.org/jira/browse/HADOOP-12845 & https://issues.apache.org/jira/browse/HADOOP-11216

Resulting libcrypto symlinks (we need /usr/lib64/libcrypto.so)

lrwxrwxrwx 1 root root      33 Jan  6 20:36 /lib/libcrypto.so -> /lib/x86_64-linux-gnu/libcrypt.so
lrwxrwxrwx 1 root root      33 Jan  6 20:36 /usr/lib64/libcrypto.so -> /lib/x86_64-linux-gnu/libcrypt.so
lrwxrwxrwx 1 root root      33 Jan  6 20:36 /usr/lib/libcrypto.so -> /lib/x86_64-linux-gnu/libcrypt.so
-rw-r--r-- 1 root root 5541876 Nov 24 13:20 /usr/lib/x86_64-linux-gnu/libcrypto.a
lrwxrwxrwx 1 root root      16 Nov 24 13:20 /usr/lib/x86_64-linux-gnu/libcrypto.so -> libcrypto.so.1.1
-rw-r--r-- 1 root root 2954080 Nov 24 13:20 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

Comment on lines +153 to +154
markdown:
docker: [{ image: 'raviqqe/liche:0.1.1' }]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we'd opted out of this check by default in our templates due to the issues with links to other github projects?

@carterkozak
Copy link
Contributor

@schlosna Ordering doesn't make any difference to me, doesn't impact the other PRs that are up and gives us more modern testing.

Was there a reason we tested on ibm jdks? Is that something we need to retain?

- checkout
- restore_cache: { key: 'gradle-wrapper-{{ checksum "gradle/wrapper/gradle-wrapper.properties" }}' }
- restore_cache: { key: 'gradle-cache-{{ checksum "versions.props" }}-{{ checksum "build.gradle" }}' }
- run: ./gradlew --no-daemon --parallel --continue crypto-core:test
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@carterkozak Some consumers of hadoop-crypto may use J9 for JVM runtime so this was running the tests using IBM's J9 JVM via https://github.com/ellisjoe/ibm-java-openssl/blob/develop/Dockerfile

Defer to you & @ellisjoe if you'd want to keep these J9 CI tests

@schlosna
Copy link
Contributor Author

schlosna commented Jan 6, 2022

Before we merge this, we'll need to adjust the Github expected checks:

  • Add Required

    • ci/circleci: check
    • ci/circleci: trial-publish
  • Remove Required

    • ci/circleci: build
    • ci/circleci: build-ibm
    • ci/circleci: publish

@schlosna
Copy link
Contributor Author

schlosna commented Oct 3, 2022

closing as this was updated via #619 . will rerun excavator

@schlosna schlosna closed this Oct 3, 2022
@schlosna schlosna deleted the ds/circle branch October 3, 2022 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants