Support for untrusted CAs in datanode remote reindex migration

[todvora]

Description

This PR is adding support for untrusted certificate authorities during the remote reindex migration to the datanode. The connection check step is now reporting that none of our trust managers is trusting the remote host. Then, user can ✔️ the trust unknown certificates checkbox. Connection check is then using a trust manager that accepts all certificates and reports unknown. These unknown certificates are then transported, together with the allowlist value, to the datanode. Datanode takes these certificates and adds them to its truststore. The truststore is regenerated during each startup, so these certificates will disappear with the next process restart and won't stay there forever.

Motivation and Context

Fixes #19759

How Has This Been Tested?

Manually, added unit tests.

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactoring (non-breaking change)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.

[gally47]

Frontend LGTM. Thanks for working on this.

[moesterheld]

Tested, works and lgtm.
However, I still see a little flaw which needs more info for the user or she will trip over it (like i did)

If you have the CA used by OS in the truststore used by Graylog, provision the data nodes and restart Graylog without disabling the truststore, the migration will think it is legit and will not add it to the data node's truststore.
For the time being, I would add a comment to the restart screen of the migration to disable the truststore.

[moesterheld]

without having tested it, I am not sure if OS will be able to access e.g. a s3 bucket for data tiering if overriding the default trust store

[todvora]

Good point, let me see if we can merge the default truststore with our and provide a complete set of certificates to the opensearch.

[todvora]

Tested, works and lgtm. However, I still see a little flaw which needs more info for the user or she will trip over it (like i did)

If you have the CA used by OS in the truststore used by Graylog, provision the data nodes and restart Graylog without disabling the truststore, the migration will think it is legit and will not add it to the data node's truststore. For the time being, I would add a comment to the restart screen of the migration to disable the truststore.

To mitigate that, we could always propagate the certificate of the remote opensearch. Automatically if graylog trusts it, with confirmation if the graylog doesn't trust it.

Other, more advanced approach would be to let datanodes run the connection check and indices discovery, ideally each node once. This would provide even safer verification that the communication datanode<->remote_opensearch is working fine.

[moesterheld]

Tested, works and lgtm. However, I still see a little flaw which needs more info for the user or she will trip over it (like i did)
If you have the CA used by OS in the truststore used by Graylog, provision the data nodes and restart Graylog without disabling the truststore, the migration will think it is legit and will not add it to the data node's truststore. For the time being, I would add a comment to the restart screen of the migration to disable the truststore.

To mitigate that, we could always propagate the certificate of the remote opensearch. Automatically if graylog trusts it, with confirmation if the graylog doesn't trust it.

Other, more advanced approach would be to let datanodes run the connection check and indices discovery, ideally each node once. This would provide even safer verification that the communication datanode<->remote_opensearch is working fine.

I would implement the automatic propagation. Whilst the advanced approach sounds good, it raises more questions on error handling for individual failing nodes and so on.

[moesterheld]

lgtm now. thank you for adding this feature.
tested with new self-signed CA for data node and existing, uploaded CA