experimental support for Spark 4

[razvan]

Description

Part of stackabletech/issues#622

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

Beta Give feedback

1. Changes are OpenShift compatible
2. CRD changes approved
3. CRD documentation for all fields, following the style guide.
4. Helm chart can be installed and deployed operator works
5. Integration tests passed (for non trivial changes)
6. Changes need to be "offline" compatible

Reviewer

Beta Give feedback

1. Code contains useful comments
2. Code contains useful logging statements
3. (Integration-)Test cases added
4. Documentation added or updated. Follows the style guide.
5. Changelog updated
6. Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Beta Give feedback

1. Feature Tracker has been updated
2. Proper release label has been added
3. Roadmap has been updated

[razvan]

CI: https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/spark-k8s-operator-it-custom/2/

Only one test failed:

        --- FAIL: kuttl/harness/ldap_druid-28.0.1_zookeeper-latest-3.9.2_opa-0.66.0_hadoop-latest-3.4.0_ldap-use-tls-false_ldap-no-bind-credentials-false_openshift-false (660.35s)