chore: support for Spark 3.5.6

[razvan]

Description

Part of: stackabletech/docker-images#1141

Depends on stackabletech/docker-images#1142

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

• Changes are OpenShift compatible
• CRD changes approved
• CRD documentation for all fields, following the style guide.
• Helm chart can be installed and deployed operator works
• Integration tests passed (for non trivial changes)
• Changes need to be "offline" compatible
• Links to generated (nightly) docs added
• Release note snippet added

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Links to generated (nightly) docs added
• Release note snippet added
• Add type/deprecation label & add to the deprecation schedule
• Add type/experimental label & add to the experimental features tracker

[razvan]

First round of tests:

--- FAIL: kuttl (3380.60s)
    --- FAIL: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-false_spark-3.5.6_ny-tlc-report-0.2.0 (275.85s)
        --- PASS: kuttl/harness/resources_openshift-false_spark-3.5.6 (127.66s)
        --- PASS: kuttl/harness/custom-log-directory_openshift-false_spark-3.5.6_hdfs-latest-3.4.1_zookeeper-latest-3.9.3 (303.71s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-false_spark-3.5.6 (182.47s)
        --- FAIL: kuttl/harness/spark-ny-public-s3_openshift-false_spark-3.5.6_s3-use-tls-true (114.52s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-false_spark-3.5.6 (150.40s)
        --- PASS: kuttl/harness/hbase-connector_openshift-false_spark-3.5.6_hbase-2.6.1_hdfs-latest-3.4.1_zookeeper-latest-3.9.3 (356.24s)
        --- PASS: kuttl/harness/iceberg_openshift-false_spark-3.5.6 (95.63s)
        --- PASS: kuttl/harness/overrides_openshift-false_spark-3.5.6 (174.98s)
        --- FAIL: kuttl/harness/spark-connect_openshift-false_spark-connect-3.5.6_spark-connect-client-3.5.6 (292.64s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-3.5.6_s3-use-tls-true (301.52s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-3.5.6_s3-use-tls-true (209.23s)
        --- PASS: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-3.5.6_delta-3.1.0 (258.17s)
        --- PASS: kuttl/harness/logging_openshift-false_spark-3.5.6_ny-tlc-report-0.2.0 (460.84s)
        --- PASS: kuttl/harness/spark-examples_openshift-false_spark-3.5.6 (75.77s)
FAIL

[razvan]

Tests now pass locally

Here is another CI run https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/spark-k8s-operator-it-custom/32/

[NickLarsenNZ]

LGTM