[SPARK-47010][K8S] Support csi driver for volume type

[zgzzbws]

What changes were proposed in this pull request?

This PR adds support for mounting volumes using CSI drivers.

Why are the changes needed?

By adding CSI driver support, Spark running on k8s can now dynamically and on-demand leverage diverse storage services (such as high-performance SSDs, low-cost HDDs, etc.) provided by the underlying infrastructure. This enhances platform independence and simplifies operations and maintenance.

Does this PR introduce any user-facing change?

Users can now using CSI driver by add configs like:

spark-submit \
--conf spark.kubernetes.executor.volumes.csiVolumeClaim.mount.path=/mnt/disk  \
--conf spark.kubernetes.executor.volumes.csiVolumeClaim.csiDriverName=file.csi.azure.com \
--conf spark.kubernetes.executor.volumes.csiVolumeClaim.options.shareName=EXISTING_SHARE_NAME \
--conf spark.kubernetes.executor.volumes.csiVolumeClaim.options.secretName=azure-secret
...

How was this patch tested?

add ut test

Was this patch authored or co-authored using generative AI tooling?

No