[SPARK-58748][CORE][K8S] Preserve advertised driver host for wildcard bind addresses

[sunchao]

What changes were proposed in this pull request?

Teach Spark to distinguish the address a Kubernetes driver listens on from the address executors should connect to when direct driver-Pod-IP mode is enabled.

Normally, using the driver's bind address as its advertised address works: a Kubernetes-managed driver binds directly to its Pod IP, and executors can connect to that IP without going through a driver Service. That assumption breaks when a driver intentionally binds to every network interface. An address such as 0.0.0.0 or :: means "listen everywhere," but it does not identify a reachable peer.

This change makes the choice depend on the address itself. A concrete bind address continues to use the existing direct-Pod-IP path. A wildcard bind address instead preserves the separately configured, routable spark.driver.host. Both driver initialization and executor endpoint construction apply the same rule, so the address advertised by the driver and the address given to executors remain consistent. Wildcard recognition supports IPv4 and IPv6 without performing DNS lookups, and existing IPv6 normalization is preserved.

Why are the changes needed?

SPARK-58748

Spark deliberately separates two network settings:

# Where the driver opens its listening sockets.
spark.driver.bindAddress=0.0.0.0

# The reachable address advertised to executors.
spark.driver.host=10.0.0.42

This is a legitimate and common configuration for Kubernetes client-mode applications and Spark Connect servers: the process listens on all local interfaces, while executors are told to connect to the driver's actual Pod IP or a routable Service hostname.

The problem appears when direct driver-Pod-IP mode is enabled:

spark.master=k8s://https://kubernetes.example:6443
spark.driver.bindAddress=0.0.0.0
spark.driver.host=10.0.0.42
spark.kubernetes.executor.useDriverPodIP=true

Spark currently assumes the driver's bind address is always its Pod IP. Consequently, SparkContext replaces the configured advertised host with 0.0.0.0, and Kubernetes executor creation independently uses that same wildcard to construct the scheduler endpoint:

Before
  Driver listens on:       0.0.0.0:7078
  Advertised driver host:  0.0.0.0
  Executor driver URL:     spark://CoarseGrainedScheduler@0.0.0.0:7078
  Result:                  executors cannot connect or register

After
  Driver listens on:       0.0.0.0:7078
  Advertised driver host:  10.0.0.42
  Executor driver URL:     spark://CoarseGrainedScheduler@10.0.0.42:7078
  Result:                  executors connect normally

The driver starts successfully because binding to 0.0.0.0 is valid. The failure only becomes visible when executors try to register, leaving the application running but unable to execute tasks. Fixing only one side is insufficient: the driver and executor each make their own address-selection decision, and both must preserve the advertised host.

The same distinction applies to IPv6. For example, a driver may listen on :: while advertising a concrete address such as [2001:db8::42]; executors must receive the concrete address, never the IPv6 wildcard. Compressed, bracketed, and expanded IPv6 wildcard forms are handled consistently.

Spark 4.1 and 4.2 are affected when spark.kubernetes.executor.useDriverPodIP is explicitly enabled. Spark 4.3 and current master enable it by default, so previously valid wildcard-bind configurations can fail without any application-level configuration change.

Does this PR introduce any user-facing change?

Yes. Kubernetes applications that bind their driver to all interfaces now retain the reachable address configured in spark.driver.host, allowing their executors to register and run. This restores the existing documented distinction between the driver's listening address and its advertised address.

Applications that bind directly to a concrete Pod IP continue using direct-IP routing, including existing IPv6 normalization. Applications with direct driver-Pod-IP mode disabled, and applications outside Kubernetes, retain their existing behavior. No configuration defaults or public APIs change.

How was this patch tested?

build/sbt -Pkubernetes \
  'core/testOnly org.apache.spark.SparkContextSuite -- -z "SPARK-58748"' \
  'core/testOnly org.apache.spark.util.UtilsSuite -- -z "SPARK-58748"' \
  'kubernetes/testOnly org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStepSuite' \
  'core/scalastyle' \
  'core/Test/scalastyle' \
  'kubernetes/scalastyle' \
  'kubernetes/Test/scalastyle'

All 41 tests passed: one targeted SparkContextSuite regression, one targeted UtilsSuite regression, and all 39 BasicExecutorFeatureStepSuite tests. Together, they cover default-enabled driver initialization, explicitly enabled and disabled executor configuration, IPv4 and IPv6 wildcard addresses, concrete IPv4 and IPv6 addresses, executor driver URLs, and existing IPv6 normalization. All four Scala style checks also passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex

[sunchao]

cc @sarutak @uros-b @dongjoon-hyun

[sarutak]

nit: When the wildcard fallback is taken, could we add a log line so users know why DRIVER_BIND_ADDRESS wasn't used as the advertised address? e.g.:

logInfo(log"spark.kubernetes.executor.useDriverPodIP is true but bind address " +
  log"${MDC(LogKeys.BIND_ADDRESS, _conf.get(DRIVER_BIND_ADDRESS))} is a wildcard; " +
  log"preserving advertised driver host ${MDC(LogKeys.HOST, _conf.get(DRIVER_HOST_ADDRESS))}")

This would help users debug cases where they set useDriverPodIP=true but the bind address is intentionally 0.0.0.0 or ::.