Splunk Operator: VolumeSpec CEL validation rule "region is required when provider is aws" errors on missing provider field instead of allowing it (optional field)

[wirehack7]

Please select the type of request

Bug

Tell us more

Splunk Operator version: 3.1.0 (CRDs generated with controller-gen.kubebuilder.io/version: v0.18.0)

Affected CRDs: All CRDs embedding VolumeSpec (e.g. ClusterManager, IndexerCluster, SearchHeadCluster, Standalone) - the same x-kubernetes-validations rule appears on both spec.smartstore.volumes[] and spec.appRepo.volumes[] (and their status mirrors).

Bug

The CEL rule introduced in 3.1.0 for VolumeSpec:

message: region is required when provider is aws
rule: self.provider != 'aws' || size(self.region) > 0

assumes self.provider is always accessible. But provider is documented as an optional string field on VolumeSpec, with no default value. When provider is entirely absent from a volume entry (not just an empty string - genuinely unset, which is a valid, supported configuration per the field's own docs), evaluating self.provider in CEL fails with a runtime error rather than being treated as empty/false. The resulting API error is:

spec.smartstore.volumes[0]: Invalid value: "object": no such key: provider evaluating rule: region is required when provider is aws

Impact

Because Kubernetes validates the entire object (including unrelated, previously-persisted status fields) on every write to the resource, once an object has ever been persisted with a provider-less volume entry (e.g. before upgrading the operator/CRDs to 3.1.0, when this rule didn't exist), every subsequent write to that object fails this rule - including attempts to PATCH the object to fix it, and even attempts to remove finalizers as part of deleting it. The object becomes permanently stuck: it can't be updated, and it can't be deleted, because any write (even a finalizer-only patch) is rejected by this same rule evaluating against the already-invalid, untouched part of the object.

Steps to reproduce

1. On an operator version before 3.1.0 (or by any means that skips this validation), create a ClusterManager (or similar) with a smartstore.volumes entry that has no provider field set.
2. Upgrade the CRDs to 3.1.0.
3. Attempt any further write to that object (a spec change via Helm/kubectl, or kubectl delete followed by removing its finalizers) - every attempt fails with the error above, even one that includes/fixes provider going forward, because the other untouched half of the object (whichever of spec/status a given patch doesn't touch) still lacks it.

Expected behavior

provider being unset should not error at all, matching its documented "optional" status - the rule should only enforce region when provider is explicitly "aws".

Suggested fix

Guard the field access with has(), standard practice for optional CEL fields:

rule: '!has(self.provider) || self.provider != ''aws'' || size(self.region) > 0'

We applied this locally to unblock ourselves, but it means anyone upgrading straight to 3.1.0 with existing objects that predate this rule can get stuck with an object that can neither be updated nor deleted through the Kubernetes API.

[greshetniak-splunk]

Hi @wirehack7 , thanks for reporting this!

[rlieberman-splunk]

Tracked internally as https://splunk.atlassian.net/browse/CSPL-4725.