Skip to content

OCPBUGS-100298: pin internal-lb-monitor pollers to worker nodes #31466

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mkowalski wants to merge 1 commit into
base: main
Choose a base branch
from
+15 −0
Open

OCPBUGS-100298: pin internal-lb-monitor pollers to worker nodes #31466

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions pkg/monitortests/kubeapiserver/disruptioninclusterapiserver/manifests/dep-internal-lb.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,21 @@ spec:
securityContext:
privileged: true
runAsUser: 0
# Pin the internal-lb pollers to worker nodes. On cloud platforms with
# passthrough load balancers (notably GCP), a control-plane node cannot
# reach the api-int VIP through the LB fabric while its local
# kube-apiserver is shutting down: apiserver-watcher removes the local
# VIP redirect ~15s into the graceful shutdown, but the cloud LB health
# check only deprograms the backend ~25-30s in, and in between the
# node's own new connections to the VIP are blackholed (hairpin). A
# poller scheduled on a master therefore reports ~10s of internal-lb
# new-connections disruption on every kube-apiserver rollout of that
# node, which is a measurement artifact rather than user-perceived
# unavailability (OCPBUGS-100298, see also OCPBUGS-83523). The master
# toleration below is kept so the pods remain schedulable on compact
# clusters, where control-plane nodes also carry the worker role.
nodeSelector:
node-role.kubernetes.io/worker: ""
Comment on lines +70 to +71

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -eu
target='pkg/monitortests/kubeapiserver/disruptioninclusterapiserver/manifests/dep-internal-lb.yaml'
printf '%s\n' '--- target files ---'
git ls-files 'pkg/monitortests/kubeapiserver/disruptioninclusterapiserver/manifests/*'
printf '%s\n' '--- target outline/size ---'
wc -l "$target"
printf '%s\n' '--- target manifest ---'
cat -n "$target"
printf '%s\n' '--- related anti-affinity, strategy, topology, and selector references ---'
rg -n -C 4 'dep-internal-lb|podAntiAffinity|topologyKey|maxUnavailable|maxSurge|node-role\.kubernetes\.io/worker|replicas:' \
  pkg/monitortests/kubeapiserver/disruptioninclusterapiserver \
  pkg/monitortests/kubeapiserver 2>/dev/null | head -n 500

Repository: openshift/origin

Length of output: 34076

🏁 Script executed:

#!/bin/bash
set -eu
target='pkg/monitortests/kubeapiserver/disruptioninclusterapiserver/manifests/dep-internal-lb.yaml'
printf '%s\n' '--- target files ---'
git ls-files 'pkg/monitortests/kubeapiserver/disruptioninclusterapiserver/manifests/*'
printf '%s\n' '--- target manifest ---'
cat -n "$target"
printf '%s\n' '--- related scheduling references ---'
rg -n -C 4 'dep-internal-lb|podAntiAffinity|topologyKey|maxUnavailable|maxSurge|node-role\.kubernetes\.io/worker|replicas:' \
  pkg/monitortests/kubeapiserver/disruptioninclusterapiserver \
  pkg/monitortests/kubeapiserver 2>/dev/null | head -n 500

Repository: openshift/origin

Length of output: 33917

🏁 Script executed:

#!/bin/bash
set -eu
target='pkg/monitortests/kubeapiserver/disruptioninclusterapiserver/manifests/dep-internal-lb.yaml'

python3 - "$target" <<'PY'
import re
import sys
from decimal import Decimal, ROUND_CEILING, ROUND_FLOOR

path = sys.argv[1]
text = open(path, encoding="utf-8").read()

def scalar(pattern, name):
    match = re.search(pattern, text, re.MULTILINE)
    if not match:
        raise SystemExit(f"missing {name}")
    return match.group(1)

replicas = int(scalar(r"^  replicas:\s+(\d+)\s*$", "replicas"))
strategy = re.search(r"^  strategy:\s*$", text, re.MULTILINE)
anti_affinity = bool(re.search(
    r"requiredDuringSchedulingIgnoredDuringExecution:.*?"
    r"topologyKey:\s*[\"']kubernetes\.io/hostname[\"']",
    text, re.DOTALL))
worker_selector = bool(re.search(
    r"^      nodeSelector:\s*\n"
    r"^        node-role\.kubernetes\.io/worker:\s*[\"']?[\"']?\s*$",
    text, re.MULTILINE))

def percent(value, replicas, rounding):
    number = Decimal(value.rstrip("%")) * replicas / Decimal(100)
    return int(number.to_integral_value(rounding=rounding))

# Deployment defaults: maxSurge=25% rounds up; maxUnavailable=25% rounds down.
max_surge = percent("25%", replicas, ROUND_CEILING)
max_unavailable = percent("25%", replicas, ROUND_FLOOR)

print({
    "replicas": replicas,
    "explicit_strategy": bool(strategy),
    "required_hostname_anti_affinity": anti_affinity,
    "worker_selector": worker_selector,
    "default_maxSurge": max_surge,
    "default_maxUnavailable": max_unavailable,
    "single_eligible_node_replacement_pending": (
        replicas == 1 and anti_affinity and worker_selector
        and max_surge >= 1 and max_unavailable == 0
    ),
})

assert replicas == 1
assert not strategy
assert anti_affinity
assert worker_selector
assert max_surge == 1
assert max_unavailable == 0
assert replicas == 1 and anti_affinity and worker_selector
PY

printf '%s\n' '--- target file change relative to parent ---'
git diff --stat HEAD^ -- "$target" 2>/dev/null || true
git diff --unified=3 HEAD^ -- "$target" 2>/dev/null | head -n 160 || true

printf '%s\n' '--- explicit rollout strategies in related manifests ---'
rg -n -C 3 '^[[:space:]]+strategy:|maxUnavailable:|maxSurge:' \
  pkg/monitortests/kubeapiserver/disruptioninclusterapiserver/manifests

Repository: openshift/origin

Length of output: 3849

🏁 Script executed:

#!/bin/bash
set -eu
target='pkg/monitortests/kubeapiserver/disruptioninclusterapiserver/manifests/dep-internal-lb.yaml'

python3 - "$target" <<'PY'
import re
import sys
from decimal import Decimal, ROUND_CEILING, ROUND_FLOOR

path = sys.argv[1]
text = open(path, encoding="utf-8").read()

def scalar(pattern, name):
    match = re.search(pattern, text, re.MULTILINE)
    if not match:
        raise SystemExit(f"missing {name}")
    return match.group(1)

replicas = int(scalar(r"^  replicas:\s+(\d+)\s*$", "replicas"))
strategy = re.search(r"^  strategy:\s*$", text, re.MULTILINE)
anti_affinity = bool(re.search(
    r"requiredDuringSchedulingIgnoredDuringExecution:.*?"
    r"topologyKey:\s*[\"']kubernetes\.io/hostname[\"']",
    text, re.DOTALL))
worker_selector = bool(re.search(
    r"^      nodeSelector:\s*\n"
    r"^        node-role\.kubernetes\.io/worker:\s*[\"']?[\"']?\s*$",
    text, re.MULTILINE))

def percent(value, replicas, rounding):
    number = Decimal(value.rstrip("%")) * replicas / Decimal(100)
    return int(number.to_integral_value(rounding=rounding))

max_surge = percent("25%", replicas, ROUND_CEILING)
max_unavailable = percent("25%", replicas, ROUND_FLOOR)

print({
    "replicas": replicas,
    "explicit_strategy": bool(strategy),
    "required_hostname_anti_affinity": anti_affinity,
    "worker_selector": worker_selector,
    "default_maxSurge": max_surge,
    "default_maxUnavailable": max_unavailable,
    "single_eligible_node_replacement_pending": (
        replicas == 1 and anti_affinity and worker_selector
        and max_surge >= 1 and max_unavailable == 0
    ),
})

assert replicas == 1
assert not strategy
assert anti_affinity
assert worker_selector
assert max_surge == 1
assert max_unavailable == 0
PY

printf '%s\n' '--- target file change relative to parent ---'
git diff --stat HEAD^ -- "$target" 2>/dev/null || true
git diff --unified=3 HEAD^ -- "$target" 2>/dev/null | head -n 160 || true

printf '%s\n' '--- explicit rollout strategies in related manifests ---'
rg -n -C 3 '^[[:space:]]+strategy:|maxUnavailable:|maxSurge:' \
  pkg/monitortests/kubeapiserver/disruptioninclusterapiserver/manifests

Repository: openshift/origin

Length of output: 3849

Prevent a single-node rollout deadlock.

If only one worker node is eligible, required hostname anti-affinity blocks the replacement Pod. With one replica, the default RollingUpdate strategy sets maxUnavailable to 0, so the old Pod remains and the rollout stays Pending.

Set maxUnavailable: 1 and maxSurge: 0, use Recreate, or make anti-affinity preferred.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@pkg/monitortests/kubeapiserver/disruptioninclusterapiserver/manifests/dep-internal-lb.yaml`
around lines 70 - 71, Update the deployment strategy associated with the
nodeSelector in the internal load balancer manifest to prevent a single-worker
rollout deadlock: use Recreate, or configure RollingUpdate with maxUnavailable
set to 1 and maxSurge set to 0. Preserve the existing hostname anti-affinity
behavior.

Sources: Coding guidelines, MCP tools

hostNetwork: true
serviceAccountName: disruption-monitor-sa
volumes:
Expand Down