K8SPG-1086: Add logVolumeClaimSpec to store logs on a separate PVC

[yoav-katz]

Problem:
PostgreSQL and pgBackRest write their logs onto the data volume (/pgdata/logs/postgres and the pgBackRest log directory). Logs compete with database data for the same PVC, so a burst of logging can fill the data volume and take the cluster down, and log storage can't be sized, backed by a different storage class, or managed independently.

Cause:
The log directories are hardcoded under the data volume and there was no API to place them elsewhere - the operator only provisioned data, WAL and tablespace volumes per instance.

Solution:
Add an optional logVolumeClaimSpec to each instance set (spec.instances[].logVolumeClaimSpec), gated on cluster version >= 3.1.0 (I dont think it will make it to 3.1.0 but worth the try 😄 ):

• The operator provisions a -pglogs PVC (labeled with role pglogs) and mounts it at /pglogs.
• The startup script creates /pglogs/postgres and /pglogs/pgbackrest/log and symlinks the log directories on the data volume to them. Existing real directories are replaced with symlinks during this one-time transition; logs are expendable. An existing symlink is left alone.
• If the log volume is later removed from the spec, the startup script detects the broken symlinks and removes them, so PostgreSQL and pgBackRest recreate real directories on the data volume. The PVC itself is retained for manual cleanup or reuse.

Covered by unit tests for the PVC reconciler (reconcilePostgresLogVolume), the pod spec (mounts/volumes), the startup script, and the version gate on ToCrunchy.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported PG version?
• Does the change support oldest and newest supported Kubernetes version?

[yoav-katz]

questions for reviewers:

1. I havent saw any e2e test related to tablespace / wal pvc - so I didnt add one for the logs pvc too. will be happy to add an e2e test for all the three of them.
2. not sure about gating on cluster version - will be happy for input 😄

[egegunes]

hi @yoav-katz

I havent saw any e2e test related to tablespace / wal pvc - so I didnt add one for the logs pvc too. will be happy to add an e2e test for all the three of them.

yes, we lack such a test. adding it would be great.

not sure about gating on cluster version

i don't think this can make into v3.1.0, we should gate on 3.2.0

[Copilot]

Pull request overview

This PR adds support for configuring a dedicated PersistentVolumeClaim for PostgreSQL + pgBackRest logs per instance set, so log storage can be sized and managed independently of the data PVC (mitigating outages due to log bursts filling the data volume).

Changes:

• Adds logVolumeClaimSpec to instance set APIs/CRDs (Percona API v2 and upstream v1beta1) and updates generated deepcopy + CRD manifests.
• Implements log-PVC reconciliation, naming/labeling, and Pod mounting at /pglogs.
• Extends the startup init script to migrate log directories via symlinks and clean up broken symlinks when the log volume is removed.

Reviewed changes

Copilot reviewed 17 out of 20 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pkg/apis/upstream.pgv2.percona.com/v1beta1/zz_generated.deepcopy.go	Generated deepcopy updated for new LogVolumeClaimSpec.
pkg/apis/upstream.pgv2.percona.com/v1beta1/postgrescluster_types.go	Adds logVolumeClaimSpec to upstream PostgresInstanceSetSpec.
pkg/apis/pgv2.percona.com/v2/zz_generated.deepcopy.go	Generated deepcopy updated for new LogVolumeClaimSpec.
pkg/apis/pgv2.percona.com/v2/perconapgcluster_types.go	Plumbs CRVersion into instance-set conversion and gates logVolumeClaimSpec in ToCrunchy.
pkg/apis/pgv2.percona.com/v2/perconapgcluster_types_test.go	Adds/updates unit tests for ToCrunchy version gating and signature change.
internal/postgres/reconcile.go	Adds /pglogs mount wiring into the instance Pod spec.
internal/postgres/reconcile_test.go	Updates Pod spec/startup-script expectations; adds log-volume mounting test case.
internal/postgres/config.go	Adds startup-script logic to migrate logs to /pglogs and handle broken symlinks.
internal/naming/names.go	Adds deterministic naming for the log PVC (<instance>-pglogs).
internal/naming/labels.go	Adds RolePostgresLog (pglogs) label role.
internal/controller/postgrescluster/postgres.go	Implements reconcilePostgresLogVolume PVC reconciliation.
internal/controller/postgrescluster/postgres_test.go	Adds unit tests for log PVC creation and retention-on-removal behavior.
internal/controller/postgrescluster/instance.go	Threads the reconciled log PVC into postgres.InstancePod.
deploy/cw-bundle.yaml	Updates bundled manifests with new CRD schema field.
deploy/crd.yaml	Updates CRD schema with new logVolumeClaimSpec field.
deploy/cr.yaml	Adds example snippet for logVolumeClaimSpec.
deploy/bundle.yaml	Updates bundled manifests with new CRD schema field.
config/crd/bases/upstream.pgv2.percona.com_postgresclusters.yaml	CRD base updated to include logVolumeClaimSpec.
config/crd/bases/pgv2.percona.com_perconapgclusters.yaml	CRD base updated to include logVolumeClaimSpec.
build/crd/percona/generated/pgv2.percona.com_perconapgclusters.yaml	Generated CRD updated to include logVolumeClaimSpec.

Files not reviewed (2)

• pkg/apis/pgv2.percona.com/v2/zz_generated.deepcopy.go: Generated file
• pkg/apis/upstream.pgv2.percona.com/v1beta1/zz_generated.deepcopy.go: Generated file

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[yoav-katz]

While looking into this I found the existing tablespaces e2e test, and it made me realize I may not have understood the e2e approach here.
Each test under e2e-tests/tests/ seems to cover one feature in isolation: deploy the operator, create a cluster with just that feature enabled, verify it, tear down. That's end-to-end in depth - real operator, real cluster, real backup/restore - but narrow in breadth. What I'd normally expect from an e2e suite is at least one test that turns several features on at once and checks they don't interfere: e.g. a cluster with a WAL volume and a log volume and tablespaces, since they all touch the same startup script and volume mounts.
Is the per-feature split deliberate (e.g. for isolation, faster triage, or CI time), with cross-feature interactions covered somewhere else that I may have missed?
I came across pretty much the same issue with my new DCS implementation: it really needs an e2e test that goes through a more complete cluster lifecycle - create, backup, upgrade, change parameters, add instances, etc.

For now, I'll add two new e2e tests, one for the WAL volume and one for the log volume. But I think it would be worth considering a broader lifecycle/feature-interaction e2e test as well.

[egegunes]

Is the per-feature split deliberate (e.g. for isolation, faster triage, or CI time), with cross-feature interactions covered somewhere else that I may have missed?

We don't have a strict rule for isolating features. There are cases where we test multiple features in tandem because they make sense to be used together.

In this case also I think we can add a single test for WAL and log volumes and maybe even a separate tablespace volume.

[yoav-katz]

Ok, will change the tests from separate wal-volume and log-volume to
additional-volumes that will verify all volumes in a single e2e test.

[JNKPercona]

Test Name	Result	Time
additional-volumes	passed	00:13:27
backup-enable-disable	passed	00:14:34
builtin-extensions	passed	00:06:57
custom-extensions	failure	00:13:43
cert-manager-tls	passed	00:12:47
cert-management-policy	passed	00:04:07
custom-envs	passed	00:23:53
custom-tls	passed	00:07:10
database-init-sql	passed	00:02:52
demand-backup	passed	00:29:04
demand-backup-offline-snapshot	failure	00:29:27
dynamic-configuration	passed	00:03:34
extra-volumes	passed	00:04:12
finalizers	passed	00:04:25
init-deploy	passed	00:03:06
huge-pages	passed	00:03:23
logical-replica	passed	00:11:27
major-upgrade-14-to-15	failure	00:03:54
major-upgrade-15-to-16	failure	00:04:01
major-upgrade-16-to-17	failure	00:04:00
major-upgrade-17-to-18	failure	00:07:46
ldap	passed	00:04:07
ldap-tls	passed	00:05:29
logcollection	passed	00:08:34
monitoring	passed	00:10:12
one-pod	passed	00:06:32
repo-host-autogrow	passed	00:02:13
operator-self-healing	passed	00:11:52
pgbouncer	passed	00:08:04
pg-tde	passed	00:11:35
pg-tde-wal-encrypt	passed	00:16:06
pitr	passed	00:12:05
scaling	passed	00:06:32
scheduled-backup	passed	00:29:16
self-healing	passed	00:10:35
sidecars	passed	00:03:13
standby-pgbackrest	passed	00:21:14
standby-streaming	passed	00:17:08
start-from-backup	passed	00:13:20
start-from-volumes	passed	00:03:47
telemetry-transfer	passed	00:05:00
upgrade-consistency	failure	00:06:36
upgrade-minor	passed	00:08:02
users	failure	00:03:48
migration-from-crunchy-standby	passed	00:19:22
migration-from-crunchy-pv	passed	00:09:18
migration-from-crunchy-backup-restore	passed	00:20:29

Summary	Value
Tests Run	47/47
Job Duration	02:35:22
Total Test Time	08:02:43

commit: 4947a02
image: perconalab/percona-postgresql-operator:PR-1760-4947a0272