Initial commit for new 1.6 release with Postgres 13 support. (#1257)

* Initial commit for new 1.6 release with Postgres 13 support.
* Updating maintainers, Go version, Codeowners.
* Use lazy upgrade image that contains pg13.
* fix typo for ownerReference
* fix clusterrole in helm chart
* reflect GCP logical backup in validation
* improve PostgresTeam docs
* change defaults for enable_pgversion_env_var and storage_resize_mode
* explain manual part of in-place upgrade
* remove gsoc docs

Co-authored-by: Felix Kunde <[email protected]>

Author: Jan-M

.github/workflows/run_e2e.yaml

@@ -14,7 +14,7 @@ jobs:
     - uses: actions/checkout@v1
     - uses: actions/setup-go@v2
       with:
-          go-version: "^1.15.5"
+          go-version: "^1.15.6"
     - name: Make dependencies
       run: make deps mocks
     - name: Compile

.github/workflows/run_tests.yaml

@@ -14,7 +14,7 @@ jobs:
     - uses: actions/checkout@v1
     - uses: actions/setup-go@v2
       with:
-          go-version: "^1.15.5"
+          go-version: "^1.15.6"
     - name: Make dependencies
       run: make deps mocks
     - name: Compile

CODEOWNERS

@@ -1,2 +1,2 @@
 # global owners
-*        @alexeyklyukin @erthalion @sdudoladov @Jan-M @CyberDem0n @avaczi @FxKu @RafiaSabih
+*        @erthalion @sdudoladov @Jan-M @CyberDem0n @avaczi @FxKu @RafiaSabih

MAINTAINERS

@@ -1,3 +1,5 @@
-Oleksii Kliukin <[email protected]>
 Dmitrii Dolgov <[email protected]>
 Sergey Dudoladov <[email protected]>
+Felix Kunde <[email protected]>
+Jan Mussler <[email protected]>
+Rafia Sabih <[email protected]>

README.md

@@ -14,7 +14,7 @@ pipelines with no access to Kubernetes API directly, promoting infrastructure as
 ### Operator features
 
 * Rolling updates on Postgres cluster changes, incl. quick minor version updates
-* Live volume resize without pod restarts (AWS EBS, others pending)
+* Live volume resize without pod restarts (AWS EBS, PvC)
 * Database connection pooler with PGBouncer
 * Restore and cloning Postgres clusters (incl. major version upgrade)
 * Additionally logical backups to S3 bucket can be configured
@@ -23,10 +23,12 @@ pipelines with no access to Kubernetes API directly, promoting infrastructure as
 * Basic credential and user management on K8s, eases application deployments
 * UI to create and edit Postgres cluster manifests
 * Works well on Amazon AWS, Google Cloud, OpenShift and locally on Kind
+* Support for custom TLS certificates
+* Base support for AWS EBS gp3 migration (iops, throughput pending)
 
 ### PostgreSQL features
 
-* Supports PostgreSQL 12, starting from 9.6+
+* Supports PostgreSQL 13, starting from 9.6+
 * Streaming replication cluster via Patroni
 * Point-In-Time-Recovery with
 [pg_basebackup](https://www.postgresql.org/docs/11/app-pgbasebackup.html) /
@@ -48,7 +50,25 @@ pipelines with no access to Kubernetes API directly, promoting infrastructure as
 [timescaledb](https://github.com/timescale/timescaledb)
 
 The Postgres Operator has been developed at Zalando and is being used in
-production for over two years.
+production for over three years.
+
+## Notes on Postgres 13 support
+
+If you are new to the operator, you can skip this and just start using the Postgres operator as is, Postgres 13 is ready to go.
+
+The Postgres operator supports Postgres 13 with the new Spilo Image that includes also the recent Patroni version to support PG13 settings.
+More work on optimizing restarts and rolling upgrades is pending.
+
+If you are already using the Postgres operator in older version with a Spilo 12 Docker Image you need to be aware of the changes for the backup path.
+We introduce the major version into the backup path to smooth the major version upgrade that is now supported manually.
+
+The new operator configuration, sets a compatilibty flag *enable_spilo_wal_path_compat* to make Spilo look in current path but also old format paths for wal segments.
+This comes at potential perf. costs, and should be disabled after a few days.
+
+The new Spilo 13 image is: `registry.opensource.zalan.do/acid/spilo-13:2.0-p1`
+
+The last Spilo 12 image is: `registry.opensource.zalan.do/acid/spilo-12:1.6-p5`
+
 
 ## Getting started
 

charts/postgres-operator/crds/operatorconfigurations.yaml

@@ -319,6 +319,10 @@ spec:
                 properties:
                   logical_backup_docker_image:
                     type: string
+                  logical_backup_google_application_credentials:
+                    type: string
+                  logical_backup_provider:
+                    type: string
                   logical_backup_s3_access_key_id:
                     type: string
                   logical_backup_s3_bucket:

charts/postgres-operator/templates/clusterrole.yaml

@@ -44,34 +44,51 @@ rules:
   - get
   - patch
   - update
-# to read configuration from ConfigMaps
+# to send events to the CRs
 - apiGroups:
   - ""
   resources:
-  - configmaps
+  - events
   verbs:
+  - create
   - get
-# to send events to the CRs
+  - list
+  - patch
+  - update
+  - watch
+# to manage endpoints/configmaps which are also used by Patroni
+{{- if toString .Values.configGeneral.kubernetes_use_configmaps | eq "true" }}
 - apiGroups:
   - ""
   resources:
-  - events
+  - configmaps
   verbs:
   - create
+  - delete
+  - deletecollection
   - get
   - list
   - patch
   - update
   - watch
-# to manage endpoints/configmaps which are also used by Patroni
 - apiGroups:
   - ""
   resources:
-{{- if toString .Values.configGeneral.kubernetes_use_configmaps | eq "true" }}
-  - configmaps
+  - endpoints
+  verbs:
+  - get
 {{- else }}
+# to read configuration from ConfigMaps
+- apiGroups:
+  - ""
+  resources:
+  - configmaps
+  verbs:
+  - get
+- apiGroups:
+  - ""
+  resources:
   - endpoints
-{{- end }}
   verbs:
   - create
   - delete
@@ -81,6 +98,7 @@ rules:
   - patch
   - update
   - watch
+{{- end }}
 # to CRUD secrets for database access
 - apiGroups:
   - ""

charts/postgres-operator/values-crd.yaml

@@ -22,7 +22,7 @@ configGeneral:
   # update only the statefulsets without immediately doing the rolling update
   enable_lazy_spilo_upgrade: false
   # set the PGVERSION env var instead of providing the version via postgresql.bin_dir in SPILO_CONFIGURATION
-  enable_pgversion_env_var: false
+  enable_pgversion_env_var: true
   # start any new database pod without limitations on shm memory
   enable_shm_volume: true
   # enables backwards compatible path between Spilo 12 and Spilo 13 images

charts/postgres-operator/values.yaml

@@ -25,7 +25,7 @@ configGeneral:
   # update only the statefulsets without immediately doing the rolling update
   enable_lazy_spilo_upgrade: "false"
   # set the PGVERSION env var instead of providing the version via postgresql.bin_dir in SPILO_CONFIGURATION
-  enable_pgversion_env_var: "false"
+  enable_pgversion_env_var: "true"
   # start any new database pod without limitations on shm memory
   enable_shm_volume: "true"
   # enables backwards compatible path between Spilo 12 and Spilo 13 images

delivery.yaml

@@ -16,7 +16,7 @@ pipeline:
         - desc: 'Install go'
           cmd: |
             cd /tmp
-            wget -q https://storage.googleapis.com/golang/go1.15.5.linux-amd64.tar.gz -O go.tar.gz
+            wget -q https://storage.googleapis.com/golang/go1.15.6.linux-amd64.tar.gz -O go.tar.gz
             tar -xf go.tar.gz
             mv go /usr/local
             ln -s /usr/local/go/bin/go /usr/bin/go

docs/administrator.md

@@ -11,15 +11,29 @@ switchover (planned failover) of the master to the Pod with new minor version.
 The switch should usually take less than 5 seconds, still clients have to
 reconnect.
 
-Major version upgrades are supported either via [cloning](user.md#how-to-clone-an-existing-postgresql-cluster)or in-place.
-
-With cloning, the new cluster manifest must have a higher `version` string than the source
-cluster and will be created from a basebackup. Depending of the cluster size,
-downtime in this case can be significant as writes to the database should be
-stopped and all WAL files should be archived first before cloning is started.
-
-Starting with Spilo 13, Postgres Operator can do in-place major version upgrade, which should be faster than cloning. To trigger the upgrade, simply increase the version in the cluster manifest. As the very last step of
-processing the manifest update event, the operator will call the `inplace_upgrade.py` script in Spilo. The upgrade is usually fast, well under one minute for most DBs. Note the changes become irrevertible once `pg_upgrade` is called. To understand the upgrade procedure, refer to the [corresponding PR in Spilo](https://github.com/zalando/spilo/pull/488).
+Major version upgrades are supported either via [cloning](user.md#how-to-clone-an-existing-postgresql-cluster)
+or in-place.
+
+With cloning, the new cluster manifest must have a higher `version` string than
+the source cluster and will be created from a basebackup. Depending of the
+cluster size, downtime in this case can be significant as writes to the database
+should be stopped and all WAL files should be archived first before cloning is
+started.
+
+Starting with Spilo 13, Postgres Operator can do in-place major version upgrade,
+which should be faster than cloning. However, it is not fully automatic yet.
+First, you need to make sure, that setting the PG_VERSION environment variable
+is enabled in the configuration. Since `v1.6.0`, `enable_pgversion_env_var` is
+enabled by default.
+
+To trigger the upgrade, increase the version in the cluster manifest. After
+Pods are rotated `configure_spilo` will notice the version mismatch and start
+the old version again. You can then exec into the Postgres container of the
+master instance and call `python3 /scripts/inplace_upgrade.py N` where `N`
+is the number of members of your cluster (see `number_of_instances`). The
+upgrade is usually fast, well under one minute for most DBs. Note, that changes
+become irrevertible once `pg_upgrade` is called. To understand the upgrade
+procedure, refer to the [corresponding PR in Spilo](https://github.com/zalando/spilo/pull/488).
 
 ## CRD Validation
 

docs/developer.md

@@ -286,7 +286,7 @@ manifest files:
 
 Postgres manifest parameters are defined in the [api package](../pkg/apis/acid.zalan.do/v1/postgresql_type.go).
 The operator behavior has to be implemented at least in [k8sres.go](../pkg/cluster/k8sres.go).
-Validation of CRD parameters is controlled in [crd.go](../pkg/apis/acid.zalan.do/v1/crds.go).
+Validation of CRD parameters is controlled in [crds.go](../pkg/apis/acid.zalan.do/v1/crds.go).
 Please, reflect your changes in tests, for example in:
 * [config_test.go](../pkg/util/config/config_test.go)
 * [k8sres_test.go](../pkg/cluster/k8sres_test.go)

docs/gsoc-2019/ideas.md

@@ -80,7 +80,7 @@ Those are top-level keys, containing both leaf keys and groups.
   The default is `false`.
 
 * **enable_pgversion_env_var**
-  With newer versions of Spilo, it is preferable to use `PGVERSION` pod environment variable instead of the setting `postgresql.bin_dir` in the `SPILO_CONFIGURATION` env variable. When this option is true, the operator sets `PGVERSION` and omits `postgresql.bin_dir` from  `SPILO_CONFIGURATION`. When false, the `postgresql.bin_dir` is set. This setting takes precedence over `PGVERSION`; see PR 222 in Spilo. The default is `false`.
+  With newer versions of Spilo, it is preferable to use `PGVERSION` pod environment variable instead of the setting `postgresql.bin_dir` in the `SPILO_CONFIGURATION` env variable. When this option is true, the operator sets `PGVERSION` and omits `postgresql.bin_dir` from  `SPILO_CONFIGURATION`. When false, the `postgresql.bin_dir` is set. This setting takes precedence over `PGVERSION`; see PR 222 in Spilo. The default is `true`.
 
 * **enable_spilo_wal_path_compat**
   enables backwards compatible path between Spilo 12 and Spilo 13 images. The default is `false`.
@@ -375,7 +375,7 @@ configuration they are grouped under the `kubernetes` key.
 * **storage_resize_mode**
   defines how operator handels the difference between requested volume size and
   actual size. Available options are: ebs - tries to resize EBS volume, pvc -
-  changes PVC definition, off - disables resize of the volumes. Default is "ebs".
+  changes PVC definition, off - disables resize of the volumes. Default is "pvc".
   When using OpenShift please use one of the other available options.
 
 ## Kubernetes resource requests