High Availability Support in Active/Passive Mode (#22)

This BOSH release now support Highly-Available PostgreSQL, in
an active/passive setup, with automatic failover behind a single
VRRP-managed virtual IP address.

On bootstrap, if there is no data directory, the postgres job will
revert to a normal, index-based setup.  The first node will assume
the role of the master, and the second will become a replica.

Once the data directory has been populated, future restarts of the
postgres job will attempt to contact the other node to see if it
is a master.  If the other node responds, and reports itself as a
master, the local node will attempt a `pg_basebackup` from the
master and assume the role of a replica.

If the other node doesn't respond, or reports itself as a replica,
the local node will keep trying, for up to
`postgres.replication.grace` seconds, at which point it will
assume the mantle of leadership and become the master node,
using its current data directory as the canonical truth.

Each node then starts up a `monitor` process; this process is
responsible for ultimately promoting a local replica to be a
master, in the event that the real master goes offline.  It works
like this:

  1. Busy-loop (via 1-second sleeps) until the local postgres
     instance is available on its configured port.  This prevents
     monitor from trying to restart the postgres while it is
     running a replica `pg_basebackup`.

  2. Busy-loop (again via 1-second sleeps) for as long as the
     local postgres is a master.

  3. Busy-loop (again via 1-second sleeps), checking the master
     status of the other postgres node, until it detects that
     either the master node has gone away (via a connection
     timeout), or the master node has somehow become a replica.

  4. Promote the local postgres node to a master.

This has been tested by hand in vSphere with CF's ccdb, uaa, and
diegodb using postgres as it's storage medium. We noticed a `x < 5`
second outage during master failover.

Author: proplex

README.md

@@ -50,27 +50,89 @@ templates/make_manifest openstack-nova my-networking.yml
 bosh -n deploy
 ```
 
-### Development
-
-As a developer of this release, create new releases and upload them:
-
-```
-bosh create release --force && bosh -n upload release
-```
-
-### Final releases
-
-To share final releases:
-
-```
-bosh create release --final
-```
-
-By default the version number will be bumped to the next major number. You can specify alternate versions:
-
-
-```
-bosh create release --final --version 2.1
-```
-
-After the first release you need to contact [Dmitriy Kalinin](mailto://[email protected]) to request your project is added to https://bosh.io/releases (as mentioned in README above).
+## High Availability
+
+HA is implemented with automatic failover, if you set
+`postgres.replication.enabled` to true.
+
+On bootstrap, if there is no data directory, the postgres job will
+revert to a normal, index-based setup.  The first node will assume
+the role of the master, and the second will become a replica.
+
+Once the data directory has been populated, future restarts of the
+postgres job will attempt to contact the other node to see if it
+is a master.  If the other node responds, and reports itself as a
+master, the local node will attempt a `pg_basebackup` from the
+master and assume the role of a replica.
+
+If the other node doesn't respond, or reports itself as a replica,
+the local node will keep trying, for up to
+`postgres.replication.grace` seconds, at which point it will
+assume the mantle of leadership and become the master node,
+using its current data directory as the canonical truth.
+
+Each node then starts up a `monitor` process; this process is
+responsible for ultimately promoting a local replica to be a
+master, in the event that the real master goes offline.  It works
+like this:
+
+  1. Busy-loop (via 1-second sleeps) until the local postgres
+     instance is available on its configured port.  This prevents
+     monitor from trying to restart the postgres while it is
+     running a replica `pg_basebackup`.
+
+  2. Busy-loop (again via 1-second sleeps) for as long as the
+     local postgres is a master.
+
+  3. Busy-loop (again via 1-second sleeps), checking the master
+     status of the other postgres node, until it detects that
+     either the master node has gone away (via a connection
+     timeout), or the master node has somehow become a replica.
+
+  4. Promote the local postgres node to a master.
+
+Intelligent routing can be done by colocating the `haproxy` and
+`keepalived` jobs on the instance groups with `postgres`.  HAproxy
+is configured with an external check that will only treat the
+master postgres node as healthy.  This ensures that either load
+balancer node will only ever route to the write master.
+
+The `keepalived` node trades a VRRP VIP between the `haproxy`
+instances.  This ensures that the cluster can be accessed over a
+single, fixed IP address.  Each keepalived process watches its own
+haproxy process; if it notices haproxy is down, it will terminate,
+to allow the VIP to transgress to the other node, who is assumed
+to be healthy.
+
+It is possible to "instance-up" a single postgres node deploy to a
+HA cluster by adding the `vip` job and changing postgres `instances`
+to 2. More information about this can be found in `manifests/ha.yml`
+
+For backup purposes, a route is exposed through haproxy which
+routes directly to the read-only replica for backup jobs. By default
+it is port `7432`, but is also configurable via `vip.readonly_port`
+
+Here's a diagram:
+
+![High Availability Diagram](docs/ha.png)
+
+The following parameters affect high availability:
+
+  - `postgres.replication.enabled` - Enables replication, which is
+    necessary for HA.  Defaults to `false`.
+
+  - `postgres.replication.grace` - How many seconds to wait for
+    the other node to report itself as a master, during boot.
+    Defaults to `15`.
+
+  - `postgres.replication.connect_timeout` - How many seconds to
+    allow a `psql` health check to attempt to connect to the other
+    node before considering it a failure.  The lower this value,
+    the faster your cluster will failover, but the higher a risk
+    of accidental failover and split-brain.  Defaults to `5`.
+
+  - `vip.readonly_port` - Which port to access the read-only node
+    of the cluster. Defaults to `7542`.
+  
+  - `vip.vip` - Which IP to use as a VIP that is traded between the
+    two nodes. 

ci/release_notes.md

@@ -0,0 +1,7 @@
+# New Features
+* Added option to deploy Postgres as a two-node HA cluster with HAProxy and a
+  VRRP VIP. It features auto-failover and auto-recovery. Uses streaming 
+  replication via WAL.
+
+More information about HA postgres can be found in the `README`, and an example
+manifest has been provided under `manifests/ha.yml`

config/blobs.yml

@@ -1,13 +1,24 @@
----
+haproxy/haproxy-1.8.10.tar.gz:
+  size: 2058928
+  object_id: 5fce0e5c-c863-4bdd-6859-485f7ffcfc7f
+  sha: f2fd8671c8c40aa85f9d13e9b6c0aebc22a71d33
+haproxy/pcre2-10.31.tar.gz:
+  size: 2130574
+  object_id: 81491cd7-e484-460a-7f08-9dc953b79614
+  sha: 7a77476a908c16cb26ad8a26363b67f00c8303bd
+haproxy/socat-1.7.3.1.tar.gz:
+  size: 606049
+  object_id: abe408af-41c4-49a2-7209-62d64c8efd3f
+  sha: a6f1d8ab3e85f565dbe172f33a9be6708dd52ffb
+keepalived/keepalived-1.2.24.tar.gz:
+  size: 601873
+  object_id: 09205505-e300-42b0-6513-24d4c4388960
+  sha: a69b2c40627a9bd69698a57e81a8c8c97826025d
 pgrt/pgrt:
+  size: 7391896
   object_id: b7968265-d373-404d-b6e0-12caa15f9f98
   sha: 2f6c7cc5a7b89712be68f2663fdf3fa9f997a2fa
-  size: 7391896
 postgres/postgresql-9.5.1.tar.bz2:
+  size: 18441638
   object_id: c3acc49c-a9ec-49a1-ae82-e97a00672695
   sha: 905bc31bc4d500e9498340407740a61835a2022e
-  size: 18441638
-pgpool2/pgpool-II-3.5.4.tar.gz:
-  object_id: c0821167-44de-470e-ad15-48309d6dd44a
-  sha: 4ea15dc8bb740baf720b18f182b400ea60b1ae45
-  size: 2237911