HA Tuning (#30)

Previously the number of times psql was allowed to fail was hard-coded at 3. We found a need to adjust this this number so it is now a parameter defaulting to 3.

Author: patjones

README.md

@@ -145,6 +145,13 @@ The following parameters affect high availability:
     the faster your cluster will failover, but the higher a risk
     of accidental failover and split-brain.  Defaults to `5`.
 
+  - `postgres.replication.psql_error_count` - How many failed PSQL
+    commands allowed before considering it a failure. The health
+    checks are PSQL commands executed every second. Poor network
+    conditions may result in a "Connection dropped" PSQL error.
+    The lower this value, the higher potential for accidental
+    failover and split-brain. Defaults to `3`.
+
   - `vip.readonly_port` - Which port to access the read-only node
     of the cluster. Defaults to `7542`.
 

ci/release_notes.md

@@ -0,0 +1,5 @@
+# Improvements
+
+- Added `postgres.replication.psql_error_count` configuration value
+  which allows an operator to define how many failed health checks
+  occur before failover.

jobs/postgres/spec

@@ -99,13 +99,15 @@ properties:
   postgres.replication.master:
     description: IP address of the preferred master node (should be the 0th postgres node's IP)
     default: ~
-
   postgres.replication.grace:
     description: Grace period (in seconds) to look for an existing PostgreSQL master node on boot.
     default: 15
   postgres.replication.connect_timeout:
-    description: How long (in seconds) to wait before timimg out a failover health check from one node to the other.
+    description: How long (in seconds) to wait before timing out a failover health check from one node to the other.
     default: 5
+  postgres.replication.psql_error_count:
+    description: How many failed attempts to check the other node's status before assuming failure.
+    default: 3
 
   postgres.users:
     description: "A list of {username: ..., password: ...} objects for defining PostgreSQL users.  Setting the 'admin:' key on a user will make them a superuser."
@@ -122,4 +124,4 @@ properties:
               - porcupine
               - hedgehog
             extensions: # optional array of extensions to enable on this database
-              - citext
+              - citext

jobs/postgres/templates/bin/functions

@@ -13,8 +13,9 @@ is_master() {
 
   # psql can experience transient issues (like connection reset)
   # make is_master more resilient against these kinds of errors
+  error_tolerance=<%= p('postgres.replication.psql_error_count') %>
   error_count=0
-  while (( $error_count < 3 )) ; do
+  while (( $error_count < $error_tolerance )) ; do
     tf=$(echo $(psql $opts postgres -t -c 'SELECT pg_is_in_recovery()' 2>&1));
 
     if [[ "$tf" == "f" ]]; then
@@ -25,12 +26,12 @@ is_master() {
       echo "[monitor] received unexpected response from postgres DB while checking master/replica status:"
       echo "[monitor] $tf"
       ((error_count++))
-      echo "[monitor] will attempt to check master/replica status again (check $error_count of 3)"
+      echo "[monitor] will attempt to check master/replica status again (check $error_count of $error_tolerance)"
       sleep 1
       continue
     fi
   done
-  # we errored out 3 times, return that other node is not master
+  # we errored out <%= p('postgres.replication.psql_error_count') %> times, return that other node is not master
   echo "[monitor] couldn't determine who was master or replica due to postgres errors. assuming i'm master."
   return 1 
 }