Skip to content

Commit

Permalink
Data Server Updates (#106)
Browse files Browse the repository at this point in the history
  • Loading branch information
thpang authored Aug 18, 2021
1 parent 2f92c4d commit 0c16714
Show file tree
Hide file tree
Showing 12 changed files with 257 additions and 273 deletions.
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ RUN apt-get install -y jq \
&& chmod 755 ./kubectl /viya4-iac-gcp/docker-entrypoint.sh \
&& mv ./kubectl /usr/local/bin/kubectl \
&& chmod g=u -R /etc/passwd /etc/group /viya4-iac-gcp \
&& chdir /viya4-iac-gcp ; terraform init
&& terraform init

ENV TF_VAR_iac_tooling=docker
ENTRYPOINT ["/viya4-iac-gcp/docker-entrypoint.sh"]
Expand Down
73 changes: 55 additions & 18 deletions docs/CONFIG-VARS.md
Original file line number Diff line number Diff line change
Expand Up @@ -227,25 +227,62 @@ stateful = {
| filestore_tier | The service tier for the Google Filestore Instance | string | "BASIC_HDD" | Valid Values: "BASIC_HDD", "BASIC_SSD" (previously called "STANDARD" and "PREMIUM" respectively.) |
| filestore_size_in_gb | Size in GB of Filesystem in the Google Filestore Instance | number | 1024 for BASIC_HDD, 2560 for BASIC_SDD | 2560 GB is the minimum size for the BASIC_SSD tier. The BASIC_HDD tier allows a minimum size of 1024 GB. |

## Postgres Servers

## Postgres
When setting up ***external database servers***, you must provide information about those servers in the `postgres_servers` variable block. Each entry in the variable block represents a ***single database server***.

This code only configures database servers. No databases are created during the infrastructure setup.

The variable has the following format:

```terraform
postgres_servers = {
default = {},
...
}
```

**NOTE**: The `default = {}` elements is always required when creating external databases. This is the systems default database server.

Each server element, like `foo = {}`, can contain none, some, or all of the parameters listed below:

| Name | Description | Type | Default | Notes |
| :--- | ---: | ---: | ---: | ---: |
| create_postgres | Create a PostgreSQL server instance | bool | false | |
| postgres_name | The name of the PostgreSQL Server | string | <computed> | Once used, a name cannot be reused for up to [one week](https://cloud.google.com/sql/docs/mysql/delete-instance) |
| postgres_machine_type| The machine type for the PostgreSQL server VMs" | string | "db-custom-8-30720" | Google Cloud Postgres supports only shared-core machine types such as db-f1-micro, and custom machine types such as db-custom-2-13312.
| postgres_storage_gb | Minimum storage allowed for the PostgreSQL server | number | 10 | |
| postgres_administrator_login | The Administrator Login for the PostgreSQL Server. Changing this forces a new resource to be created. | string | "pgadmin" | | |
| postgres_administrator_password | The Password associated with the postgres_administrator_login for the PostgreSQL Server | string | | |
| postgres_server_version | The version of the PostgreSQL server instance | string | "11" | Supported values are 11 and 12 |
| postgres_ssl_enforcement_enabled | Enforce SSL on connection to the PostgreSQL database | bool | false | |
| postgres_db_charset | Charset for the PostgreSQL Database | string | "UTF8" | Needs to be a valid PostgreSQL Charset. |
| postgres_db_collation | Collation for the PostgreSQL Database | string | "en_US.UTF8" | Needs to be a valid PostgreSQL Collation. |
| postgres_backups_enabled | Enables postgres backups | bool | true | |
| postgres_backups_start_time | Start time for postgres backups | string | "21:00" | |
| postgres_backups_location | TODO | string | null | |
| postgres_backups_point_in_time_recovery_enabled | Enable point-in-time recovery | bool | false | |
| postgres_db_names | The list of names of PostgreSQL database to create | list | [] | |
| postgres_availability_type | The availability type for the master instance. | string | "ZONAL" | This is only used to set up high availability for the PostgreSQL instance. Can be either `ZONAL` or `REGIONAL`."
| postgres_database_flags | Database flags for the master instance. | list of objects | | More details: https://cloud.google.com/sql/docs/postgres/flags |
| machine_type| The machine type for the PostgreSQL server VMs" | string | "db-custom-8-30720" | Google Cloud Postgres supports only shared-core machine types such as db-f1-micro, and custom machine types such as db-custom-2-13312.
| storage_gb | Minimum storage allowed for the PostgreSQL server | number | 10 | |
| backups_enabled | Enables postgres backups | bool | true | |
| backups_start_time | Start time for postgres backups | string | "21:00" | |
| backups_location | TODO | string | null | |
| backups_point_in_time_recovery_enabled | Enable point-in-time recovery | bool | false | |
| backup_count | The number of automated backups to retain, from 1 to 365 | string | "7" | Take note this is a **COUNT** not number of days |
| administrator_login | The Administrator Login for the PostgreSQL Server. Changing this forces a new resource to be created. | string | "pgadmin" | | |
| administrator_password | The Password associated with the administrator_login for the PostgreSQL Server | string | "my$up3rS3cretPassw0rd" | |
| server_version | The version of the PostgreSQL server instance | string | "11" | Supported values are 11 and 12 |
| ssl_enforcement_enabled | Enforce SSL on connection to the PostgreSQL database | bool | true | |
| availability_type | The availability type for the master instance. | string | "ZONAL" | This is only used to set up high availability for the PostgreSQL instance. Can be either `ZONAL` or `REGIONAL`. |
| database_flags | Database flags for the master instance. | list(object({})) | | More details can be found [here](https://cloud.google.com/sql/docs/postgres/flags) |

Here is a sample of the `postgres_servers` variable with the `default` entry only overriding the `administrator_password` parameter and the `cps` entry overriding all of the parameters:

```terraform
postgres_servers = {
default = {
administrator_password = "D0ntL00kTh1sWay"
},
another_server = {
machine_type = "db-custom-8-30720"
storage_gb = 10
backups_enabled = true
backups_start_time = "21:00"
backups_location = null
backups_point_in_time_recovery_enabled = false
backup_count = 7 # Number of backups to retain, not in days
administrator_login = "pgadmin"
administrator_password = "my$up3rS3cretPassw0rd"
server_version = "11"
availability_type = "ZONAL"
ssl_enforcement_enabled = true
database_flags = [{ name = "foo" value = "true"}, { name = "bar", value = "false"}]
}
}
```
10 changes: 6 additions & 4 deletions examples/sample-input-byo.tfvars
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,12 @@ nat_address_name = "<name-of-existing-nat-ip-address>"
# add labels to the created resources
tags = {} # e.g., { "key1" = "value1", "key2" = "value2" }

# Postgres config
create_postgres = true # set this to "false" when using internal Crunchy Postgres
postgres_ssl_enforcement_enabled = false
postgres_administrator_password = "mySup3rS3cretPassw0rd"
# Postgres config - By having this entry a database server is created. If you do not
# need an external database server remove the 'postgres_servers'
# block below.
postgres_servers = {
default = {},
}

# GKE config
default_nodepool_min_nodes = 2
Expand Down
10 changes: 6 additions & 4 deletions examples/sample-input-ha.tfvars
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,12 @@ default_public_access_cidrs = [] # e.g., ["123.45.6.89/32"]
# add labels to the created resources
tags = {} # e.g., { "key1" = "value1", "key2" = "value2" }

# Postgres config
create_postgres = true # set this to "false" when using internal Crunchy Postgres
postgres_ssl_enforcement_enabled = false
postgres_administrator_password = "mySup3rS3cretPassw0rd"
# Postgres config - By having this entry a database server is created. If you do not
# need an external database server remove the 'postgres_servers'
# block below.
postgres_servers = {
default = {},
}

# GKE config
default_nodepool_min_nodes = 2
Expand Down
10 changes: 6 additions & 4 deletions examples/sample-input-minimal.tfvars
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,12 @@ default_public_access_cidrs = [] # e.g., ["123.45.6.89/32"]
# add labels to the created resources
tags = {} # e.g., { "key1" = "value1", "key2" = "value2" }

# Postgres config
create_postgres = false # set this to "false" when using internal Crunchy Postgres
postgres_ssl_enforcement_enabled = false
postgres_administrator_password = "mySup3rS3cretPassw0rd"
# Postgres config - By having this entry a database server is created. If you do not
# need an external database server remove the 'postgres_servers'
# block below.
# postgres_servers = {
# default = {},
# }

# GKE config
default_nodepool_min_nodes = 1
Expand Down
10 changes: 6 additions & 4 deletions examples/sample-input.tfvars
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,12 @@ default_public_access_cidrs = [] # e.g., ["123.45.6.89/32"]
# add labels to the created resources
tags = {} # e.g., { "key1" = "value1", "key2" = "value2" }

# Postgres config
create_postgres = true # set this to "false" when using internal Crunchy Postgres
postgres_ssl_enforcement_enabled = false
postgres_administrator_password = "mySup3rS3cretPassw0rd"
# Postgres config - By having this entry a database server is created. If you do not
# need an external database server remove the 'postgres_servers'
# block below.
postgres_servers = {
default = {},
}

# GKE config
default_nodepool_min_nodes = 2
Expand Down
97 changes: 97 additions & 0 deletions locals.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
locals {

# get the region from "location", or else from the local config
region = var.location != "" ? regex("^[a-z0-9]*-[a-z0-9]*", var.location) : data.google_client_config.current.region

# get the zone from "location", or else from the local config. If none is set, default to the first zone in the region
is_region = var.location != "" ? var.location == regex("^[a-z0-9]*-[a-z0-9]*", var.location) : false
first_zone = length(data.google_compute_zones.available.names) > 0 ? data.google_compute_zones.available.names[0] : ""
# all_zones = length(data.google_compute_zones.available.names) > 0 ? join(",", [for item in data.google_compute_zones.available.names : format("%s", item)]) : ""
zone = ( var.location != "" ? (local.is_region ? local.first_zone : var.location) : (data.google_client_config.current.zone == "" ? local.first_zone : data.google_client_config.current.zone) )
location = var.location != "" ? var.location : local.zone

# CIDR/Network
default_public_access_cidrs = var.default_public_access_cidrs == null ? [] : var.default_public_access_cidrs
vm_public_access_cidrs = var.vm_public_access_cidrs == null ? local.default_public_access_cidrs : var.vm_public_access_cidrs
postgres_public_access_cidrs = var.postgres_public_access_cidrs == null ? local.default_public_access_cidrs : var.postgres_public_access_cidrs

ssh_public_key = file(var.ssh_public_key)

# Kubernetes
kubeconfig_path = var.iac_tooling == "docker" ? "/workspace/${var.prefix}-gke-kubeconfig.conf" : "${var.prefix}-gke-kubeconfig.conf"

taint_effects = {
NoSchedule = "NO_SCHEDULE"
PreferNoSchedule = "PREFER_NO_SCHEDULE"
NoExecute = "NO_EXECUTE"
}

node_pools_and_accelerator_taints = {
for node_pool, settings in var.node_pools: node_pool => {
accelerator_count = settings.accelerator_count
accelerator_type = settings.accelerator_type
local_ssd_count = settings.local_ssd_count
max_nodes = settings.max_nodes
min_nodes = settings.min_nodes
node_labels = settings.node_labels
os_disk_size = settings.os_disk_size
vm_type = settings.vm_type
node_taints = settings.accelerator_count >0 ? concat( settings.node_taints, ["nvidia.com/gpu=present:NoSchedule"]) : settings.node_taints
}
}

node_pools = merge(local.node_pools_and_accelerator_taints, {
default = {
"vm_type" = var.default_nodepool_vm_type
"os_disk_size" = var.default_nodepool_os_disk_size
"min_nodes" = var.default_nodepool_min_nodes
"max_nodes" = var.default_nodepool_max_nodes
"node_taints" = var.default_nodepool_taints
"node_labels" = merge(var.tags, var.default_nodepool_labels,{"kubernetes.azure.com/mode"="system"})
"local_ssd_count" = var.default_nodepool_local_ssd_count
"accelerator_count" = 0
"accelerator_type" = ""
}
})

subnet_names_defaults = {
gke = "${var.prefix}-gke-subnet"
misc = "${var.prefix}-misc-subnet"
gke_pods_range_name = "${var.prefix}-gke-pods"
gke_services_range_name = "${var.prefix}-gke-services"
}

subnet_names = length(var.subnet_names) == 0 ? local.subnet_names_defaults : var.subnet_names

gke_subnet_cidr = length(var.subnet_names) == 0 ? var.gke_subnet_cidr : module.vpc.subnets["gke"].ip_cidr_range
misc_subnet_cidr = length(var.subnet_names) == 0 ? var.misc_subnet_cidr : module.vpc.subnets["misc"].ip_cidr_range

gke_pod_range_index = length(var.subnet_names) == 0 ? index(module.vpc.subnets["gke"].secondary_ip_range.*.range_name, local.subnet_names["gke_pods_range_name"]) : 0
gke_pod_subnet_cidr = length(var.subnet_names) == 0 ? var.gke_pod_subnet_cidr : module.vpc.subnets["gke"].secondary_ip_range[local.gke_pod_range_index].ip_cidr_range

filestore_size_in_gb = (
var.filestore_size_in_gb == null
? ( contains(["BASIC_HDD","STANDARD"], upper(var.filestore_tier)) ? 1024 : 2560 )
: var.filestore_size_in_gb
)

# PostgreSQL
postgres_servers = var.postgres_servers == null ? {} : { for k, v in var.postgres_servers : k => merge( var.postgres_server_defaults, v, )}

postgres_outputs = length(module.postgresql) != 0 ? { for k,v in module.postgresql :
k => {
"server_name" : module.postgresql[k].instance_name,
"fqdn" : module.postgresql[k].private_ip_address,
"admin" : local.postgres_servers[k].administrator_login,
"password" : local.postgres_servers[k].administrator_password,
"server_port" : "5432", # TODO - Create a var when supported
"ssl_enforcement_enabled" : local.postgres_servers[k].ssl_enforcement_enabled,
"connection_name" : module.postgresql[k].instance_connection_name,
"server_public_ip" : length(local.postgres_public_access_cidrs) > 0 ? module.postgresql[k].public_ip_address : null,
"server_cert" : module.postgresql[k].instance_server_ca_cert.0.cert,
"service_account" : module.sql_proxy_sa.0.service_account.email,
"internal" : false,
}
} : {}

}
Loading

0 comments on commit 0c16714

Please sign in to comment.