diff --git a/docs/user/PostPostgreSQLDataTransfer_Experimental.md b/docs/user/PostPostgreSQLDataTransfer_Experimental.md deleted file mode 100644 index 5a7441ce..00000000 --- a/docs/user/PostPostgreSQLDataTransfer_Experimental.md +++ /dev/null @@ -1,33 +0,0 @@ -## Post PostgreSQL Data Transfer steps for viya4-deployment (Experimental) - -### Using IAC to create an external PostgreSQL database server -If you prefer to use the IAC project used to create your cluster to create an external PostgreSQL database server, refer to IAC project link below corresponding to your cloud environment for PostgreSQL database server configuration examples. - -**Note**: Before using IAC to create a new external PostgreSQL database server, we recommend that you follow the steps to [Stop a SAS Viya Platform Deployment](https://documentation.sas.com/?cdcId=sasadmincdc&cdcVersion=v_044&docsetId=calchkadm&docsetTarget=p17xfmmjjkma1dn1b5dcx3e5ejxq.htm#p0butgo7gtfyi0n14umtfv0voydt). Once the external PostgreSQL database server has been created by IAC, follow the steps to [Start a SAS Viya Platform Deployment](https://documentation.sas.com/?cdcId=sasadmincdc&cdcVersion=v_044&docsetId=calchkadm&docsetTarget=p17xfmmjjkma1dn1b5dcx3e5ejxq.htm#p0butgo7gtfyi0n14umtfv0voydt). - -[Azure PostgreSQL Server](https://github.com/sassoftware/viya4-iac-azure/blob/main/docs/CONFIG-VARS.md#postgres-servers) - -[AWS PostgreSQL Server](https://github.com/sassoftware/viya4-iac-aws/blob/main/docs/CONFIG-VARS.md#postgresql-server) - -[GCP PostgreSQL Server](https://github.com/sassoftware/viya4-iac-gcp/blob/main/docs/CONFIG-VARS.md#postgres-servers) - -### Situation: - -You had an existing SAS Viya platform deployment that was created using the viya4-deployment project and configured to use Crunchy as the database, and you have now completed the manual steps outlined in the [PostgreSQL Data Transfer Guide](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=pgdatamig&docsetTarget=titlepage.htm) to move your data over to an external PostgreSQL cluster. You now want to use the viya4-deployment project to manage your installation again. - -### Recommended Steps: - -1. After following the [PostgreSQL Data Transfer Guide](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=pgdatamig&docsetTarget=titlepage.htm) under "Steps for Crunchy Data 5" you manually added entries for the external PostgreSQL clusters into the kustomization.yaml and produced a file that includes details for **both** the internal Crunchy and external Postgres instance. You then built and applied the manifest so that the Viya deployment is reconfigured to point to the external Postgres instance. -2. The final step in the [PostgreSQL Data Transfer Guide](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=pgdatamig&docsetTarget=titlepage.htm) under "Steps for Crunchy Data 5" asks you to manually remove entries to related to Crunchy and rebuild your manifests again. At this point in time you could instead remove the manual modifications you made to the kustomization.yaml and make changes to your `ansible-vars.yaml` so that the viya4-deployment project can manage your installation again. - 1. First remove any files you manually copied over to site-config to configure the external PostgreSQL clusters. Like `sas-bases/examples/postgres/postgres-user.env` and `site-config/postgres/dataserver-transformer.yaml`, viya4-deployment will be generating this for you. - * If as part of your data transfer, you provisioned your PostgreSQL clusters using any [Viya 4 IAC projects](https://github.com/search?q=org%3Asassoftware+viya4-iac-&type=repositories), then your PostgreSQL configuration and connection information should already be present in the .tfstate file, so you don't need to add those entries in your `ansible-vars.yaml`. You should modify the `V4_CFG_POSTGRES_SERVERS` variable if it's still configured to use an internal Crunchy instance like so: - ```yaml - # modify as below to use external instance - V4_CFG_POSTGRES_SERVERS: - default: - internal: false - ``` - * If as part of your data transfer, you provisioned your PostgreSQL clusters without the use of a Viya 4 IAC projects, then you will need to manually add definitions for each of your PostgreSQL clusters. You can see an example definition here in our [CONFIG-VARS.md documentation](https://github.com/sassoftware/viya4-deployment/blob/main/docs/CONFIG-VARS.md#postgresql) - 2. If your PostgreSQL Cluster requires a certificate for connection, in your `ansible-vars.yaml` ensure that `V4_CFG_TLS_TRUSTED_CA_CERTS` is set and points to either the certificate or a directory containing that certificate. - 3. In your `ansible-vars.yaml` set `DEPLOY` to false - 4. Run the ansible-playbook again with the `viya,install` tags. Since `DEPLOY` is set to false, the SAS Viya platform deployment will not be modified, however in your deployment directory you should see an updated kustomization.yaml with generated entries for your PostgreSQL clusters and those files should automatically be present in your site-config directory. \ No newline at end of file diff --git a/docs/user/PostgreSQL.md b/docs/user/PostgreSQL.md new file mode 100644 index 00000000..969578f0 --- /dev/null +++ b/docs/user/PostgreSQL.md @@ -0,0 +1,57 @@ +# PostgreSQL + +* [PostgreSQL](#postgresql) + * [Use IAC To Create an External PostgreSQL Database Cluster](#use-iac-to-create-an-external-postgresql-database-cluster) + * [Post Data Transfer Steps for viya4-deployment (Experimental)](#post-data-transfer-steps-for-viya4-deployment-experimental) + * [Crunchy Data 5](#crunchy-data-5) + * [Crunchy Data 4](#crunchy-data-4) + +## Use IAC To Create an External PostgreSQL Database Cluster + +To use the IAC project to create an external PostgreSQL database cluster, refer to the IAC project link below that corresponds to your cloud environment. Each link goes to provider-specific PostgreSQL database cluster configuration examples. + +**Note**: Before using IAC to create a new external PostgreSQL database cluster, SAS recommends that you follow the steps to [Stop a SAS Viya Platform Deployment](https://documentation.sas.com/?cdcId=sasadmincdc&cdcVersion=v_044&docsetId=calchkadm&docsetTarget=p17xfmmjjkma1dn1b5dcx3e5ejxq.htm#p0butgo7gtfyi0n14umtfv0voydt). After the external PostgreSQL database cluster has been created by IAC, follow the steps to [Start a SAS Viya Platform Deployment](https://documentation.sas.com/?cdcId=sasadmincdc&cdcVersion=v_044&docsetId=calchkadm&docsetTarget=p17xfmmjjkma1dn1b5dcx3e5ejxq.htm#p0butgo7gtfyi0n14umtfv0voydt). + +[Azure PostgreSQL Cluster](https://github.com/sassoftware/viya4-iac-azure/blob/main/docs/CONFIG-VARS.md#postgres-servers) + +[AWS PostgreSQL Cluster](https://github.com/sassoftware/viya4-iac-aws/blob/main/docs/CONFIG-VARS.md#postgresql-server) + +[GCP PostgreSQL Cluster](https://github.com/sassoftware/viya4-iac-gcp/blob/main/docs/CONFIG-VARS.md#postgres-servers) + +## Post Data Transfer Steps for viya4-deployment (Experimental) + +After you complete the steps outlined in the [PostgreSQL Data Transfer Guide](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=pgdatamig&docsetTarget=titlepage.htm) to move your data from an internal PostgreSQL server to an external PostgreSQL cluster, you can use the viya4-deployment project to manage your installation again. + +### Crunchy Data 5 + +The final step in the [PostgreSQL Data Transfer Guide](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=pgdatamig&docsetTarget=titlepage.htm) under "Post-transfer Steps for Crunchy Data 5" tells you to remove entries related to Crunchy Data and rebuild your manifests. Instead, you can remove the manual modifications you made to the `kustomization.yaml` and revise your `ansible-vars.yaml` file so that the viya4-deployment project can manage your installation again. + +1. Remove any files you manually copied to site-config to configure the external PostgreSQL clusters, such as `sas-bases/examples/postgres/postgres-user.env` and `site-config/postgres/dataserver-transformer.yaml`. The viya4-deployment project will be generating the files for you. + * If you provisioned your PostgreSQL clusters using any [Viya 4 IAC projects](https://github.com/search?q=org%3Asassoftware+viya4-iac-&type=repositories), then your PostgreSQL configuration and connection information should already be present in the .tfstate file. Therefore, you do not need to add those entries in your `ansible-vars.yaml` file. However, you should modify the `V4_CFG_POSTGRES_SERVERS` variable if it's still configured to use an internal Crunchy instance. Here is an example: + ```yaml + # modify as below to use external instance + V4_CFG_POSTGRES_SERVERS: + default: + internal: false + ``` + * If you provisioned your PostgreSQL clusters without the use of a Viya 4 IAC project, then you must manually add definitions for each of your PostgreSQL clusters. You can see an example definition at [CONFIG-VARS.md documentation](https://github.com/sassoftware/viya4-deployment/blob/main/docs/CONFIG-VARS.md#postgresql). +2. If your PostgreSQL cluster requires a certificate for connection, ensure that `V4_CFG_TLS_TRUSTED_CA_CERTS` is set in your `ansible-vars.yaml` file and that it points to either the certificate or a directory containing the certificate. +3. In your `ansible-vars.yaml` file, set `DEPLOY` to false. +4. Run the ansible-playbook again with the `viya,install` tags. Because `DEPLOY` is set to false, the SAS Viya platform deployment will not be modified. However, in your deployment directory you should see an updated `kustomization.yaml` file with generated entries for your PostgreSQL clusters. Those files should automatically be present in your site-config directory. + +### Crunchy Data 4 + +1. In the [PostgreSQL Data Transfer Guide](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=pgdatamig&docsetTarget=titlepage.htm) under "Post-transfer Steps for Crunchy Data 4", in Step 5 you are asked to remove all entries in your `kustomization.yaml` file that contain a set of strings that are used for Crunchy Data 4 PostgreSQL configuration. Because the viya4-deployment project automatically manages the configuration and creation of the PostgreSQL related entries in your `kustomization.yaml` file, you can skip Step 5 from the "Post-transfer Steps for Crunchy Data 4". +2. Configure your `ansible-vars.yaml` file to make the switch over from Crunchy Data PostgreSQL to an external PostgreSQL cluster. + * If you provisioned your PostgreSQL clusters using any [Viya 4 IAC projects](https://github.com/search?q=org%3Asassoftware+viya4-iac-&type=repositories), then your PostgreSQL configuration and connection information should already be present in the .tfstate file. Therefore, you do not need to add those entries in your `ansible-vars.yaml` file. However, you should modify the `V4_CFG_POSTGRES_SERVERS` variable if it's still configured to use an internal Crunchy instance. Here is an example: + ```yaml + # modify as below to use external instance + V4_CFG_POSTGRES_SERVERS: + default: + internal: false + ``` + * If you provisioned your PostgreSQL clusters without the use of a Viya 4 IAC project, then you must manually add definitions for each of your PostgreSQL clusters. You can see an example definition at [CONFIG-VARS.md documentation](https://github.com/sassoftware/viya4-deployment/blob/main/docs/CONFIG-VARS.md#postgresql). +3. If your PostgreSQL cluster requires a certificate for connection, ensure that `V4_CFG_TLS_TRUSTED_CA_CERTS` is set in your `ansible-vars.yaml` file and that it points to either the certificate or a directory containing the certificate. This replaces Step 6 from the "Post-transfer Steps for Crunchy Data 4" documentation. Skip over Step 7 from the "Post-transfer Steps for Crunchy Data 4" documentation, viya4-deployment performs this automatically. +4. Going back to the [PostgreSQL Data Transfer Guide](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=pgdatamig&docsetTarget=titlepage.htm) under "Post-transfer Steps for Crunchy Data 4" perform Step 8 and start the operator if your deployment is managed by the SAS Deployment Operator (if `V4_DEPLOYMENT_OPERATOR_ENABLED` was set to true in your `ansible-vars.yaml` file). Otherwise, skip this step. +5. Run the ansible-playbook again with the `viya,install` tags. This updates the `kustomization.yaml` by removing entries related to Crunchy Data 4 and adding entries for your external PostgreSQL cluster. The manifest will be rebuilt and reapplied to the cluster. This replaces step 9 from the "Post-transfer Steps for Crunchy Data 4" documentation. +6. Return to the [PostgreSQL Data Transfer Guide](https://documentation.sas.com/?cdcId=itopscdc&cdcVersion=default&docsetId=pgdatamig&docsetTarget=titlepage.htm) under "Post-transfer Steps for Crunchy Data 4". Perform Steps 10 and the remainder of the steps to complete the data transfer. diff --git a/roles/baseline/defaults/main.yml b/roles/baseline/defaults/main.yml index d9cd7636..7f87225d 100644 --- a/roles/baseline/defaults/main.yml +++ b/roles/baseline/defaults/main.yml @@ -111,6 +111,17 @@ NFS_CLIENT_CONFIG: storageClass: archiveOnDelete: "false" name: sas +# EFS best practice NFS mount options for the aws provider +NFS_EFS_CLIENT_CONFIG: + nfs: + mountOptions: + - noresvport + - rsize=1048576 + - wsize=1048576 + - soft + - timeo=600 + - retrans=2 + - _netdev ## pg-storage storage class config PG_NFS_CLIENT_NAME: nfs-subdir-external-provisioner-pg-storage diff --git a/roles/baseline/tasks/nfs-subdir-external-provisioner.yaml b/roles/baseline/tasks/nfs-subdir-external-provisioner.yaml index c2bf3c6e..c499966d 100644 --- a/roles/baseline/tasks/nfs-subdir-external-provisioner.yaml +++ b/roles/baseline/tasks/nfs-subdir-external-provisioner.yaml @@ -36,6 +36,18 @@ - uninstall - update +- name: Update NFS_CLIENT_CONFIG configurations for EFS + set_fact: + NFS_CLIENT_CONFIG: "{{ NFS_CLIENT_CONFIG | combine(NFS_EFS_CLIENT_CONFIG, recursive=True) }}" + PG_NFS_CLIENT_CONFIG: "{{ PG_NFS_CLIENT_CONFIG | combine(NFS_EFS_CLIENT_CONFIG, recursive=True) }}" + when: + - PROVIDER == "aws" + - STORAGE_TYPE_BACKEND is defined + - STORAGE_TYPE_BACKEND == "efs" + tags: + - install + - update + - name: Deploy nfs-subdir-external-provisioner-sas kubernetes.core.helm: name: "{{ NFS_CLIENT_NAME }}" diff --git a/roles/common/tasks/main.yaml b/roles/common/tasks/main.yaml index ef3eae9a..698488c8 100644 --- a/roles/common/tasks/main.yaml +++ b/roles/common/tasks/main.yaml @@ -192,6 +192,12 @@ when: - tfstate.ssh_private_key is defined - tfstate.ssh_private_key.value|length > 0 + - name: tfstate - storage type backend var # noqa: name[casing] + set_fact: + STORAGE_TYPE_BACKEND: "{{ tfstate.storage_type_backend.value }}" + when: + - tfstate.storage_type_backend is defined + - tfstate.storage_type_backend.value|length > 0 ### Deprecations - name: tfstate - postgres admin # noqa: name[casing] set_fact: