introduce profiles adding SingleReplica (SNO) topology on AWS (#24)

* supporting stacks to create SNO node and resources * feat: intro profiles with sno/SingleReplica * creating a working SNO with profiles * doc: add sno install steps * doc: deployment guide * fix: rename from topology vars to cluster_profile * doc: add disk layout * chore: remove unused comments
mtulio · Feb 8, 2023 · 55271eb · 55271eb
1 parent 633bfe5
commit 55271eb
Show file tree

Hide file tree

Showing 49 changed files with 1,079 additions and 192 deletions.
diff --git a/docs/installing/aws-sno.md b/docs/installing/aws-sno.md
@@ -0,0 +1,174 @@
+# AWS Single Node Openshift
+
+Install a single node replica OpenShift/OKD.
+
+The steps will create every infrastrucure stack to deploy a SNO on the AWS provider.
+
+The infra resources created will be:
+- VPC and it's subnets on a single AZ
+- Security Groups
+- Load Balancers for API (public and private) and Apps
+- DNS Zones and RRs
+- Compute resources: Bootstrap and single node control plane
+
+## Deployment considerations
+
+The deployment described in this document is introducing a more performant disk layout to avoid disruptions and concurrency between resources on the same disk (by default). The disk layout is when using EC2 instance `m6id.xlarge`:
+- Ephemeral disk (local storage) for `/var/lib/containers`
+- Dedicated etcd EBS mounted on `/var/lib/etcd`
+
+```text
+$ cat ~/opct/results/opct-sno-aws/sno2-run-lsblk.txt
+NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
+nvme0n1     259:0    0   128G  0 disk 
+|-nvme0n1p1 259:4    0     1M  0 part 
+|-nvme0n1p2 259:5    0   127M  0 part 
+|-nvme0n1p3 259:6    0   384M  0 part /boot
+`-nvme0n1p4 259:7    0 127.5G  0 part /sysroot
+nvme1n1     259:1    0    32G  0 disk 
+`-nvme1n1p1 259:3    0    32G  0 part /var/lib/etcd
+nvme2n1     259:2    0 220.7G  0 disk /var/lib/containers
+```
+
+Using this layout we decreased the amount of memory used by monitoring stack (Prometheus), and, consequently the etcd when using a single/shared-disk deployment. The API disruptions decreased drastically, allowing to use smaller instance types with 16GiB of RAM and 4 vCPU.
+
+Steps:
+- Generate the SNO ignitions
+- Create the Stacks: Network, IAM, DNS, LB
+- Create the Compute with ignition
+
+
+## Create the configuration variables
+
+```bash
+cat <<EOF> ./vars-sno.yaml
+provider: aws
+cluster_name: sno-aws
+
+config_base_domain: devcluster.openshift.com
+config_ssh_key: "$(cat ~/.ssh/id_rsa.pub)"
+config_pull_secret_file: ${HOME}/.openshift/pull-secret-latest.json
+config_cluster_region: us-east-1
+
+cluster_profile: sno
+create_worker: no
+destroy_bootstrap: no
+
+config_compute_replicas: 0
+config_controlplane_replicas: 1
+cert_expected_nodes: 0
+config_bootstrapinplace_disk: /dev/nvme0n1
+
+# Choose the instance type for SNO node.
+# NOTE: the okd-installer does not support yet the spot
+#- m6i.xlarge: ~140/od ~52/spot
+#- m6id.xlarge: ~173/od ~52/spot
+#- m6idn.xlarge: ~232/od ~52/spot
+#- r5d.xlarge: ~210/od ~52/spot
+#- r6id.xlarge: ~220/od ~54/spot
+#- t4g.xlarge: ~98/od 29/spot
+#- m6gd.xlarge: ~131/od ~52/spot
+#- r6gd.2xlaarge: ~168/od ~62/spot
+controlplane_instance: m6id.xlarge
+
+# Patch manifests to:
+# 1) mount ephemeral disk on /var/lib/containers
+# 2) mount extra disk for etcd (/var/lib/etcd)
+# 3) remove machine api objects
+config_patches:
+- mc_varlibcontainers
+- mc_varlibetcd
+- rm-capi-machines
+
+cfg_patch_mc_varlibcontainers:
+  device_path: /dev/nvme2n1
+  device_name: nvme2n1
+  machineconfiguration_roles:
+  - master
+EOF
+```
+
+## Client
+
+See [Install the Clients](./install-openshift-install.md)
+
+## Config
+
+Create the installation configuration:
+
+```bash
+ansible-playbook mtulio.okd_installer.config \
+    -e mode=create \
+    -e @./vars-sno.yaml
+```
+
+## Deploy each stack
+
+### Network Stack
+
+```bash
+ansible-playbook mtulio.okd_installer.stack_network \
+    -e @./vars-sno.yaml
+```
+
+### IAM Stack
+
+
+```bash
+ansible-playbook mtulio.okd_installer.stack_iam \
+    -e @./vars-sno.yaml
+```
+
+### DNS Stack
+
+```bash
+ansible-playbook mtulio.okd_installer.stack_dns \
+    -e @./vars-sno.yaml
+```
+
+```bash
+ansible-playbook mtulio.okd_installer.stack_loadbalancer \
+    -e @./vars-sno.yaml
+```
+
+### Compute Stack
+
+- Create the Bootstrap Node
+
+```bash
+ansible-playbook mtulio.okd_installer.create_node \
+    -e @./vars-sno.yaml \
+    -e node_role=controlplane
+```
+
+## Deploy cluster
+
+Deploy a cluster creating all the resources with a single execution/playbook:
+
+```bash
+ansible-playbook mtulio.okd_installer.create_all \
+    -e @./vars-sno.yaml
+```
+
+You can check when the bootstrap finished, or the Single Replica node have joined to the cluster:
+
+```bash
+$ KUBECONFIG=$HOME/.ansible/okd-installer/clusters/opct-sno/auth/kubeconfig oc get nodes
+NAME             STATUS   ROLES                               AGE   VERSION
+ip-10-0-50-187   Ready    control-plane,master,tests,worker   24m   v1.25.4+77bec7a
+
+```
+
+The you can destroy the bootstrap node:
+
+```bash
+ansible-playbook mtulio.okd_installer.destroy_bootstrap \
+    -e @./vars-sno.yaml
+```
+
+## Destroy
+
+```bash
+ansible-playbook mtulio.okd_installer.destroy_cluster \
+    -e @./vars-sno.yaml
+```
diff --git a/playbooks/create_all.yaml b/playbooks/create_all.yaml
@@ -35,24 +35,15 @@
 - name: OKD Installer | Create all | create stack | Load Balancer Router
   ansible.builtin.import_playbook: stack_loadbalancer.yaml
   vars:
-    var_file: "{{ playbook_dir }}/vars/{{ config_provider }}/loadbalancer-router-default.yaml"
-  when:
-    - (config_provider is defined)
-    - (config_platform|d('') != "aws")
+    var_file: "{{ playbook_dir }}/vars/{{ config_provider }}/profiles/{{ cluster_profile  }}/loadbalancer-router-default.yaml"
+  when: config_platform|d('') == "none"
 
 - name: OKD Installer | Create all | create stack | approve certs
   ansible.builtin.import_playbook: approve_certs.yaml
   vars:
     certs_max_retries: 8
     cert_wait_interval_sec: 60
-  when:
-    - (config_provider == 'aws') or (config_platform == 'none')
-
-- name: OKD Installer | Create all | create basic image-registry
-  ansible.builtin.import_playbook: create_imageregistry.yaml
-  when:
-    - (config_provider == 'aws') or (config_platform == 'none')
-    - create_registry|d('no') == 'yes'
+  when: config_platform == 'none'
 
 - name: OKD Installer | Create all | Load Config
   ansible.builtin.import_playbook: config.yaml
@@ -61,26 +52,14 @@
 
 - name: OKD Installer | Create all | Bootstrap Destroy
   ansible.builtin.import_playbook: destroy_bootstrap.yaml
-  when:
-    - (config_provider == 'aws')
-    - destroy_bootstrap | d('no') == 'yes'
+  when: destroy_bootstrap | d('yes') == 'yes'
 
-# - name: OKD Installer | Create ALL Finish
-#   hosts: '{{ target | default("localhost") }}'
-#   connection: local
 
 - name: OKD Installer | Create ALL | End
   hosts: '{{ target|default("localhost") }}'
   connection: local
   gather_facts: true
 
-  # tasks:
-  #   - name: OKD Installer | Create all | Timer end
-  #     ansible.builtin.debug:
-  #       msg:
-  #         - "start=[{{ datetime_start | d('') }}] end=[{{ ansible_date_time.iso8601 }}]"
-  #         - "export KUBECONFIG={{ config_install_dir }}/auth/kubeconfig"
-
   tasks:
     - name: OKD Installer | Create all | Timer end
       ansible.builtin.set_fact:

diff --git a/playbooks/create_node.yaml b/playbooks/create_node.yaml
@@ -9,6 +9,8 @@
 - name: okd-installer | Create Stack | Compute
   hosts: '{{ target|default("localhost") }}'
   connection: local
+  vars:
+    profile_path: "{{ playbook_dir }}/vars/{{ config_provider }}/profiles/{{ cluster_profile|d('default') }}"
 
   vars_prompt:
     - name: node_role
@@ -25,17 +27,17 @@
         file: "{{ var_file }}"
       when: var_file is defined
 
-    - name: Include Compute Variables
+    - name: okd-installer | Stack | Compute | Include Vars - Pre-build topologies
       ansible.builtin.include_vars:
-        file: "./vars/{{ provider }}/node-{{ node_role }}.yaml"
-      when:
-        - (compute_resources is not defined) or (compute_resources | length <= 0)
+        file: "{{ profile_path }}/node-{{ node_role }}.yaml"
+      when: var_file is not defined
 
   roles:
     - role: bootstrap
-      when: node_role|d('') == 'bootstrap'
+      when: (node_role | d('') == 'bootstrap') or (cluster_profile | d('') == 'sno')
     - role: cloud_compute
 
+
 - name: okd-installer | Create Stack | Compute | Save state
   ansible.builtin.import_playbook: config.yaml
   vars:

diff --git a/playbooks/create_node_all.yaml b/playbooks/create_node_all.yaml
@@ -9,86 +9,68 @@
 - name: okd-installer | Stack | Compute ALL | Create
   hosts: '{{ target|default("localhost") }}'
   connection: local
+  vars:
+    profile_path: "{{ playbook_dir }}/vars/{{ config_provider }}/profiles/{{ cluster_profile|d('default') }}"
 
   tasks:
     # Create Compute: Bootstrap node
-    - name: okd-installer | Stack | Compute | Set User provided (Bootstrap)
-      ansible.builtin.include_vars:
-        file: "{{ var_file_bootstrap }}"
-      when:
-        - var_file_bootstrap is defined
-        - topology_compute is not defined
-
-    - name: okd-installer | Stack | Compute | Set Topology - {{ topology_compute }}
-      ansible.builtin.include_vars:
-        file: "./vars/{{ provider }}/topologies/{{ topology_compute }}/node-bootstrap.yaml"
-      when:
-        - topology_compute is defined
+    - name: okd-installer | Stack | Compute | Bootstrap
+      block:
+      - name: okd-installer | Stack | Compute | Set User provided (Bootstrap)
+        ansible.builtin.include_vars:
+          file: "{{ var_file_bootstrap }}"
+        when: var_file_bootstrap is defined
 
-    - name: okd-installer | Stack | Compute | Set Defaults
-      ansible.builtin.include_vars:
-        file: "./vars/{{ provider }}/node-bootstrap.yaml"
-      when:
-        - topology_compute is not defined
+      - name: okd-installer | Stack | Compute | Include Topology - {{ cluster_profile }}
+        ansible.builtin.include_vars:
+          file: "{{ profile_path }}/node-bootstrap.yaml"
+        when: var_file_bootstrap is not defined
 
-    - name: okd-installer | Stack | Compute | Create Bootstrap
-      ansible.builtin.include_role:
-        name: "{{ item }}"
-      loop:
-        - 'bootstrap'
-        - 'cloud_compute'
+      - name: okd-installer | Stack | Compute | Create Bootstrap
+        ansible.builtin.include_role:
+          name: "{{ item }}"
+        loop:
+          - "bootstrap"
+          - "cloud_compute"
 
     # Create Compute: Control Plane nodes
-    - name: okd-installer | Stack | Compute | Set User provided (CPlane)
-      ansible.builtin.include_vars:
-        file: "{{ var_file_controlplane }}"
-      when:
-        - var_file_controlplane is defined
-        - topology_compute is not defined
+    - name: okd-installer | Stack | Compute | Control Plane
+      block:
+      - name: okd-installer | Stack | Compute | Set User provided (CPlane)
+        ansible.builtin.include_vars:
+          file: "{{ var_file_controlplane }}"
+        when: var_file_controlplane is defined
 
-    - name: okd-installer | Stack | Compute | Set Topology - {{ topology_compute }}
-      ansible.builtin.include_vars:
-        file: "./vars/{{ provider }}/topologies/{{ topology_compute }}/node-controlplane.yaml"
-      when:
-        - topology_compute is defined
+      - name: okd-installer | Stack | Compute | Set Topology - {{ cluster_profile }}
+        ansible.builtin.include_vars:
+          file: "{{ profile_path }}/node-controlplane.yaml"
+        when: var_file_controlplane is not defined
 
-    - name: okd-installer | Stack | Compute | Set Defaults
-      ansible.builtin.include_vars:
-        file: "./vars/{{ provider }}/node-controlplane.yaml"
-      when:
-        - topology_compute is not defined
-
-    - name: okd-installer | Stack | Compute | Create controlplane
-      ansible.builtin.include_role:
-        name: "{{ item }}"
-      loop:
-        - 'cloud_compute'      
+      - name: okd-installer | Stack | Compute | Create controlplane
+        ansible.builtin.include_role:
+          name: "{{ item }}"
+        loop:
+          - "cloud_compute"
 
     # Create Compute: Compute nodes
-    - name: okd-installer | Stack | Compute | Set User provided (CMP)
-      ansible.builtin.include_vars:
-        file: "{{ var_file_compute }}"
-      when:
-        - var_file_compute is defined
-        - topology_compute is not defined
-
-    - name: okd-installer | Stack | Compute | Set Topology {{ topology_compute }}
-      ansible.builtin.include_vars:
-        file: "./vars/{{ provider }}/topologies/{{ topology_compute }}/node-compute.yaml"
-      when:
-        - topology_compute is defined
+    - name: okd-installer | Stack | Compute | Worker
+      when: create_worker | d('yes') == 'yes'
+      block:
+      - name: okd-installer | Stack | Compute | Set User provided (CMP)
+        ansible.builtin.include_vars:
+          file: "{{ var_file_compute }}"
+        when: var_file_compute is defined
 
-    - name: okd-installer | Stack | Compute | Set Default
-      ansible.builtin.include_vars:
-        file: "./vars/{{ provider }}/node-compute.yaml"
-      when:
-        - topology_compute is not defined
+      - name: okd-installer | Stack | Compute | Include Topology {{ cluster_profile }}
+        ansible.builtin.include_vars:
+          file: "{{ profile_path }}/node-compute.yaml"
+        when: var_file_compute is not defined
 
-    - name: okd-installer | Stack | Compute | Create compute nodes
-      ansible.builtin.include_role:
-        name: "{{ item }}"
-      loop:
-        - 'cloud_compute'    
+      - name: okd-installer | Stack | Compute | Create compute nodes
+        ansible.builtin.include_role:
+          name: "{{ item }}"
+        loop:
+          - "cloud_compute"
 
 - name: okd-installer | Stack | Compute ALL | Save state
   import_playbook: config.yaml