roles: new roles to deploy cluster

Added new per-component roles(vmstorage, vminsert, vmselect) to deploy a cluster by using binaries, similar to how other existing roles do the deployment. Added a deprecation notice to the old roles.

New roles allow to create a full cluster setup with vmauth as a load balancer.

Signed-off-by: Zakhar Bessarab <[email protected]>

.github/workflows/test.yml

@@ -24,3 +24,4 @@ jobs:
           make init-venv
           
           make molecule-converge
+          make molecule-converge-integration

Makefile

@@ -24,7 +24,29 @@ molecule-converge-vmalert:
 	make activate-venv; \
 	cd roles/vmalert && molecule converge -s docker
 
-molecule-converge: molecule-converge-single molecule-converge-vmagent molecule-converge-vmalert
+molecule-converge-vminsert:
+	make activate-venv; \
+	cd roles/vminsert && molecule converge
+
+molecule-converge-vmstorage:
+	make activate-venv; \
+	cd roles/vmstorage && molecule converge
+
+molecule-converge-vmselect:
+	make activate-venv; \
+	cd roles/vmselect && molecule converge
+
+molecule-converge-vmauth:
+	make activate-venv; \
+	cd roles/vmauth && molecule converge
+
+molecule-converge-cluster-integration:
+	make activate-venv; \
+	cd playbooks/ && molecule converge -s cluster
+
+molecule-converge: molecule-converge-single molecule-converge-vmagent molecule-converge-vmalert molecule-converge-vminsert molecule-converge-vmstorage molecule-converge-vmselect molecule-converge-vmauth
+
+molecule-converge-integration: molecule-converge-cluster-integration
 
 molecule-destroy-single:
 	make activate-venv; \
@@ -38,4 +60,26 @@ molecule-destroy-vmalert:
 	make activate-venv; \
 	cd roles/vmalert && molecule destroy -s docker
 
-molecule-destroy: molecule-destroy-single molecule-destroy-vmagent molecule-destroy-vmalert
+molecule-destroy-vminsert:
+	make activate-venv; \
+	cd roles/vminsert && molecule destroy
+
+molecule-destroy-vmstorage:
+	make activate-venv; \
+	cd roles/vmstorage && molecule destroy
+
+molecule-destroy-vmselect:
+	make activate-venv; \
+	cd roles/vmselect && molecule destroy
+
+molecule-destroy-vmauth:
+	make activate-venv; \
+	cd roles/vmauth && molecule destroy
+
+molecule-destroy-cluster-integration:
+	make activate-venv; \
+	cd playbooks/ && molecule destroy -s cluster
+
+molecule-destroy: molecule-destroy-single molecule-destroy-vmagent molecule-destroy-vmalert molecule-destroy-vminsert molecule-destroy-vmstorage molecule-destroy-vmselect molecule-destroy-vmauth molecule-destroy-cluster-integration
+
+molecule-destroy-integration: molecule-destroy-cluster-integration

README.md

@@ -16,22 +16,67 @@ ansible-galaxy collection install victoriametrics.cluster
 
 Collection includes the following roles:
 
-- [docker](./roles/docker) - role for docker installation
-- [load_balancer](./roles/load_balancer) - nginx setup for load balancing between select and insert nodes
-- [cluster](./roles/cluster) - installs and configures VictoriaMetrics cluster running in docker
 - [single](./roles/single) - installs and configures VictoriaMetrics single node
 - [vmagent](./roles/vmagent) - installs and configures `vmagent`
 - [vmalert](./roles/vmalert) - installs and configures `vmalert`
+- [vmselect](./roles/vmselect) - installs and configures `vmselect`
+- [vmstorage](./roles/vmstorage) - installs and configures `vmstorage`
+- [vminsert](./roles/vminsert) - installs and configures `vminsert`
+- [vmauth](./roles/vmauth) - installs and configures `vmauth`
+
+See [cluster](playbooks/cluster.yml) and [single](playbooks/single.yml) playbooks for examples of how to use these
+roles.
+
+Deprecated roles:
+
+- [docker](./roles/docker) - role for docker installation
+- [load_balancer](./roles/load_balancer) - nginx setup for load balancing between select and insert nodes
+- [cluster](./roles/cluster) - installs and configures VictoriaMetrics cluster running in docker
+
+These roles are deprecated in favor of roles for each component. They are still available in the collection, but will be
+removed in future releases.
+Note that per-component roles are using binary releases of VictoriaMetrics, while cluster role is using docker images.
+
+## Cluster deployment
+
+Cluster deployment example is available in [playbooks/cluster.yml](./playbooks/cluster.yml).
+The playbook deploys VictoriaMetrics cluster and vmauth to act as a load balancer and authentication proxy.
+See [inventory](./inventory_example/cluster-inventory) for example of inventory file.
+
+Here is a diagram of the cluster deployment:
+
+```mermaid
+graph TD
+    A[vmauth] -- /select/* --> B[vmselect-1]
+    A[vmauth] -- /select/* --> B2[vmselect-2]
+    A -- /insert/* --> C[vminsert-1]
+    A -- /insert/* --> C2[vminsert-2]
+    C --> D[vmstorage-1]
+    C --> D2[vmstorage-2]
+    C --> D3[vmstorage-3]
+    C2 --> D[vmstorage-1]
+    C2 --> D2[vmstorage-2]
+    C2 --> D3[vmstorage-3]
+    B --> D[vmstorage-1]
+    B --> D2[vmstorage-1]
+    B --> D3[vmstorage-1]
+    B2 --> D[vmstorage-1]
+    B2 --> D2[vmstorage-1]
+    B2 --> D3[vmstorage-1]
+```
 
-# TODO
-- add non-docker environment
-- fix hardcoded ports
+It's also possible to use molecule scenario to create a local cluster for testing.
+See [molecule](./playbooks/molecule/cluster) directory for details. The scenario uses docker as a driver and
+sets up a container for each component. The scenario can be deployed by
+using `make molecule-converge-cluster-integration` command.
 
 # Development
 
 In order to set up development environment, you need to have `docker`, `python` and `make` installed.
-Run `make init-venv` to create virtual environment and install required packages for linting and testing with [molecule](https://ansible.readthedocs.io/projects/molecule).
+Run `make init-venv` to create virtual environment and install required packages for linting and testing
+with [molecule](https://ansible.readthedocs.io/projects/molecule).
 
-Please, note that [cluster](./roles/cluster) role is tested with `vagrant` and `libvirt` provider and requires `vagrant` [to be installed](https://www.vagrantup.com/downloads).
+Please, note that [cluster](./roles/cluster) role is tested with `vagrant` and `libvirt` provider and
+requires `vagrant` [to be installed](https://www.vagrantup.com/downloads).
 
 Refer to [Makefile](./Makefile) for available commands for linting and molecule testing.

inventory_example/cluster-inventory

@@ -1,30 +1,22 @@
-[victoria_storage]
-victoria-storage-01
-victoria-storage-02
-victoria-storage-03
+[vmstorage]
+vmstorage-01
+vmstorage-02
+vmstorage-03
 
-[victoria_insert]
-victoria-insert-01
-victoria-insert-02
-victoria-insert-03
+[vminsert]
+vminsert-01
+vminsert-02
+vminsert-03
 
-[victoria_select]
-victoria-select-01
-victoria-select-02
-
-[victoria_storage:vars]
-vm_role=victoria-storage
-
-[victoria_insert:vars]
-vm_role=victoria-insert
-
-[victoria_select:vars]
-vm_role=victoria-select
-
-[load-balancer]
-load-balancer-01
+[vmselect]
+vmselect-01
+vmselect-02
 
 [victoria_cluster:children]
-victoria_select
-victoria_insert
-victoria_storage
+vmselect
+vminsert
+vmstorage
+
+[vmauth]
+vmauth-01
+vmauth-02

playbooks/cluster.yml

@@ -0,0 +1,50 @@
+---
+- name: Setup storage
+  become: true
+  hosts: vmstorage
+  roles:
+    - vmstorage
+
+- name: Setup vmselect
+  become: true
+  hosts: vmselect
+  vars:
+    vmselect_cache_dir: "/var/lib/vmselect"
+    vmselect_config:
+      cacheDataPath: "/var/lib/vmselect"
+      storageNode: "{{ groups['vmstorage'] | join(',') }}"
+  roles:
+    - vmselect
+
+- name: Setup vminsert
+  become: true
+  hosts: vminsert
+  vars:
+    vminsert_config:
+      storageNode: "{{ groups['vmstorage'] | join(',') }}"
+  roles:
+    - vminsert
+
+- name: Setup vmauth
+  become: true
+  hosts: vmauth
+  vars:
+    # See: https://docs.victoriametrics.com/vmauth/#load-balancer-for-victoriametrics-cluster
+    vmauth_auth_config: |-
+      unauthorized_user:
+        url_map:
+          - src_paths:
+              - "/insert/.+"
+            url_prefix:
+            {% for insert in groups['vminsert'] %}
+            - "http://{{ insert }}:8480/"
+            {% endfor %}
+          - src_paths:
+              - "/select/.+"
+            url_prefix:
+           {% for select in groups['vmselect'] %}
+           - "http://{{ select }}:8481/"
+           {% endfor %}
+
+  roles:
+    - vmauth

playbooks/molecule/cluster/converge.yml

@@ -0,0 +1,50 @@
+---
+- name: Setup storage
+  become: true
+  hosts: vmstorage
+  roles:
+    - vmstorage
+
+- name: Setup vmselect
+  become: true
+  hosts: vmselect
+  vars:
+    vmselect_cache_dir: "/var/lib/vmselect"
+    vmselect_config:
+      cacheDataPath: "/var/lib/vmselect"
+      storageNode: "{{ groups['vmstorage'] | join(',') }}"
+  roles:
+    - vmselect
+
+- name: Setup vminsert
+  become: true
+  hosts: vminsert
+  vars:
+    vminsert_config:
+      storageNode: "{{ groups['vmstorage'] | join(',') }}"
+  roles:
+    - vminsert
+#
+- name: Setup vmauth
+  become: true
+  hosts: vmauth
+  vars:
+    # See: https://docs.victoriametrics.com/vmauth/#load-balancer-for-victoriametrics-cluster
+    vmauth_auth_config: |-
+      unauthorized_user:
+        url_map:
+          - src_paths:
+              - "/insert/.+"
+            url_prefix:
+            {% for insert in groups['vminsert'] %}
+            - "http://{{ insert }}:8480/"
+            {% endfor %}
+          - src_paths:
+              - "/select/.+"
+            url_prefix:
+           {% for select in groups['vmselect'] %}
+           - "http://{{ select }}:8481/"
+           {% endfor %}
+
+  roles:
+    - vmauth