Skip to content

Commit 7ec0f41

Browse files
NickLarsenNZNickLarsenNZ
committed
Add guide for deploying operators and CSI drivers on separate nodes
NOTE: This is work-in-progress
1 parent dd1e05d commit 7ec0f41

File tree

2 files changed

+354
-0
lines changed

2 files changed

+354
-0
lines changed

‎modules/guides/nav.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,4 @@
66
** xref:viewing-and-verifying-sboms.adoc[]
77
** xref:enabling-verification-of-image-signatures.adoc[]
88
** xref:kubernetes-cluster-domain.adoc[]
9+
** xref:deploy-operators-and-csi-drivers-separately.adoc[]

‎modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc‎

Lines changed: 353 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,353 @@
1+
= Deploying operators and CSI drivers on separate nodes
2+
:related-issue: https://github.com/stackabletech/issues/issues/763
3+
:secret-operator-values: https://github.com/stackabletech/secret-operator/blob/main/deploy/helm/secret-operator/values.yaml
4+
:listener-operator-values: https://github.com/stackabletech/listener-operator/blob/main/deploy/helm/listener-operator/values.yaml
5+
:commons-operator-values: https://github.com/stackabletech/commons-operator/blob/main/deploy/helm/commons-operator/values.yaml
6+
:nifi-operator-values: https://github.com/stackabletech/nifi-operator/blob/main/deploy/helm/nifi-operator/values.yaml
7+
// TODO: Is there a way to make the links above go to the right place?
8+
// Eg: we can search/replace "0.0.0-dev" with the release, but in the above case we would need to replace "main".
9+
10+
Operators can be installed on nodes separate from where the workloads will run.
11+
There is a caveat when it comes to two operators: Secret Operator and Listener Operator.
12+
They make use of the Container Storage Interface (CSI) and have components that must run on nodes with workloads that mount CSI volumes.
13+
14+
This guide will show how to schedule operators on one group of nodes (for example, a Karpenter NodePool), while scheduling applicable components where workload will run.
15+
16+
== Setup
17+
18+
You will need a Kubernetes cluster with multiple nodes split into two groups:
19+
20+
* stackable-operators
21+
* stackable-workloads
22+
23+
TODO: tabs for kind or Karpenter NodePools
24+
25+
[NOTE]
26+
====
27+
This guide will use _KinD_ to demonstrate, but if you are using Karpenter (eg: AWK EKS),
28+
you can adjust the labels to be based on the name of your NodePools.
29+
30+
For example:
31+
32+
* `karpenter.sh/nodepool: stackable-operators`
33+
* `karpenter.sh/nodepool: stackable-workloads`
34+
35+
====
36+
37+
Create a KinD config called `kind-config.yaml` containing:
38+
39+
[source,yaml]
40+
----
41+
kind: Cluster
42+
apiVersion: kind.x-k8s.io/v1alpha4
43+
nodes:
44+
- role: control-plane
45+
- role: worker
46+
labels:
47+
nodepool: stackable-operators
48+
- role: worker
49+
labels:
50+
nodepool: stackable-operators
51+
- role: worker
52+
labels:
53+
nodepool: stackable-workloads
54+
- role: worker
55+
labels:
56+
nodepool: stackable-workloads
57+
----
58+
59+
Launch the cluster:
60+
61+
[source,bash]
62+
----
63+
kind create cluster --name stackable --config kind-config.yaml
64+
----
65+
66+
You can see which nodes are in which _nodepool_ by using the following command:
67+
68+
[source,bash]
69+
----
70+
kubectl get nodes -o json | jq '
71+
.items[]
72+
| .metadata.name as $name
73+
| .metadata.labels["nodepool"] as $nodepool
74+
| $nodepool//empty
75+
| {"nodename": $name, "nodepool": $nodepool}
76+
'
77+
----
78+
79+
== Prepare Helm Values for the Stackable Operators
80+
81+
[NOTE]
82+
====
83+
Most Stackable operators use the same Helm Values structure, however Secret and
84+
Listener operator differ slightly - which is what allows the components to be
85+
configured independently of each other.
86+
====
87+
88+
// TODO: Move these into files and include them (so we can run them easily)
89+
90+
[tabs]
91+
====
92+
Secret Operator::
93+
+
94+
--
95+
Store the values in a file called `stackable-secret-operator.yaml`.
96+
97+
// TODO: Link to default values
98+
99+
[source,yaml]
100+
----
101+
controllerService:
102+
nodeSelector:
103+
nodepool: stackable-operators
104+
105+
csiNodeDriver:
106+
# Node Drivers need to run on the same nodes as the workloads using them
107+
nodeSelector:
108+
nodepool: stackable-workloads
109+
----
110+
111+
--
112+
113+
Listener Operator::
114+
+
115+
--
116+
Store the values in a file called `stackable-listener-operator.yaml`.
117+
118+
// TODO: Link to default values
119+
120+
[source,yaml]
121+
----
122+
csiProvisioner:
123+
nodeSelector:
124+
nodepool: stackable-operators
125+
126+
csiNodeDriver:
127+
# Node Drivers need to run on the same nodes as the workloads using them
128+
nodeSelector:
129+
nodepool: stackable-workloads
130+
----
131+
132+
--
133+
134+
Remaining operators::
135+
+
136+
--
137+
Store the values in a file called `stackable-operators.yaml`.
138+
139+
// TODO: Link to default values for remaining operators used in this guide
140+
141+
[source,yaml]
142+
----
143+
nodeSelector:
144+
nodepool: stackable-operators
145+
----
146+
147+
--
148+
====
149+
150+
NOTE: If you would like to run on nodes with taints, you can list `tolerations` next to the `nodeSelector`.
151+
152+
== Install the Stackable Operators
153+
154+
Now install the operators to the applicable node pools by using the Helm overrides
155+
156+
[tabs]
157+
====
158+
Secret Operator::
159+
+
160+
--
161+
NOTE: This operator uses a specific values file.
162+
163+
[source,bash]
164+
----
165+
helm install secret-operator \
166+
--version=0.0.0-dev \
167+
--values=stackable-secret-operator.yaml \
168+
oci://oci.stackable.tech/sdp-charts/secret-operator
169+
----
170+
171+
--
172+
173+
Listener Operator::
174+
+
175+
--
176+
NOTE: This operator uses a specific values file.
177+
178+
[source,bash]
179+
----
180+
helm install listener-operator \
181+
--version=0.0.0-dev \
182+
--values=stackable-listener-operator.yaml \
183+
oci://oci.stackable.tech/sdp-charts/listener-operator
184+
----
185+
186+
--
187+
188+
Remaining operators::
189+
+
190+
--
191+
NOTE: These operator use the same values file.
192+
193+
[source,bash]
194+
----
195+
helm install commons-operator \
196+
--version=0.0.0-dev \
197+
--values=stackable-operators.yaml \
198+
oci://oci.stackable.tech/sdp-charts/commons-operator
199+
200+
helm install nifi-operator \
201+
--version=0.0.0-dev \
202+
--values=stackable-operators.yaml \
203+
oci://oci.stackable.tech/sdp-charts/nifi-operator
204+
----
205+
206+
--
207+
====
208+
209+
You should now see that the operators are running on the `stackable-operator` nodes, while the CSI drivers are running on the `stackable-workload` nodes.
210+
211+
Pods running on the `stackable-operators` node pool:
212+
213+
[source,bash]
214+
----
215+
OPERATORS_NODEPOOL=$(kubectl get nodes -l nodepool=stackable-operators -o jsonpath="{.items[*].metadata.name}" | tr ' ' ',')
216+
echo "Nodes in operators pool: $OPERATORS_NODEPOOL\n"
217+
kubectl get pods -o json | jq --raw-output --arg nodepool "$OPERATORS_NODEPOOL" '.items[] | .metadata.name as $podname | .spec.nodeName as $nodename | select($nodename | IN($nodepool | split(",")[])) | $podname'
218+
----
219+
220+
You should see similar output showing the Stackable Operators are running only on nodes with the label `nodepool: stackable-operators`.
221+
222+
[source]
223+
---
224+
Nodes in operators pool: stackable-worker,stackable-worker2
225+
226+
commons-operator-deployment-674c469b47-nm5vb
227+
listener-operator-csi-provisioner-85b686d48-hv5kf
228+
nifi-operator-deployment-7c59778bb8-r26b8
229+
secret-operator-66b85c669d-7hsxs
230+
---
231+
232+
Pods running on the `stackable-workloads` node pool:
233+
234+
[source,bash]
235+
----
236+
WORKLOADS_NODEPOOL=$(kubectl get nodes -l nodepool=stackable-workloads -o jsonpath="{.items[*].metadata.name}" | tr ' ' ',')
237+
echo "Nodes in workloads pool: $WORKLOADS_NODEPOOL\n"
238+
kubectl get pods -o json | jq --raw-output --arg nodepool "$WORKLOADS_NODEPOOL" '.items[] | .metadata.name as $podname | .spec.nodeName as $nodename | select($nodename | IN($nodepool | split(",")[])) | $podname'
239+
----
240+
241+
You should see similar output showing the Stackable CSI Node Drivers are running only on nodes with the label `nodepool: stackable-workloads`.
242+
243+
[source]
244+
---
245+
Nodes in workloads pool: stackable-worker3,stackable-worker4
246+
247+
listener-operator-csi-node-driver-lv5r4
248+
listener-operator-csi-node-driver-vdzsq
249+
secret-operator-csi-node-driver-d8sqw
250+
secret-operator-csi-node-driver-zkrv6
251+
---
252+
253+
The CSI Node Drivers register as such.
254+
This can be seen with the driver count being 2 (one for listener-operator volumes, and one for secret-operator volumes) for nodes in the workloads pool:
255+
256+
[source,console]
257+
----
258+
$ kubectl get csinodes
259+
NAME DRIVERS AGE
260+
stackable-control-plane 0 3h40m
261+
stackable-worker 0 3h39m
262+
stackable-worker2 0 3h39m
263+
stackable-worker3 2 3h39m
264+
stackable-worker4 2 3h39m
265+
----
266+
267+
== Install a workload
268+
269+
We'll install a NiFi cluster onto a `stackable-workload` node. Create a new file called `nifi.yaml` with the following contents:
270+
// This is taken from the NiFi Getting Started, but with some modifications.
271+
// TODO: Update nifi getting started to remove zookeeper.
272+
273+
[source,yaml]
274+
----
275+
---
276+
apiVersion: v1
277+
kind: Secret
278+
metadata:
279+
name: simple-admin-credentials
280+
stringData:
281+
admin: admin
282+
---
283+
apiVersion: authentication.stackable.tech/v1alpha1
284+
kind: AuthenticationClass
285+
metadata:
286+
name: simple-nifi-users
287+
spec:
288+
provider:
289+
static:
290+
userCredentialsSecret:
291+
name: simple-admin-credentials
292+
---
293+
apiVersion: nifi.stackable.tech/v1alpha1
294+
kind: NifiCluster
295+
metadata:
296+
name: simple-nifi
297+
spec:
298+
image:
299+
productVersion: 2.6.0
300+
clusterConfig:
301+
authentication:
302+
- authenticationClass: simple-nifi-users
303+
sensitiveProperties:
304+
keySecret: nifi-sensitive-property-key
305+
autoGenerate: true
306+
nodes:
307+
roleGroups:
308+
default:
309+
replicas: 1
310+
config:
311+
# Run NiFi nodes in the workloads pool
312+
affinity:
313+
nodeSelector:
314+
nodepool: stackable-workloads
315+
----
316+
317+
Apply it to Kubernetes:
318+
319+
[source,console]
320+
----
321+
$ kubectl apply -f nifi.yaml
322+
----
323+
324+
Then take a look at the pods running on nodes with the label `nodepool: stackable-workloads`:
325+
326+
[source,bash]
327+
----
328+
WORKLOADS_NODEPOOL=$(kubectl get nodes -l nodepool=stackable-workloads -o jsonpath="{.items[*].metadata.name}" | tr ' ' ',')
329+
echo "Nodes in workloads pool: $WORKLOADS_NODEPOOL\n"
330+
kubectl get pods -o json | jq --raw-output --arg nodepool "$WORKLOADS_NODEPOOL" '.items[] | .metadata.name as $podname | .spec.nodeName as $nodename | select($nodename | IN($nodepool | split(",")[])) | $podname'
331+
----
332+
333+
You should see similar output as last time, but now with the NiFi pod.
334+
335+
[source]
336+
---
337+
Nodes in workloads pool: stackable-worker3,stackable-worker4
338+
339+
listener-operator-csi-node-driver-lv5r4
340+
listener-operator-csi-node-driver-vdzsq
341+
secret-operator-csi-node-driver-d8sqw
342+
secret-operator-csi-node-driver-zkrv6
343+
simple-nifi-node-default-0
344+
---
345+
346+
== Cleanup
347+
348+
Once done, you can delete the KinD cluster like so:
349+
350+
[source,bash]
351+
----
352+
kind delete cluster --name stackable
353+
----

0 commit comments

Comments
 (0)