You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We now support any container engine CRI compliant in kata-monitor.
Update documentation to reflect it.
Fixes: kata-containers#980
Signed-off-by: Francesco Giudici <[email protected]>
Copy file name to clipboardExpand all lines: docs/design/kata-2-0-metrics.md
+54-37Lines changed: 54 additions & 37 deletions
Original file line number
Diff line number
Diff line change
@@ -1,21 +1,21 @@
1
1
# Kata 2.0 Metrics Design
2
2
3
-
Kata implement CRI's API and support[`ContainerStats`](https://github.com/kubernetes/kubernetes/blob/release-1.18/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto#L101) and [`ListContainerStats`](https://github.com/kubernetes/kubernetes/blob/release-1.18/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto#L103) interfaces to expose containers metrics. User can use these interface to get basic metrics about container.
3
+
Kata implements CRI's API and supports[`ContainerStats`](https://github.com/kubernetes/kubernetes/blob/release-1.18/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto#L101) and [`ListContainerStats`](https://github.com/kubernetes/kubernetes/blob/release-1.18/staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto#L103) interfaces to expose containers metrics. User can use these interfaces to get basic metrics about containers.
4
4
5
-
But unlike`runc`, Kata is a VM-based runtime and has a different architecture.
5
+
Unlike`runc`, Kata is a VM-based runtime and has a different architecture.
6
6
7
-
## Limitations of Kata 1.x and the target of Kata 2.0
7
+
## Limitations of Kata 1.x and target of Kata 2.0
8
8
9
9
Kata 1.x has a number of limitations related to observability that may be obstacles to running Kata Containers at scale.
10
10
11
-
In Kata 2.0, the following components will be able to provide more details about the system.
11
+
In Kata 2.0, the following components will be able to provide more details about the system:
12
12
13
13
- containerd shim v2 (effectively `kata-runtime`)
14
14
- Hypervisor statistics
15
15
- Agent process
16
16
- Guest OS statistics
17
17
18
-
> **Note**: In Kata 1.x, the main user-facing component was the runtime (`kata-runtime`). From 1.5, Kata then introduced the Kata containerd shim v2 (`containerd-shim-kata-v2`) which is essentially a modified runtime that is loaded by containerd to simplify and improve the way VM-based containers are created and managed.
18
+
> **Note**: In Kata 1.x, the main user-facing component was the runtime (`kata-runtime`). From 1.5, Kata introduced the Kata containerd shim v2 (`containerd-shim-kata-v2`) which is essentially a modified runtime that is loaded by containerd to simplify and improve the way VM-based containers are created and managed.
19
19
>
20
20
> For Kata 2.0, the main component is the Kata containerd shim v2, although the deprecated `kata-runtime` binary will be maintained for a period of time.
21
21
>
@@ -25,14 +25,15 @@ In Kata 2.0, the following components will be able to provide more details about
25
25
26
26
Kata 2.0 metrics strongly depend on [Prometheus](https://prometheus.io/), a graduated project from CNCF.
27
27
28
-
Kata Containers 2.0 introduces a new Kata component called `kata-monitor` which is used to monitor the other Kata components on the host. It's the monitor interface with Kata runtime, and we can do something like these:
28
+
Kata Containers 2.0 introduces a new Kata component called `kata-monitor` which is used to monitor the Kata components on the host. It's shipped with the Kata runtime to provide an interface to:
29
29
30
30
- Get metrics
31
31
- Get events
32
32
33
-
In this document we will cover metrics only. And until now it only supports metrics function.
33
+
At present, `kata-monitor` supports retrieval of metrics only: this is what will be covered in this document.
34
34
35
-
This is the architecture overview metrics in Kata Containers 2.0.
35
+
36
+
This is the architecture overview of metrics in Kata Containers 2.0:
@@ -45,38 +46,38 @@ For a quick evaluation, you can check out [this how to](../how-to/how-to-set-pro
45
46
46
47
### Kata monitor
47
48
48
-
`kata-monitor`is a management agent on one node, where many Kata containers are running. `kata-monitor`'s work include:
49
+
The `kata-monitor` management agent should be started on each node where the Kata containers runtime is installed. `kata-monitor` will:
49
50
50
-
> **Note**: node is a single host system or a node in K8s clusters.
51
+
> **Note**: a *node* running Kata containers will be either a single host system or a worker node belonging to a K8s cluster capable of running Kata pods.
51
52
52
-
- Aggregate sandbox metrics running on this node, and add`sandbox_id` label
53
-
-As a Prometheus target, all metrics from Kata shim on this node will be collected by Prometheus indirectly. This can easy the targets count in Prometheus, and also need not to expose shim's metrics by `ip:port`
53
+
- Aggregate sandbox metrics running on the node, adding the`sandbox_id` label to them.
54
+
-Expose a new Prometheus target, allowing all node metrics coming from the Kata shim to be collected by Prometheus indirectly. This simplifies the targets count in Prometheus and avoids exposing shim's metrics by `ip:port`.
54
55
55
-
Only one `kata-monitor` process are running on one node.
56
+
Only one `kata-monitor` process runs in each node.
56
57
57
-
`kata-monitor`is using a different communication channel other than that `conatinerd` communicating with Kata shim, and Kata shim listen on a new socket address for communicating with`kata-monitor`.
58
+
`kata-monitor`uses a different communication channel than the one used by the container engine (`containerd`/`CRI-O`) to communicate with the Kata shim. The Kata shim exposes a dedicated socket address reserved to`kata-monitor`.
58
59
59
-
The way `kata-monitor` get shim's metrics socket file(`monitor_address`) like that `containerd` get shim address. The socket is an abstract socket and saved as file `abstract` with the same directory of `address` for `containerd`.
60
+
The shim's metrics socket fileis created under the virtcontainers sandboxes directory, i.e. `vc/sbs/${PODID}/shim-monitor.sock`.
60
61
61
-
> **Note**: If there is no Prometheus server is configured, i.e., there is no scrape operations, `kata-monitor` will do nothing initiative.
62
+
> **Note**: If there is no Prometheus server configured, i.e., there are no scrape operations, `kata-monitor` will not collect any metrics.
@@ -93,33 +94,49 @@ The `metrics` field is Prometheus encoded content. This can avoid defining a fix
93
94
94
95
### Performance and overhead
95
96
96
-
Metrics should not become the bottleneck of system, downgrade the performance, and run with minimal overhead.
97
+
Metrics should not become a bottleneck for the system or downgrade the performance: they should run with minimal overhead.
97
98
98
99
Requirements:
99
100
100
101
* Metrics **MUST** be quick to collect
101
-
* Metrics **MUST** be small.
102
+
* Metrics **MUST** be small
102
103
* Metrics **MUST** be generated only if there are subscribers to the Kata metrics service
103
104
* Metrics **MUST** be stateless
104
105
105
-
In Kata 2.0, metrics are collected mainly from `/proc` filesystem, and consumed by Prometheus, based on a pull mode, that is mean if there is no Prometheus collector is running, so there will be zero overhead if nobody cares the metrics.
106
+
In Kata 2.0, metrics are collected only when needed (pull mode), mainly from the `/proc` filesystem, and consumed by Prometheus. This means that if the Prometheus collector is not running (so no one cares about the metrics) the overhead will be zero.
107
+
108
+
The metrics service also doesn't hold any metrics in memory.
106
109
107
-
Metrics service also doesn't hold any metrics in memory.
*Metrics size*: response size of one Prometheus scrape request.
119
+
120
+
It's easy to estimate the size of one metrics fetch request issued by Prometheus.
121
+
The formula to calculate the expected size when no gzip compression is in place is:
122
+
9 + (144 - 9) * `number of kata sandboxes`
123
+
124
+
Prometheus supports `gzip compression`. When enabled, the response size of each request will be smaller:
125
+
2 + (10 - 2) * `number of kata sandboxes`
126
+
127
+
**Example**
128
+
We have 10 sandboxes running on a node. The expected size of one metrics fetch request issued by Prometheus against the kata-monitor agent running on that node will be:
129
+
9 + (144 - 9) * 10 = **1.35M**
114
130
115
-
*Metrics size*: Response size of one Prometheus scrape request.
131
+
If `gzip compression` is enabled:
132
+
2 + (10 - 2) * 10 = **82K**
116
133
117
-
It's easy to estimated that if there are 10 sandboxes running in the host, the size of one metrics fetch request issued by Prometheus will be about to 9 + (144 - 9) * 10 = 1.35M (not `gzipped`) or 2 + (10 - 2) * 10 = 82K (`gzipped`). Of course Prometheus support `gzip` compression, that can reduce the response size of every request.
134
+
#### Metrics delay ####
118
135
119
136
And here is some test data:
120
137
121
-
- End-to-end (from Prometheus server to `kata-monitor` and `kata-monitor` write response back): 20ms(avg)
122
-
- Agent(RPC all from shim to agent): 3ms(avg)
138
+
- End-to-end (from Prometheus server to `kata-monitor` and `kata-monitor` write response back): **20ms**(avg)
139
+
- Agent(RPC all from shim to agent): **3ms**(avg)
123
140
124
141
Test infrastructure:
125
142
@@ -128,13 +145,13 @@ Test infrastructure:
128
145
129
146
**Scrape interval**
130
147
131
-
Prometheus default `scrape_interval` is 1 minute, and usually it is set to 15s. Small `scrape_interval`will cause more overhead, so user should set it on monitor demand.
148
+
Prometheus default `scrape_interval` is 1 minute, but it is usually set to 15 seconds. A smaller `scrape_interval`causes more overhead, so users should set it depending on their monitoring needs.
132
149
133
150
## Metrics list
134
151
135
-
Here listed is all supported metrics by Kata 2.0. Some metrics is dependent on guest kernels in the VM, so there may be some different by your environment.
152
+
Here are listed all the metrics supported by Kata 2.0. Some metrics are dependent on the VM guest kernel, so the available ones may differ based on the environment.
136
153
137
-
Metrics is categorized by component where metrics are collected from and for.
154
+
Metrics are categorized by the component from/for which the metrics are collected.
138
155
139
156
*[Metric types](#metric-types)
140
157
*[Kata agent metrics](#kata-agent-metrics)
@@ -145,15 +162,15 @@ Metrics is categorized by component where metrics are collected from and for.
> * Labels here are not include `instance` and `job` labels that added by Prometheus.
165
+
> * Labels here do not include the `instance` and `job` labels added by Prometheus.
149
166
> * Notes about metrics unit
150
167
> *`Kibibytes`, abbreviated `KiB`. 1 `KiB` equals 1024 B.
151
-
> * For some metrics (like network devices statistics from file `/proc/net/dev`), unit is depend on label( for example `recv_bytes` and `recv_packets`are having different units).
152
-
> * Most of these metrics is collected from `/proc` filesystem, so the unit of metrics are keeping the same unit as `/proc`. See the `proc(5)` manual page for further details.
168
+
> * For some metrics (like network devices statistics from file `/proc/net/dev`), unit depends on label( for example `recv_bytes` and `recv_packets`have different units).
169
+
> * Most of these metrics are collected from the `/proc` filesystem, so the unit of each metric matches the unit of the relevant `/proc` entry. See the `proc(5)` manual page for further details.
153
170
154
171
### Metric types
155
172
156
-
Prometheus offer four core metric types.
173
+
Prometheus offers four core metric types.
157
174
158
175
- Counter: A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase.
0 commit comments