dataclients/kubernetes: add metadata route - zone aware option 2 #2845
Conversation
AlexanderYastrebov
force-pushed
the
dataclients/kubernetes/metadata-route
branch
2 times, most recently
from
b773b22
to
fa772eb
Compare
|
I don't like it. Maybe write first a doc for this and why we need metadata. |
AlexanderYastrebov
force-pushed
the
dataclients/kubernetes/metadata-route
branch
from
113fdb0
to
fbae58c
Compare
Thank you for having a look. The idea is to use pod and node names in error logging and tracing. Zone could also be used to filter endpoints by zone before applying loadbalancing algorithm. Pod uid could be used to detect pod IP address reuse and e.g. reset endpoint detected time.
The biggest upside is that metadata about routing table is transferred along with the routing table itself so there is no other API or source of inconsistency.
This was just a proof-of-concept. I changed the format now to use https://en.wikipedia.org/wiki/Data_URI_scheme
We can decide what data of all we pass as metadata. We can start with pod name only and extend it later.
JSON is extensible so we can add new fields later and separatetly on producer and consumer side. It is also a text format and there are a lot of tools to process it. We could use binary format but the data size is <200kB (and <30kB gzipped) so I do not think its worth the inconvenience. |
AlexanderYastrebov
force-pushed
the
dataclients/kubernetes/metadata-route
branch
from
fbae58c
to
935cdd6
Compare
| @@ -24,3 +24,10 @@ type secret struct { | |||
| type secretList struct { | |||
| Items []*secret `json:"items"` | |||
| } | |||
|
|
|||
| type objectReference struct { | |||
| Kind string `json:"kind"` | |||
There was a problem hiding this comment.
this is static and always "Pod", no need to store nor pass
There was a problem hiding this comment.
We don't know this and its not documented so I check kind. See https://pkg.go.dev/k8s.io/api/discovery/v1#Endpoint
| Kind string `json:"kind"` | ||
| Name string `json:"name"` | ||
| Namespace string `json:"namespace"` | ||
| Uid string `json:"uid"` |
There was a problem hiding this comment.
This is for unmarshalling k8s API response.
uid can be used to detect pod rotation with the same name, e.g. from StatefulSet.
AlexanderYastrebov
force-pushed
the
dataclients/kubernetes/metadata-route
branch
from
935cdd6
to
d214c11
Compare
| @@ -1308,11 +1323,10 @@ func (p *Proxy) errorResponse(ctx *context, err error) { | |||
| uri = uri[:i] | |||
| } | |||
| logFunc( | |||
| `%s after %v, route %s with backend %s %s%s, status code %d: %v, remote host: %s, request: "%s %s %s", host: %s, user agent: "%s"`, | |||
| `%s after %v, route %s with backend %s%s, status code %d: %v, remote host: %s, request: "%s %s %s", host: %s, user agent: "%s"`, | |||
There was a problem hiding this comment.
This mess should be refactored into structured logging
AlexanderYastrebov
force-pushed
the
dataclients/kubernetes/metadata-route
branch
2 times, most recently
from
59373aa
to
4de9f1a
Compare
Adds a special route with a predefined id `kube__metadata` that contains `False` predicate, no filters and endpoints metadata. Endpoints metadata is encoded as JSON via [data URI scheme](https://en.wikipedia.org/wiki/Data_URI_scheme) into the route backend address field. Example eskip: ``` kube__metadata: False() -> "data:application/json;base64,eyJhZGRyZXNzZXMiO..."; ... ``` A special route pre-processor detects and removes this route, decodes metadata and stores it as EndpointRegistry metrics. The endpoint metrics then used to obtain endpoint node and pod names for tracing and logging. TODO: - [ ] add flag to enable metadata route - [ ] tests Fixes #1559 Signed-off-by: Alexander Yastrebov <[email protected]>
AlexanderYastrebov
force-pushed
the
dataclients/kubernetes/metadata-route
branch
from
4de9f1a
to
5b46917
Compare
szuecs
changed the title
Adds a special route with a predefined id
kube__metadatathat contains
Falsepredicate, no filters and endpoints metadata.Endpoints metadata is encoded as JSON via data URI scheme
into the route backend address field.
Example eskip:
A special route pre-processor detects and removes this route, decodes metadata
and stores it as EndpointRegistry metrics.
The endpoint metrics then used to obtain endpoint zone, node and pod names
for tracing and logging.
TODO:
Fixes #1559