New components remote.kubernetes.secret and remote.kubernetes.configmap #4854

captncraig · 2023-08-17T17:20:17Z

Implements #2976

rfratto · 2023-08-17T18:28:37Z

docs/sources/flow/reference/components/remote.kubernetes.secret.md

+`poll_frequency` | `duration` | Frequency to poll the kubernetes API. | `"1m"` | no
+`poll_timeout` | `duration` | Timeout when polling the kubernetes API.. | `"15s"` | no


Should these components do a watch call against the Kubernetes API instead of a poll?

I am worried about the number of connections we'd take up to do that. Each component would have its own watch for each agent, or we combine internally and have to manage either a watch per namespace or an overly broad watch on the cluster.

For something that is not expected to change at high frequency, I thought this was a good trade off.

Mention that some managed Kubernetes services also have some lower rate limits or throttles.

Maybe a component can be watch multiple configmaps/secrets by an selector.

That's true, but that would also add significant complexity to the implementation here, as well as to what the user needs to understand about it. There is always an alternative to mount things directly to the agent pod if such limits are a serious problem.

There are use cases where we may want more "dynamic" loading of config maps (like for dynamically loading flow modules from configmaps with a certain selector), but for now we don't really have the river features (map/reduce) needed to take advantage of that. remote.kubernetes.configmaps["configmap_name"]["key_name"] could be possible, but I'm not a huge fan at present. I'd rather get this simpler version out, and go from there.

clayton-cornell

Minor spelling and phrase corrections

docs/sources/flow/reference/components/remote.kubernetes.configmap.md

docs/sources/flow/reference/components/remote.kubernetes.secret.md

…gmap.md Co-authored-by: Clayton Cornell <[email protected]>

…t.md Co-authored-by: Clayton Cornell <[email protected]>

…gmap.md Co-authored-by: Clayton Cornell <[email protected]>

…t.md Co-authored-by: Clayton Cornell <[email protected]>

…gmap.md Co-authored-by: Clayton Cornell <[email protected]>

…t.md Co-authored-by: Clayton Cornell <[email protected]>

…mote_k8s

spartan0x117 · 2023-09-14T16:40:30Z

component/remote/kubernetes/kubernetes.go

+	if args.PollFrequency <= 0 {
+		return fmt.Errorf("poll_frequency must be greater than 0")
+	}
+	return nil


Should there also be a check that the PollTimeout is > 0? Otherwise it will always cancel the request

docs/sources/flow/reference/components/remote.kubernetes.secret.md

docs/sources/flow/reference/components/remote.kubernetes.configmap.md

docs/sources/flow/reference/components/remote.kubernetes.secret.md

component/remote/kubernetes/kubernetes.go

docs/sources/flow/reference/components/remote.kubernetes.configmap.md

docs/sources/flow/reference/components/remote.kubernetes.secret.md

…gmap.md Co-authored-by: Mischa Thompson <[email protected]>

…t.md Co-authored-by: Mischa Thompson <[email protected]>

…gmap.md Co-authored-by: Mischa Thompson <[email protected]>

Co-authored-by: Mischa Thompson <[email protected]>

…gmap.md Co-authored-by: Mischa Thompson <[email protected]>

…t.md Co-authored-by: Mischa Thompson <[email protected]>

Co-authored-by: Mischa Thompson <[email protected]>

…t.md Co-authored-by: Mischa Thompson <[email protected]>

spartan0x117

LGTM!

clayton-cornell

Docs are OK as-is.

captncraig added 6 commits August 10, 2023 09:56

initial work

7811445

Merge branch 'main' into cmp_remote_k8s

2ddf059

move to subdir

da27c63

cleanup

07c0c9d

add docs

48588df

add all files

36d13b7

captncraig requested review from a team and clayton-cornell as code owners August 17, 2023 17:20

lint

6bbcc7b

rfratto reviewed Aug 17, 2023

View reviewed changes

clayton-cornell reviewed Aug 25, 2023

View reviewed changes

captncraig and others added 19 commits August 28, 2023 12:43

Update docs/sources/flow/reference/components/remote.kubernetes.confi…

13a25b8

…gmap.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.confi…

cab2dfa

…gmap.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.confi…

889c17f

…gmap.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.secre…

8c3dbc5

…t.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.secre…

183359e

…t.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.secre…

854c260

…t.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.secre…

1f15a59

…t.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.secre…

b13bc6a

…t.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.confi…

c15604a

…gmap.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.confi…

7cf1e78

…gmap.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.confi…

6812df9

…gmap.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.secre…

6a2400a

…t.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.confi…

6620e9b

…gmap.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.confi…

ba13c0f

…gmap.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.secre…

8f65945

…t.md Co-authored-by: Clayton Cornell <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.secre…

c7e7fa8

…t.md Co-authored-by: Clayton Cornell <[email protected]>

Merge branch 'main' into cmp_remote_k8s

28adfa1

Merge branch 'cmp_remote_k8s' of github.com:grafana/agent into cmp_re…

3fa70c4

…mote_k8s

switch to correct package after river move

908606e

captncraig added 3 commits September 8, 2023 13:47

changelog

c91f4aa

Merge branch 'main' into cmp_remote_k8s

1a972a9

Merge branch 'main' into cmp_remote_k8s

7458ded

spartan0x117 reviewed Sep 14, 2023

View reviewed changes

captncraig and others added 11 commits September 14, 2023 13:37

Update docs/sources/flow/reference/components/remote.kubernetes.confi…

f7e3cde

…gmap.md Co-authored-by: Mischa Thompson <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.secre…

e0b7562

…t.md Co-authored-by: Mischa Thompson <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.confi…

bbe8651

…gmap.md Co-authored-by: Mischa Thompson <[email protected]>

Update component/remote/kubernetes/kubernetes.go

4a1a9bb

Co-authored-by: Mischa Thompson <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.confi…

f8a91bf

…gmap.md Co-authored-by: Mischa Thompson <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.secre…

9e1cb10

…t.md Co-authored-by: Mischa Thompson <[email protected]>

Update component/remote/kubernetes/kubernetes.go

31b92b4

Co-authored-by: Mischa Thompson <[email protected]>

Update docs/sources/flow/reference/components/remote.kubernetes.secre…

1cac49b

…t.md Co-authored-by: Mischa Thompson <[email protected]>

add test, make timeout optional, and enforced

9834560

require positive timeout

ecd414a

newline

f27bcf4

spartan0x117 approved these changes Sep 14, 2023

View reviewed changes

clayton-cornell approved these changes Sep 14, 2023

View reviewed changes

petewall mentioned this pull request Sep 14, 2023

Allow basicAuth username/password to be sourced from secrets grafana/k8s-monitoring-helm#81

Closed

captncraig merged commit 7ebf224 into main Sep 14, 2023
6 checks passed

captncraig deleted the cmp_remote_k8s branch September 14, 2023 21:56

rfratto mentioned this pull request Sep 18, 2023

Flow proposal: remote.kubernetes.configmap and remote.kubernetes.secret #2976

Closed

clayton-cornell added the type/docs Docs Squad label across all Grafana Labs repos label Sep 18, 2023

github-actions bot added the frozen-due-to-age Locked due to a period of inactivity. Please open new issues or PRs if more discussion is needed. label Feb 22, 2024

github-actions bot locked as resolved and limited conversation to collaborators Feb 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New components remote.kubernetes.secret and remote.kubernetes.configmap #4854

New components remote.kubernetes.secret and remote.kubernetes.configmap #4854

captncraig commented Aug 17, 2023 •

edited

Loading

rfratto Aug 17, 2023

captncraig Aug 17, 2023 •

edited

Loading

jkroepke Aug 19, 2023 •

edited

Loading

captncraig Aug 21, 2023

clayton-cornell left a comment

spartan0x117 Sep 14, 2023

spartan0x117 left a comment

clayton-cornell left a comment

		`poll_frequency` \| `duration` \| Frequency to poll the kubernetes API. \| `"1m"` \| no
		`poll_timeout` \| `duration` \| Timeout when polling the kubernetes API.. \| `"15s"` \| no

New components remote.kubernetes.secret and remote.kubernetes.configmap #4854

New components remote.kubernetes.secret and remote.kubernetes.configmap #4854

Conversation

captncraig commented Aug 17, 2023 • edited Loading

rfratto Aug 17, 2023

Choose a reason for hiding this comment

captncraig Aug 17, 2023 • edited Loading

Choose a reason for hiding this comment

jkroepke Aug 19, 2023 • edited Loading

Choose a reason for hiding this comment

captncraig Aug 21, 2023

Choose a reason for hiding this comment

clayton-cornell left a comment

Choose a reason for hiding this comment

spartan0x117 Sep 14, 2023

Choose a reason for hiding this comment

spartan0x117 left a comment

Choose a reason for hiding this comment

clayton-cornell left a comment

Choose a reason for hiding this comment

captncraig commented Aug 17, 2023 •

edited

Loading

captncraig Aug 17, 2023 •

edited

Loading

jkroepke Aug 19, 2023 •

edited

Loading