WIP: k8s pods crawling interface, functions and reference implementation

[tatsuhirochiba]

This PR is a part of #245 , which includes several functions, interfaces and an example redis pod crawling plugin.

The modules are working well on our local k8s environment, but I have not added test cases yet.

[codecov-io]

Codecov Report

Merging #246 into master will decrease coverage by -1.2%.
The diff coverage is 40.69%.

@@            Coverage Diff            @@
##           master     #246     +/-   ##
=========================================
- Coverage   90.34%   89.15%   -1.2%     
=========================================
  Files          98      101      +3     
  Lines        3492     3577     +85     
=========================================
+ Hits         3155     3189     +34     
- Misses        337      388     +51

Impacted Files	Coverage Δ
crawler/crawlmodes.py	100% <ø> (ø)	✅
crawler/icrawl_plugin.py	100% <100%> (ø)	✅
crawler/k8s_pods_crawler.py	33.33% <33.33%> (ø)
crawler/crawler.py	88.13% <33.33%> (-2.94%)	❌
...er/plugins/applications/redis/redis_pod_crawler.py	38.46% <38.46%> (ø)
crawler/k8s_client.py	43.75% <43.75%> (ø)
crawler/plugins_manager.py	83.96% <50%> (-2.91%)	❌

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 515ac59...574202f. Read the comment docs.

[sahilsuneja1]

Haven't looked much in detail yet, but could it be possible to tie this with existing --enivornment kubernetes : https://github.com/cloudviz/agentless-system-crawler/blob/master/crawler/plugins/environments/kubernetes_environment.py
Adding @nadgowdas

[nadgowdas]

@tatsuhirochiba, do we need a separate crawl mode for k8s objects ?
we have added k8s environment plugin for crawler, in which for every crawled container, we collect their labels which includes their pod, namespace information. So user can collect/aggregate metrics about their pods/namespaces/cluster from container crawled data.

adding new crawl mode for every k8s object doesn't seem right. For example, if user wants to crawl for 'replication controller' (multiple pods) or 'service' we would need to add new crawl mode for that?

Can we add this k8s as a plugin? here, when crawling containers, we could use k8s downward API to get pod information about that container and then get query pod status? We could think how to avoid duplicate pod queries for multiple containers in pod.

[tatsuhirochiba]

@sahilsuneja1 @nadgowdas Thanks for your very useful comments!
Actually I created this PR for discussion, and I know we need to consider more this PR and proposed k8s crawl mode before merging. I would like to improve this PR/design/implementation for crawler suitable one.

First, the motivation of the k8s crawl mode is to collect system metrics (e.g. cpu/memory etc) and app metrics (e.g. redis, nginx, etc.) from k8s managed containers themselves, not from k8s objects (e.g. Services/Deployments/Replication Controllers etc.) In this context, the gap between k8s and outcontainer mode seems not so big because their crawling target is common (i.e. container), so that using k8s environment plugin to change crawling behavior is very reasonable approach.

However, there exists several gaps, and probably they are not solved by using k8s environment plugin.

• docker inspect information is not enough for k8s pod crawling
  Docker inspect from k8s managed container does not include all of pod specific information actually. For example, I would like to use redis pod to explain the situation. Once running redis pod, two containers are launched in the host, one is redis container and another one is pause container. We can get pod name, pod namespace, pod uid etc. from Config.Labels in redis container’s docker inspect entry, but can not extract pod ip, pod label, pod resource version, etc from it. Pod label means user specified one like “app=redis” or “role=master” in deployment yaml file. Pod IP (i.e. container IP) is basically managed in Config.NetworkSettings entry, but the network information is attached to pause container’s entry only, so we need to search pause container that has same pod uid to know actual redis container IP.

• crawling pause container
  Basic outcontainer mode crawls all of containers (i.e. docker ps -q) on the host, so crawler also tries to get the metrics from pause container. I think we do not need pause container metrics at all.

• emit per Pod, not per container
  This is optional, but materializing crawled metrics output per Pod is useful.

[nadgowdas]

@tatsuhirochiba can you check this k8s downward API guide:
https://kubernetes.io/docs/user-guide/downward-api/

It is possible to query POD properties from within a container, that we can extend in (not necessarily k8s environment plugin, but ) separate k8s crawler plugin

This will help us identify gaps.

[sahilsuneja1]

Thanks @tatsuhirochiba for explaining the requirements and @nadgowdas for suggesting an alternative solution. Ideally, we would like to extend crawler via plugins instead of modifications to core, so if something does not fit the current plugin model, we change the plugin model hopefully :)