docker.inspect return empty value for 3rd level JSON property

[alloran]

Hi!

I try to use docker.inspect metric to get container health status, but it returns empty string if required value is json property, not json array item.

$ zabbix_get -s 127.0.0.1 -p 10050 -k 'docker.inspect[/3f211afa76aa,Config,Env,NODE_ENV=]'
development
$ zabbix_get -s 127.0.0.1 -p 10050 -k 'docker.inspect[/3f211afa76aa,State,Health]'
{"Status":"unhealthy","FailingStreak":0,"Log":[{"Start":"2018-03-28T10:43:04.805430365+06:00","End":"2018-03-28T10:43:05.200160407+06:00","ExitCode":137,"Output":"c"},{"Start":"2018-03-28T10:52:45.934850275+06:00","End":"2018-03-28T10:52:46.063728468+06:00","ExitCode":137,"Output":""}]}
$ zabbix_get -s 127.0.0.1 -p 10050 -k 'docker.inspect[/3f211afa76aa,State,Health,Status]'
< \n here >

Accroding to docs it should return unhealthy

$ docker info 
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 64
Server Version: 17.12.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.13.0-36-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.47GiB
Name: c6963sa
ID: VAJQ:VCHN:PKFK:U5UP:VTLR:L2J4:RR67:XDN7:QL46:UY5S:JNM6:6QCW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
HTTP Proxy: http://127.0.0.1:3128/
Username: alloran
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[jangaraj]

Hi Yevgeniy,

Try to troubleshoot your problem https://github.com/monitoringartist/zabbix-docker-monitoring#troubleshooting - you will see more details in the zabbix agent log file. There will be full Docker API response.

[alloran]

 28219:20180328:153917.512 Docker's socket query: GET /containers/3f211afa76aa/json HTTP/1.0
 28219:20180328:153917.513 Docker's socket response: { edited }
 28219:20180328:153917.514 Item [State] found in the received JSON object
 28219:20180328:153917.514 Array item: Status
 28219:20180328:153917.514 Item [State][Health][Status] found in the received JSON object: 
 28219:20180328:153917.514 zbx_module_docker_inspect_exec OK: 
 28219:20180328:153917.514 Sending back []

It seems like docker.inspect metric expects 4th parameter (from docs: par3 - optional name of 3rd level JSON property or selector of item in the JSON array) only as json array item selector, not json property

docker api output:

{
  "Id": "3f211afa76aa606f5e013329ccf3dd0347e7b76214e7ad3c192ec4c5ccdac499",
  "Created": "2018-03-28T04:42:03.806546139Z",
  "Path": "node",
  "Args": [
    "./bin/www"
  ],
  "State": {
    "Status": "exited",
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 1,
    "Error": "",
    "StartedAt": "2018-03-28T09:37:00.086091983Z",
    "FinishedAt": "2018-03-28T09:38:00.197739443Z",
    "Health": {
      "Status": "unhealthy",
      "FailingStreak": 0,
      "Log": [
        {
          "Start": "2018-03-28T10:43:04.805430365+06:00",
          "End": "2018-03-28T10:43:05.200160407+06:00",
          "ExitCode": 137,
          "Output": "c"
        },
        {
          "Start": "2018-03-28T10:52:45.934850275+06:00",
          "End": "2018-03-28T10:52:46.063728468+06:00",
          "ExitCode": 137,
          "Output": ""
        },
        {
          "Start": "2018-03-28T11:16:38.505969494+06:00",
          "End": "2018-03-28T11:16:38.611610528+06:00",
          "ExitCode": 137,
          "Output": "cur"
        },
        {
          "Start": "2018-03-28T15:38:00.086922197+06:00",
          "End": "2018-03-28T15:38:00.191713371+06:00",
          "ExitCode": 137,
          "Output": "curl"
        }
      ]
    }
  },
 < edited >
 }

2nd level property works as expected

$ zabbix_get -s 127.0.0.1 -p 10050 -k 'docker.inspect[/3f211afa76aa,State,StartedAt]'
2018-03-28T09:37:00.086091983Z

[jangaraj]

You are right. 3rd parameter needs improvement:

$ zabbix_get -s 127.0.0.1 -k docker.inspect[/988f5698cb4d,GraphDriver,Data,DeviceId]
<nothing here :( >

Also 4rd parameter can be usefull, e.g.:

$ zabbix_get -s 127.0.0.1 -k docker.inspect[/988f5698cb4d,NetworkSettings,Networks,bridge,IPAddress]
<nothing here :( >

Feel free to create PR.

[gold-dream-teacher]

Hi，I also encountered this problem.