cadvisor prevents docker from removing monitored containers?

[cornelius-keller]

Hi all, I have a problem using cadvisor on centos 7. When cadvisor is running, docker failes to remove other containers saying that the containers filesystem is busy. After stopping cadvisor is stopped container removal is working again.

I demostrated that in this gist: https://gist.github.com/cornelius-keller/0fd2d23b68ccd88c9328

I also included os version and docker info in the gist.

[rjnagal]

Thanks for reporting, @cornelius-keller

what cadvisor version are you running? Can you get host:port/validate for cadvisor?
Is this a temporary situation, or does the container fs stays busy till you delete cadvisor?

[cornelius-keller]

@rjnagal
Cadvisor version is:

[root@583274-app35 ~]# docker images
REPOSITORY                                      TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
docker.io/google/cadvisor                       latest              399ae3c46a0e        47 hours ago        19.89 MB
[root@583274-app35 ~]# 

This is a permanent situation. The container fs stays busy untill I delete cadvisor.

What do you mean by getting host:port/validate for cadvisor? Cadvisor was still running and responsive on the web ui if that is what you mean. Unfortunately I can't give you any public host port to validate as cadvisor is only exposed via a vpn.

[rjnagal]

Yeah, I just need the ouput from /validate endpoint on cadvisor UI. You can
scrub any data that's private in there. Thanks

On Fri, Jun 12, 2015 at 9:54 AM, Cornelius Keller notifications@github.com
wrote:

@rjnagal https://github.com/rjnagal
Cadvisor version is:

[root@583274-app35 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZEdocker.io/google/cadvisor latest 399ae3c46a0e 47 hours ago 19.89 MB
[root@583274-app35 ~]#

This is a permanent situation. The container fs stays busy untill I delete
cadvisor.

What do you mean by getting host:port/validate for cadvisor? Cadvisor was
still running and responsive on the web ui if that is what you mean.
Unfortunately I can't give you any public host port to validate as cadvisor
is only exposed via a vpn.

—
Reply to this email directly or view it on GitHub
#771 (comment).

[cornelius-keller]

Sorry was a long day, did not get that this was an endpoint. I added the output to the gist.

[gianlucaborello]

I am facing this same issue. Essentially, running cadvisor with --volume=/:/rootfs:ro causes other containers' devicemapper mounts to be mounted inside the cadvisor container, so they can't be properly destroyed when issuing docker rm on the target container as they will appear in use.

How can this be solved?

[hoeghh]

When i run it on Fedora 21, it works fine. But when i run it on Ubuntu 14.04.2 LTS I get the same error as described above.

Error response from daemon: Cannot destroy container xxx_jenkinsMaster_1230: Driver aufs failed to remove root filesystem 13b421d0458e740e42e5fa5ac1cb68f32638f0bc723d9ba16718955214d79b7d: rename /var/lib/docker/aufs/mnt/13b421d0458e740e42e5fa5ac1cb68f32638f0bc723d9ba16718955214d79b7d /var/lib/docker/aufs/mnt/13b421d0458e740e42e5fa5ac1cb68f32638f0bc723d9ba16718955214d79b7d-removing: device or resource busy

The main difference is, that Ubuntu uses AUFS, where Fedora uses Devicemapper. Maby thats the problem.

[shredder12]

@rjnagal I can confirm that this issue happens on Ubuntu trusty x64 with Doceker 1.8.1, cadvisor:latest and devicemapper.

'1cb6051b30a1' being the container ID.

# grep -l 1cb6051b30a1 /proc/*/mountinfo
/proc/1963/mountinfo
# ps aux | grep -i 1963
root      1963  1.9  0.8 588740 71688 ?        Ssl  Aug26  30:08 /usr/bin/cadvisor
root     14767  0.0  0.0  11744   952 pts/0    S+   00:56   0:00 grep --color=auto -i 1963

Please suggest a workaround for this.

[difro]

same here with CentOS + Docker 1.8.1(devicemapper)

Had to remove --volume=/:/rootfs:ro && --volume=/var/lib/docker:/var/lib/docker:ro

[vishh]

@rjnagal: Excepting disk usage calculation, cAdvisor does not poke at any
of these directories right?

On Fri, Aug 28, 2015 at 12:26 AM, Jihoon Chung notifications@github.com
wrote:

same here with CentOS + Docker 1.8.1(devicemapper)

Had to remove --volume=/:/rootfs:ro &&
--volume=/var/lib/docker:/var/lib/docker:ro

—
Reply to this email directly or view it on GitHub
#771 (comment).

[hourliert]

Same problem here with Ubuntu 14.04.3.

@difro solution works but cadvisor can't provide docker stats anymore.

Any workaround?

[rmetzler]

The last time I ran into this problem, I digged a little bit into the cAdvisor source code. I'm not 100% sure - because it was a few weeks ago - but this is essentially the gist:

If you use cAdvisor like it is shown in README.md you'll mount /var/lib/docker as a volume into the container. This will create dead containers.

The reason, cAdvisor wants you to mount /var/lib/docker is - as far as I could see - only to display a certain info that is only interesting for admins and should be known before hand.

[jimmidyson]

We should be able to get all info from a docker inspect rather than parsing the container config file. Seems like mounting /var/lib/docker is causing more trouble than it's worth.

[svenmueller]

we also encounter the same problem (cadvisor:lastest, ubuntu 14.04)