hung on unmount operation #88
Comments
|
forcibly killing the |
|
... and this appears to confuse the system, which now has mounted two devices to the same point in the filesystem: |
|
second volume was created with these specs (should have been a new vol since it was not preexisting): |
|
rexray config: |
|
@jdef I believe the The message in the bottom of the log is showing that there is something that is still using the device that is trying to be detached. Is there any chance that the device and/or mount path is being accessed by another process on the system? Can you capture the results of |
|
@clintonskitson lsof results: |
|
@jdef Can you perform something like |
|
comes back empty On Mon, Apr 4, 2016 at 4:11 PM, Clinton Kitson [email protected]
James DeFelice |
|
@clintonskitson i'm going to try with an explicit mount under |
|
If it is empty, can you try an unmount of the path? |
|
appears to succeed: core@ip-10-0-0-142 ~ $ sudo umount /tmp/test-rexray-volume On Mon, Apr 4, 2016 at 4:14 PM, Clinton Kitson [email protected]
James DeFelice |
|
Can you replicate the problem where it hung before? It would have hung because there were open files. Technically, since the container that has the volume mounted and is using the files is done, there should not be any files open. I am curious to see if you could do a |
|
well, i just tried to launch a new task with a new volume instance but rexray is hanging on trying to mount it. this is the same slave that I just manually ran the the error I produced today was on a completely new cluster vs the error I first reported. so it can be reproduced. the containers i'm spinning up aren't touching any files in the volume, so i'm not sure what would be holding a handle to any files on it. |
|
@jdef Waiting for volume attachment to complete looks to be a EC2 based problem and might align to a bug they have in their Ubuntu OS image with the EBS driver. Can you reboot the instances and try again? |
|
FWIW this is coreos, not Ubuntu. I've rebooted the slave. Once rebooted, the task launched just fine (
i killed the task. and |
|
|
to be clear, once |
|
What is the coreos rev or AMI? I will try and reproduce. On Monday, April 4, 2016, James DeFelice [email protected] wrote:
|
|
looks like this AMI CoreOS-stable-766.5.0-hvm |
|
Can you review the devices that show up under /dev/disk/by-id? Curious if something is claiming for block devices as they appear. On Monday, April 4, 2016, James DeFelice [email protected] wrote:
|
|
core@ip-10-0-0-142 ~ $ ls -laF /dev/disk/by-uuid/ On Mon, Apr 4, 2016 at 11:34 PM, Clinton Kitson [email protected]
James DeFelice |
|
Ok looks normal, thanks. On Monday, April 4, 2016, James DeFelice [email protected] wrote:
|
|
so, it actually doesn't block indefinitely... the job I tried to start about 6 hours ago finally started about 20m ago. slave logs have this to say: immediately after this log output, my (formerly) blocked non-volume-using task was finally launched (actually it experienced a TASK_FAILED because it had started so long ago and marathon had since killed it, but ... it finally caught up to the world. marathon was then able to launch it again and that worked). |
|
rexray had something to say about the same time: |
|
@jdef With a fresh system can you try and leverage the Please replace
|
|
@jdef Can you also paste in the marathon application definition that you are using? |
|
Apparently we're running a forked version of rexray that tries to use IAM credentials instead of credentials coded into the configuration file. Upon further review it seems likely that the library code responsible for EC2 auth is failing to refresh credentials upon token expiration. That said, it's not clear to me that the credentials issue is directly related to hanging unmount commands, especially when I terminate a task so closely to when it's created -- given how easy it is to reproduce the issue, I'm betting that the credentials have not yet expired. The marathon spec I'm using depends on a development branch of marathon that's adding higher level support for external volumes (so this is subject to change prior to being merged into master): {
"id": "hello",
"instances": 1,
"cpus": 0.1,
"mem": 32,
"cmd": "/usr/bin/tail -f /dev/null",
"container": {
"type": "MESOS",
"volumes": [
{
"containerPath": "/tmp/test-rexray-volume",
"persistent": {
"size": 100,
"name": "jdef-test-vol13115",
"provider": "external",
"options": { "external/driver": "rexray" }
},
"mode": "RW"
}
]
},
"upgradeStrategy": {
"minimumHealthCapacity": 0,
"maximumOverCapacity": 0
}
}To test the mount/unmount commands manually I'll need to spin up a fresh cluster and execute the commands before the initially detected IAM creds expire. |
|
hm... IAM problems have been resolved and the hung unmount problem persists: |
|
using filesystem/linux isolator in conjunction with this isolator fixes the problem. without the filesystem/linux isolator the host mountns is contaminated with the mounts added to the container. core team says that the hosts /tmp was being polluted by the container w/o the filesystem/linux isolator |
|
I've left this issue open because this dependency on the linux/filesystem isolator should be documented |
|
@jdef Doing some more testing on our side. To me this doesn't make sense that this would be required. |
|
@jdef So the troubling thing here is that I believe I simulated relevant options by supplying So the question I am moving towards is whether there is some type of race condition introduced in a newer rev of Mesos. The failure of the detach is based on things not being cleaned up appropriately. If there was logic that changed on the cleanup side in 0.28+, then it may be interfering with our ability to properly remove mounts to enable the detach process to complete. |
|
one of the symptoms is that when a container path mount is created /tmp (in our test system) within the container mountns it's propagating back to the hosts mountns. the container mountns is destroyed upon container teardown but the mount still exists in the host mountns. rexray then tries to detach the volume but can't. our core team tells me this is because without the linux/fs isolator, /tmp exists in a shared mountns. and they also tell me that there is no other valid workaround for this problem -- that a container mountns must be properly isolated to avoid mounts propagating back to the host mountns. @clintonskitson are you saying that DVD isolator already prevents such leakage? /cc @jieyu |
|
@cantbewong @jdef Having a mountpath leak from a container to the hosts mount doesn't sound right. Is there any documentation to point us to online that discusses this? Can you try this on an Ubuntu 14.04 based host? |
|
Ubuntu 14.04 does not have this issue because by default, all mounts are private. Centos7/fedora23/coreos will have this issue because by default, all mounts are shared. |
|
@cantbewong Is it possible that when we leverage the fileystem isolator code from Mesos that we aren't invoking it as a private mount? I could see this being the problem if |
|
Assuming the mount being discussed is the bind mount used to isolate a container: the mount command is queued as follows No efforts are made to specify private vs shared so if RedHat and Ubuntu differ as to default, the private vs. shared state of the bind mount will differ. I found documentation that indicates that the RedHat default is private The Fedora documentation refers to the kernel documentation which also indicates a default of private If this is the root cause of this issue, an explicit call to declare the mount private would be useful Making this call should be harmless, other than adding the time it takes to execute, so I will put it in the next version of the isolator. |
|
I think @jieyu expressed concerns that not all versions of On Tue, Apr 12, 2016 at 7:45 PM, Steve Wong [email protected]
James DeFelice |
|
@jdef can you provide a suggestion for a supported OS list? |
|
@jdef @cantbewong the problem regarding the make-rslave|make-rprivate issue is described here in this Mesos ticket. It does not work properly on ubuntu 14.04 |
|
The docker solution here is within a mount package they have that calls the On Thursday, April 14, 2016, Jie Yu [email protected] wrote:
|
|
Same issue Apr 15 00:11:57 10.1.21.50 rexray[1747]: time="2017-04-15T00:11:57Z" level=info msg="waiting for volume detachment to complete" driverName=ec2 force=false moduleName=default-docker runAsync=false volumeID=vol-xxxxx docker ps etc wont response |
I created a task in Marathon, watched it come up, let it run for a few min, then suspended the app (sets instances to zero). I ended up with a hung unmount op on the slave the task was running on. The task appears as KILLED in mesos. I have a single slave node cluster. Now I can't launch a new task trying to use a (different) rexray vol. Attempting to do so results in a task stuck in STAGING. So this hung unmount op is blocking somewhere.
logs:
/proc/mounts:
The text was updated successfully, but these errors were encountered: