Registering private IPs thru mesos-consul

[Lax77]

I am provisioning containers on mesos framework using Calico (multi host networking solution). Calico assigns a separate IP for each of the container that I provision thru Calico from my defined IP pool.

Issue is when my container got registered thru mesos-consul, in my Consul I see the HOST IP of the container. Instead is there a way I can register container's IP thru mesos-consul and have Consul report Container's IP instead of Host IP?

Thanks
Lax

[ChrisAubuchon]

You can try changing the IP order via the --mesos-ip-order option. The default is netinfo,mesos,host.

[Lax77]

Ok. I was using mesos,host. Would preceding the order with docker would help?

[ChrisAubuchon]

You can try docker. netinfo might also work as well.

[Lax77]

Ok. Some how when I gave in like "--mesos-ip-order=netinfo,mesos,host" didnt work. Will try just having either of docker or netinfo. Thanks

[Lax77]

seeing weird behavior. When I change mesos-ip-order and re-deploy mesos-consul, I keep seeing the error 'Timed out waiting for initial ZK detection'. This causes the mesos-consul to keep re-deploying every 2 mins.

I tried pinging & telneting the ZK IP and port, that seems to work fine. Also I do see 'json.info_0000000002' file when 'ls /mesos' on ZKCli. Is there anything I am missing?

[ChrisAubuchon]

Can you post your mesos-consul config?

[Lax77]

{
"args": [
"--zk=zk://zookeeper.service.local:2181/mesos",
"--mesos-ip-order=docker,mesos,host"
],
"container": {
"type": "DOCKER",
"docker": {
"network": "HOST",
"image": "ciscocloud/mesos-consul"
}
},
"id": "mesos-consul",
"instances": 1,
"cpus": 0.1,
"mem": 128,
"labels": {
"task_type": "infra"
}
}

[ChrisAubuchon]

That looks good. Hmm. Does zookeeper.service.local resolve properly? The IP order processing occurs after the initial zookeeper connection so that shouldn't be an issue...

[Lax77]

Ok, It does resolve properly. I am re-spinning my cluster again just in case if it helps issue go away

[Lax77]

On re-spun system with the above app definition, it came up fine this time. But when I deploy a container in consul I still see Host IP though

[ChrisAubuchon]

That's not good. Can you post the task from Mesos's state.json? I'm wondering if the container IP is even in the state file...

[Lax77]

you mean content of json.info_0000000002 file?

[ChrisAubuchon]

No. curl http://mesos-master:5050/master/state.json. That's where mesos-consul gets its data from. I want to see if the container IP is in the mesos state data.

[Lax77]

Here is my task data seen in the state.json file

            {
                "id": "redis2.92a2aeca-194d-11e6-8438-467bb8096b30",
                "name": "redis2",
                "framework_id": "fa6b2f1e-0483-4dad-9867-d0794f6daf40-0000",
                "executor_id": "",
                "slave_id": "48c73e8e-81a4-423f-ae84-9a6265e6264c-S2",
                "state": "TASK_RUNNING",
                "resources": {
                    "cpus": 0.2,
                    "disk": 0,
                    "mem": 64,
                    "ports": "[4740-4740]"
                },
                "statuses": [
                    {
                        "state": "TASK_RUNNING",
                        "timestamp": 1463173182.03674,
                        "labels": [
                            {
                                "key": "Docker.NetworkSettings.IPAddress",
                                "value": ""
                            }
                        ],
                        "container_status": {
                            "network_infos": [
                                {
                                    "ip_address": "",
                                    "ip_addresses": [
                                        {
                                            "ip_address": ""
                                        }
                                    ]
                                }
                            ]
                        }
                    }
                ],
                "discovery": {
                    "visibility": "FRAMEWORK",
                    "name": "redis2",
                    "ports": {
                        "ports": [
                            {
                                "number": 4740,
                                "protocol": "tcp"
                            }
                        ]
                    }
                },
                "container": {
                    "type": "DOCKER",
                    "docker": {
                        "image": "redis",
                        "network": "HOST",
                        "privileged": false,
                        "parameters": [
                            {
                                "key": "net",
                                "value": "infrastructure"
                            }
                        ],
                        "force_pull_image": false
                    }
                }
            },

[Lax77]

Inside container I do see it has assigned IP such as 192.168.0.0, 192.168.0.64.

In the above definition ' "value": "infrastructure"' tells mesos to use custom network called 'infrastructure' that was created with Calico.

[Lax77]

In the case of calico provisioned container, as you seen above container ip is not available in mesos state data. But the container internally does have an IP, is there a way we can ask mesos-consul to pull IP from inside the container and have it registered?

[ChrisAubuchon]

Not really. mesos-consul runs on one node and would have to communicate with docker on each follower node.

[Lax77]

Finally able to get Calico's IP registered thru mesos-consul. Apparently version 0.28 of Mesos has a fix which makes calico's Ip available. Earlier I was running version 0.27.

Now one final question, since my current mesos-ip-order is docker,mesos,host. I end up seeing consul list internal IP for both Calico provisioned and Bridge provisioned. Is there any way can I have mesos-consul to register internal Ip for a specific set of containers and host ips for other set of containers?