Using bridge network for nodes #484

tao12345666333 · 2019-05-05T09:28:38Z

ref: #408 (comment) and #408 (comment)

we can using bridge network for nodes. after this we can using hostname instead of IPs.
It will push #408 forward.

things need todo:

create bridge network
create nodes using bridge netwrok

~~- [ ] using hostname instead of IPs (config files.)~~

load-balancer config
delete network while deleting cluster.

k8s-ci-robot · 2019-05-05T09:28:43Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tao12345666333
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: bentheelder

If they are not already assigned, you can assign the PR to them by writing /assign @bentheelder in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

BenTheElder · 2019-05-05T10:43:38Z

#273 (comment)

BenTheElder · 2019-05-05T10:52:34Z

The custom bridge network's magical hostname resolution is causing coreDNS to detect a loop and fail.

tao12345666333 · 2019-05-05T10:59:30Z

It seems that I missed some useful information. 😯

neolit123 · 2019-05-05T13:00:31Z

The custom bridge network's magical hostname resolution is causing coreDNS to detect a loop and fail.

hm, the following is not an option:

editing the coredns Corefile to modify the forward or disable the coredns loop detection.

we need to think of a way to avoid the loop.
docker dns configuration flags:
https://docs.docker.com/v17.09/engine/userguide/networking/default_network/configure-dns/

/cc
/cc @aojea

aojea · 2019-05-05T14:27:00Z

The DNS loop seems caused because the docker DNS uses always the address 127.0.0.11 as quoted from the docker docs

Note: The DNS server is always at 127.0.0.11

and that´s the address configured on the node´s resolv.conf


nameserver 127.0.0.11
options ndots:0

CoreDNS documentation explains why such addresses causes a loop https://github.com/coredns/coredns/blob/master/plugin/loop/README.md

A common cause of forwarding loops in Kubernetes clusters is an interaction with a local DNS cache on the host node (e.g. systemd-resolved). For example, in certain configurations systemd-resolved will put the loopback address 127.0.0.53 as a nameserver into /etc/resolv.conf. Kubernetes (via kubelet) by default will pass this /etc/resolv.conf file to all Pods using the default dnsPolicy rendering them unable to make DNS lookups (this includes CoreDNS Pods). CoreDNS uses this /etc/resolv.conf as a list of upstreams to forward requests to. Since it contains a loopback address, CoreDNS ends up forwarding requests to itself.

It offers 3 solutions, but personally I only see feasible the 1st want mentioned modifying the kubelet yaml. Let me try to see what´s needed to implement it and come back to you.

In the other hand, I don´t know if creating a different bridge per cluster will break some functionality, I´m especially concerned about the multicluster/federation scenarios. I think it will be safer first to create a network for kind and then see if we need to move to a network per cluster

aojea · 2019-05-05T22:28:10Z

@tao12345666333 in order to keep backward compatibility and avoid the DNS crash we should:

Copy the resolv.conf from the host to /kind/resolv.conf per example filtering all the loopback addresses (this is what Docker is doing when using the default bridge https://docs.docker.com/v17.09/engine/userguide/networking/default_network/configure-dns/)

When creating the container’s /etc/resolv.conf, the daemon filters out all localhost IP address nameserver entries from the host’s original file.

Tell kubelet to use /kind/resolv.conf for Pods.
Seems kubeadm has a flag to configure this,,

tao12345666333 · 2019-05-05T22:33:27Z

@aojea Thank you for the information, I will try it later.

BenTheElder · 2019-05-05T22:57:15Z

does docker only rewrite etc/resolv.conf on container creation or continually? if the former, then we can just overwrite that?

otherwise we will have inconsistent DNS for system components.

aojea · 2019-05-06T06:16:50Z

does docker only rewrite etc/resolv.conf on container creation or continually? if the former, then we can just overwrite that?

otherwise we will have inconsistent DNS for system components.

if you overwrite the resolv.conf of the nodes you lose the Docker DNS and the host discovery. ~~I mean, there is no reason to move to user defined bridges~~

BenTheElder · 2019-05-06T06:18:55Z

Getting our own IP space for the nodes is still useful, and we could selectively merge the resolv.conf *From: *Antonio Ojea <[email protected]> *Date: *Sun, May 5, 2019, 23:16 *To: *kubernetes-sigs/kind *Cc: *Benjamin Elder, Review requested does docker only rewrite etc/resolv.conf on container creation or

…

continually? if the former, then we can just overwrite that? otherwise we will have inconsistent DNS for system components. if you overwrite the resolv.conf of the nodes you lose the Docker DNS and the host discovery. I mean, there is no reason to move to user defined bridges — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#484 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHADKZ5NYQQRUQB5UIWPSTPT7ENHANCNFSM4HK2Z64Q> .

tao12345666333 · 2019-05-06T10:37:12Z

This fix can work at my side.

pkg/cluster/nodes/create.go

aojea · 2019-05-06T12:23:23Z

~~It seems I misunderstood the use of the resolv-conf flag in kubelet~~

May 06 11:05:34 kind-worker kubelet[200]: E0506 11:05:34.447977 200 kuberuntime_manager.go:693] createPodSandbox for pod "weave-net-khpnb_kube-system(0ebd9f5c-6fec-11e9-8010-0242ac120002)" failed: open /kind/resolv.conf: no such file or directory

~~The pods are not able to boot because they can´t find the new resolv.conf file. I was thinking that the file was copied inside each pod~~
The resolv.conf file has to be copied in all nodes, currently is not available on the worker nodes

pkg/cluster/nodes/util.go

tao12345666333 · 2019-05-06T16:07:51Z

/test pull-kind-conformance-parallel

BenTheElder · 2019-05-06T16:21:33Z

In the other hand, I don´t know if creating a different bridge per cluster will break some functionality, I´m especially concerned about the multicluster/federation scenarios. I think it will be safer first to create a network for kind and then see if we need to move to a network per cluster

If you look at my strawman proposal, the idea is to make it configurable so federation can configure the same bridge for all.

pkg/cluster/nodes/create.go

BenTheElder · 2019-05-06T16:23:12Z

pkg/cluster/internal/kubeadm/config.go

@@ -237,6 +239,7 @@ evictionHard:
  nodefs.available: "0%"
  nodefs.inodesFree: "0%"
  imagefs.available: "0%"
+resolvConf: /kind/resolv.conf


I really don't like the idea that containerd and debugging tools will have a different resolv conf than kubelet.

as you said

we could selectively merge the resolv.conf

After considering it, I think the current practice is relatively simple. If we do resolv.conf merge, we need a lot of testing. Why don't we put it later? 🤔

Keep this practice for a while, then change it after the verification is passed.

I hope we can move forward. ❤️

this means other components have an inconsistent resolv conf and we're not actually solving the issue. if we're going to use hostnames we need kubelet and everything else to have the same lookup.

technically having a different resolv.conf doesn't mean that have a different lookup, just that the pods will not be able to resolv the nodes names, the other requests will use the same dns server, however I agree that reality is always different and we can be broken because docker changes on the DNS behaviour, that will not be surprising 😅

Docker daemon runs an embedded DNS server which provides DNS resolution among containers connected to the same user-defined network, so that these containers can resolve container names to IP addresses. If the embedded DNS server is unable to resolve the request, it will be forwarded to any external DNS servers configured for the container. To facilitate this when the container is created, only the embedded DNS server reachable at 127.0.0.11 will be listed in the container’s resolv.conf file.

I agree with tao here, I think that we should left until last the DNS exercise

+1 on merging the resolv.confs if feasable.

pkg/cluster/internal/create/actions/loadbalancer/loadbalancer.go

tao12345666333 · 2019-05-08T14:56:55Z

The most basic features of this PR expectation have been achieved. Includes:

create bridge network
create nodes using bridge netwrok
load-balancer config
delete network while deleting cluster.

By using the bridge network, we can use Docker's embedded DNS to help push #408 forward.

After this PR, we only need to modify /etc/kubernetes/pki/admin.conf in order to deal with #408. We no longer need to regenerate the certificate because the current certificate is still available(with this PR).

i.e.

root@moe-control-plane3:/etc/kubernetes# openssl x509 -in pki/apiserver.crt -noout -text                        
Certificate:                                                                                                     
    Data:                                                                                        
        Version: 3 (0x2)                                                                                         
        Serial Number: 4829775455184542867 (0x4306cf2fef2cdc93)                                                 
        Signature Algorithm: sha256WithRSAEncryption                                                             
        Issuer: CN = kubernetes                                    
        Validity                                                                                                 
            Not Before: May  8 13:55:37 2019 GMT                             
            Not After : May  7 13:56:57 2020 GMT                                                                 
        Subject: CN = kube-apiserver                                              
        Subject Public Key Info:                                                                                 
            Public Key Algorithm: rsaEncryption                                           
                RSA Public-Key: (2048 bit)                                                                      
                Modulus:                                                                                        
                    00:c3:cb:9e:9f:e5:d4:bf:c7:b1:a8:77:98:95:e5:  
                    (ignore)
                    fb:66:2a:d0:ab:b1:23:30:25:39:e5:91:91:6a:f6:
                    a9:15
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Subject Alternative Name:
                DNS:moe-control-plane3, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:k
ubernetes.default.svc.cluster.local, DNS:localhost, IP Address:10.96.0.1, IP Address:172.28.0.6, IP Address:172.2
8.0.2
    Signature Algorithm: sha256WithRSAEncryption
         70:65:07:9c:54:50:24:9f:59:59:8b:de:4f:60:c9:4e:47:5f:
         (ignore)

I want to know your thoughts. @BenTheElder @aojea @neolit123 Thanks.

aojea · 2019-05-08T15:16:27Z

Ben created a proposal, I think that only the labeling networks thing is missing

#273 (comment)

Strawman:

if the specified network exists, we use it without creating
if it doesn't exist, we create it, and label it as belonging to the cluster
on cluster delete, we list networks labeled with the cluster, and delete only those (so not just whatever the containers use, only if we labeled it)
we name this field / functionality somehow such that it is clear that this feature is docker specific, leaving room for podman etc. in the immediate future xref #154.

There was some controversy regarding the different resolv.conf for nodes and clusters.
If we want to use the same the resolv.conf for everything I guess that the only options is rewriting it in the nodes and add the hostnames to /etc/hosts manually ... but don't know if docker is doing some magic behind the scenes on those files

Signed-off-by: Jintao Zhang <[email protected]>

tao12345666333 · 2019-06-05T17:04:09Z

/test pull-kind-conformance-parallel

tao12345666333 · 2019-06-05T17:25:10Z

It seems that the docker related commands cannot be tested in the CI environment, so I deleted the test case for the docker network.

tao12345666333 · 2019-06-05T17:26:50Z

I think we should do something to push this PR forward.

/assign @BenTheElder

BenTheElder · 2019-06-05T17:56:33Z

pkg/cluster/nodes/util.go

+	}
+
+	// filter the loopback addresses
+	re := regexp.MustCompile("(?m)[\r\n]+^.*((127.([0-9]{1,3}.){2}[0-9]{1,3})|(::1)).*$")


this shouldn't even be necessary in the hosts's resolv.conf ?

BenTheElder · 2019-06-05T17:58:13Z

pkg/container/docker/network.go

+// DeleteNetwork delete the special network
+// only when the network was created by kind
+func DeleteNetwork(networkName string) error {
+	if isNetworkCreatedByKind(networkName) {


this is orthogonal to the functionality of delete network. this package should be kind agnostic. we don't do this for delete containers

BenTheElder · 2019-06-05T17:58:30Z

pkg/container/docker/network.go

+		"docker", "network",
+		"create",
+		"--driver=bridge",
+		"--label="+fmt.Sprintf("%s=%s", constants.ClusterLabelKey, networkName),


this package should not be aware of the cluster label key. this package just implements docker functionality

BenTheElder · 2019-06-05T17:58:49Z

pkg/container/docker/network.go

+}
+
+// isNetworkCreatedByKind checks if it was created by kind
+func isNetworkCreatedByKind(networkName string) bool {


this should live somewhere in cluster/

BenTheElder · 2019-06-05T17:59:46Z

pkg/cluster/internal/kubeadm/config.go

@@ -181,6 +182,7 @@ evictionHard:
  nodefs.available: "0%"
  nodefs.inodesFree: "0%"
  imagefs.available: "0%"
+resolvConf: /kind/resolv.conf


this is not being done for all api versions

BenTheElder · 2019-06-05T18:00:10Z

pkg/cluster/internal/create/nodes.go

@@ -193,16 +214,16 @@ func nodesToCreate(cfg *config.Cluster, clusterName string) []nodeSpec {
 }

 // TODO(bentheelder): remove network in favor of []cri.PortMapping when that is in
-func (d *nodeSpec) Create(clusterLabel string) (node *nodes.Node, err error) {
+func (d *nodeSpec) Create(networkName, clusterLabel string) (node *nodes.Node, err error) {


why isn't this in the nodeSpec?

BenTheElder · 2019-06-05T18:01:06Z

pkg/cluster/config/types.go

@@ -102,4 +102,7 @@ type Networking struct {
 	// If DisableDefaultCNI is true, kind will not install the default CNI setup.
 	// Instead the user should install their own CNI after creating the cluster.
 	DisableDefaultCNI bool
+	// BridgeName is the bridge network name.
+	// Defaults to same as cluster name.
+	BridgeName string


#273 (comment)

(this config field name is not suitable, we need to separate docker specific functionality and make that clear)

BenTheElder · 2019-06-05T18:06:02Z

It seems that the docker related commands cannot be tested in the CI environment, so I deleted the test case for the docker network.

they absolutely can, but the unit tests should be unit tests, not invoking docker commands (!)

this is why we need #528

I think we should do something to push this PR forward.

@tao12345666333 unfortunately I need to prioritize some other fixes etc related to testing Kubernetes (EG the HA fixes), and we have many open PRs.. This is pretty far into P3 https://kind.sigs.k8s.io/docs/contributing/project-scope/

I'm still not convinced that the DNS approach is a good one to solve this problem and may introduce more problems, and the field name needs thought / is not in line with the proposal. Left some more specific review feedback.

k8s-ci-robot · 2019-10-02T20:52:01Z

@tao12345666333: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kind-conformance-parallel	`48977ec`	link	`/test pull-kind-conformance-parallel`
pull-kind-conformance-parallel-1-15	`48977ec`	link	`/test pull-kind-conformance-parallel-1-15`
pull-kind-unit	`48977ec`	link	`/test pull-kind-unit`
pull-kind-e2e-kubernetes	`48977ec`	link	`/test pull-kind-e2e-kubernetes`
pull-kind-conformance-parallel-1-16	`48977ec`	link	`/test pull-kind-conformance-parallel-1-16`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

BenTheElder · 2019-11-01T13:51:34Z

let's revisit a more complete design in the 0.7ish timeframe with the provider API in mind and more attention to dns config handling across platforms etc.

aojea · 2019-11-09T10:03:30Z

let's revisit a more complete design in the 0.7ish timeframe with the provider API in mind and more attention to dns config handling across platforms etc.

do we want to use docker embedded dns for the nodes and use the autodiscovery or are we fine without that functionality?
The later will simplify a lot the implementations because will only requires to deal with the resolv.conf merging.
The downside is that it won't allow using the dns names to configure the cluster, but with all the ipv6 and dual-stack features coming in, I really would love not to depend on the resolver in those environments 😅

tao12345666333 · 2019-11-09T14:36:15Z

In my opinion, if we don't use docker embedded dns and only use the new bridge network, then we may be able to reduce a lot of trouble.😄

…ubernetes-sigs#484) * Update - create cluster - instllation dock EN/ES * Update - create cluster - quick start guide dock EN/ES * Fix - IP on doc * Fix - IP on doc * Fix - IP on doc

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 5, 2019

k8s-ci-robot requested review from BenTheElder and munnerz May 5, 2019 09:28

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 5, 2019

k8s-ci-robot requested review from aojea and neolit123 May 5, 2019 13:00

aojea reviewed May 6, 2019

View reviewed changes

pkg/cluster/nodes/create.go Outdated Show resolved Hide resolved

aojea reviewed May 6, 2019

View reviewed changes

pkg/cluster/nodes/util.go Outdated Show resolved Hide resolved

BenTheElder reviewed May 6, 2019

View reviewed changes

pkg/cluster/nodes/create.go Outdated Show resolved Hide resolved

BenTheElder reviewed May 6, 2019

View reviewed changes

aojea reviewed May 8, 2019

View reviewed changes

pkg/cluster/internal/create/actions/loadbalancer/loadbalancer.go Show resolved Hide resolved

tao12345666333 force-pushed the using-network branch 4 times, most recently from 95fa838 to eda987d Compare June 5, 2019 16:56

only delete network which created by kind.

48977ec

Signed-off-by: Jintao Zhang <[email protected]>

tao12345666333 force-pushed the using-network branch from eda987d to 48977ec Compare June 5, 2019 16:59

k8s-ci-robot assigned BenTheElder Jun 5, 2019

BenTheElder reviewed Jun 5, 2019

View reviewed changes

BenTheElder mentioned this pull request Jun 19, 2019

Automatically add the docker network to no_proxy #628

Merged

tao12345666333 mentioned this pull request Jun 23, 2019

Cluster doesn't restart when docker restarts #148

Closed

aojea mentioned this pull request Oct 21, 2019

Allow specifying network name to use in docker provisioner #978

Closed

BenTheElder closed this Nov 1, 2019

fllaca mentioned this pull request Dec 24, 2019

Is it possible accessing host machine ports from Kind pods? #1200

Closed

tao12345666333 mentioned this pull request Feb 7, 2020

WIP: Add extra nics to nodes #1316

Closed

tao12345666333 mentioned this pull request Apr 19, 2022

Add start and stop command to kind. #2715

Open

Using bridge network for nodes #484

Using bridge network for nodes #484

Conversation

tao12345666333 commented May 5, 2019 • edited Loading

k8s-ci-robot commented May 5, 2019

BenTheElder commented May 5, 2019

BenTheElder commented May 5, 2019

tao12345666333 commented May 5, 2019

neolit123 commented May 5, 2019

aojea commented May 5, 2019 • edited Loading

aojea commented May 5, 2019

tao12345666333 commented May 5, 2019

BenTheElder commented May 5, 2019

aojea commented May 6, 2019 • edited Loading

BenTheElder commented May 6, 2019 via email

tao12345666333 commented May 6, 2019

aojea commented May 6, 2019 • edited Loading

tao12345666333 commented May 6, 2019

BenTheElder commented May 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neolit123 May 9, 2019 • edited Loading

Choose a reason for hiding this comment

tao12345666333 commented May 8, 2019

aojea commented May 8, 2019 • edited Loading

tao12345666333 commented Jun 5, 2019

tao12345666333 commented Jun 5, 2019

tao12345666333 commented Jun 5, 2019

BenTheElder Jun 5, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder commented Jun 5, 2019

k8s-ci-robot commented Oct 2, 2019

BenTheElder commented Nov 1, 2019

aojea commented Nov 9, 2019

tao12345666333 commented Nov 9, 2019

tao12345666333 commented May 5, 2019 •

edited

Loading

aojea commented May 5, 2019 •

edited

Loading

aojea commented May 6, 2019 •

edited

Loading

aojea commented May 6, 2019 •

edited

Loading

neolit123 May 9, 2019 •

edited

Loading

aojea commented May 8, 2019 •

edited

Loading

BenTheElder Jun 5, 2019 •

edited

Loading