Cluster DNS should resolve host /etc/resolv.conf

[dghubble]

No description provided.

[chancez]

I wrote up a long response why I think we should run a local forwarding resolver, but for now I'm actually going to say we should start by mounting in resolv.conf in both kubelet's which should probably work, and should require the minimum amount of changes. Due to the way skydns inherits DNS configuration based on the kubelet's resolv.conf (ie: in the container) we definitely want to ensure the resolv.conf for the kubelet is explicitly provided though.

Reason: When the self-hosted kubelet is started, the resolv.conf in the container will be confgured to point at the cluster-dns service (skydns). If skydns is recreated, then when it's started, it will inherit DNS from the kubelet's resolv.conf, which at this point is in the container it's pointed at skydns (so we have a dns server potentially pointed at its self).

I do think mounting in the resolv.conf isn't the best, and I would much prefer a more production ready solution such as running a local DNS server on nodes to forward queries, but it has a lot more involved to make it work, as well as an additional dependency. I also think mounting in resolv.conf will result in other bugs depending on if/how users configure their own DNS infrastructure, so it may be worth investigating that further as well.

[chancez]

According to https://github.com/kubernetes/kubernetes/blob/ee412efcefc75b4625a71073de802c755c1a9205/build/kube-dns/README.md#where-does-resolution-work
The self-hosted kubelet might be able to avoid mounting in the resolv.conf by just using hostNetwork: true it seems, but would need to verify what it's resolv.conf looks like with that.

Edit: which seems to be what we're doing now, so it may be okay already, perhaps it is just the host kubelet..

[aaronlevy]

So part of this comes down to rkt version -- as the --dns flag was broken with rkt-fly stage1.

Assuming rkt version contains the fix: rkt/rkt#2141 -- I would lean toward using the rkt --dns flag vs. manually mounting /etc/resolv.conf.

For some points of clarity:

When the self-hosted kubelet is started, the resolv.conf in the container will be confgured to point at the cluster-dns service (skydns)

Not necessarily. The kubelet runs with hostNetwork=true -- which should mean that it does not use the clusterDNS (unless you are manually adding it to the hosts /etc/resolv.conf). See: https://github.com/kubernetes/kubernetes/blob/release-1.2/cluster/addons/dns/README.md#where-does-resolution-work

So this is just a matter of making sure the DNS settings you want plumbed through are correct -- and should be relatively straight-forward:

On-host kubelet we can add --dns flag to the kubelet via RKT_OPTS in kubelet.service, or by making an upstream change to kubelet-wrapper (I could go either way on this).

If the dns settings at this stage are incorrect (for any pod) -- then it's a bug we need to file upstream (rkt/kubernetes).

ref:
coreos/coreos-kubernetes#506
coreos/bugs#1228
rkt/rkt#2044

[chancez]

@aaronlevy We cannot use --dns without running our own DNS server for this. The issue is that --dns is a static nameserver, so if your DNS server changes, the kubelet will not be updated. We could pair --dns with running a local DNS forwarder so that we can use --dns 127.0.0.1, which is fine to be static, since the local DNS resolver will properly handle changes to the host's DNS server.

[aaronlevy]

I still don't think this is a bootkube issue really. If you expect your dns server to change (why?) -- that's going to need to be resolved at the host level (local dns forwarder I guess is one option). Bootkube isn't going to resolve this for you.

[philips]

What is the problem with bind mounting in reslv.conf?

[aaronlevy]

Nothing to my knowledge.

Eventually rkt may do this on its own. The --dns flag is what rkt uses via the kubelet - and perfectly acceptable for us to use as well (if it's easier to use that than bin mount resolv.conf). Either way should work.

[dghubble]

When the network DNS server changes, that needs to propagate to hosts, to kubelet, skydns, and to pods. Mounting /etc/resolv.conf, it may be needed to roll the skydns pod, but I haven't tried this out. I'd avoid provisioning a kubelet unit with a static DNS or else run a dnsmasq DNS resolver on the host as was suggested above.

We're currently mounting /etc/resolve.conf.

[philips]

Mounting /etc/resolv.conf seems fine until we get feedback otherwise that it breaks.