kube-dns disregards stubDomains option while resolving externalName service

[thockin]

Was kubernetes/kubernetes#63821 @driusha

/kind bug

What happened:
I have:

• internal DNS "10.2.2.10" with zone "example.com" on it,
• stubDomains: {"example.com": ["10.2.2.10"]}
• externalName service "foo" which points to domain "foo.example.com"

When i try to resolve name "foo", i get NXDOMAIN.
If i try to create externalName (ex. mail -> mail.google.com) without stubDomains, everything works.

Environment:

• Kubernetes version v1.8.5
• kube-dns version 1.14.5

[bradleybluebean]

Also seeing this issue with kube-dns 1.14.10

[bradleybluebean]

One way to fix this, if you're using Kubernetes 1.9 or above, is to change the dnsPolicy of the kube-dns pod.

https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pods

In the authoritative template, this is set to Default.

https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/kube-dns/kube-dns.yaml.base#L212

This means the pod inherits the nameserver settings from the underlying node that the pod runs on. The kube-dns process then uses this nameserver to try to resolve the externalName host (foo.example.com above). If the inherited nameserver is not also able to resolve the stubdomain (example.com above), the lookup will fail. By changing the dnsPolicy to None and adding a dnsConfig, we can make the kube-dns process query the dnsmasq process (i.e., localhost) instead of an inherited nameserver.

  dnsPolicy: None
  dnsConfig:
    nameservers:
    - 127.0.0.1

Since dnsmasq won't be getting upstream nameservers from the underlying host, the kube-dns ConfigMap should also include an upstreamNameservers setting.

https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#configure-stub-domain-and-upstream-dns-servers

[chrisohaver]

Since dnsmasq won't be getting upstream nameservers from the underlying host, the kube-dns ConfigMap should also include an upstreamNameservers setting.

s/should/must/ .... Without setting an upstreamNameserver, i believe you'd have an infinite lookup loop for external domain names that don't fall in a stub domain.

You can also get into trouble (loop forever) if you try to query a member of a CNAME chain loop (e.g. two external names that point at each other).

Note that moving to CoreDNS is another option. It similarly feeds CNAME targets to itself when looking them up, but does so more internally, and also has CNAME chain loop detection.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[bowei]

/lifecycle frozen

[jhcook-ag]

/remove-lifecycle stale