Troubleshooting Approaches
1. Make sure that the cluster DNS runs normally
Container DNS passes through cluster DNS (usually CoreDNS). First, make sure that the cluster DNS runs normally. You can see the cluster IP of the DNS in the --cluster-dns startup parameter of kubelet:
$ ps -ef | grep kubelet
... /usr/bin/kubelet --cluster-dns=172.16.14.217 ...
Find the DNS Service:
$ kubectl get svc -n kube-system | grep 172.16.14.217
kube-dns ClusterIP 172.16.14.217 <none> 53/TCP,53/UDP 47d
Check for the endpoint:
$ kubectl -n kube-system describe svc kube-dns | grep -i endpoints
Endpoints: 172.16.0.156:53,172.16.0.167:53
Endpoints: 172.16.0.156:53,172.16.0.167:53
Check whether the Pod of the endpoint is normal:
$ kubectl -n kube-system get pod -o wide | grep 172.16.0.156
kube-dns-898dbbfc6-hvwlr 3/3 Running 0 8d 172.16.0.156 10.0.0.3
2. Make sure that the Pod can communicate with the cluster DNS
Check whether the Pod can connect to the cluster DNS. You can run the telnet command in the Pod to view port 53 of the DNS:
$ telnet 172.16.14.217 53
If the network is found to be disconnected, check the following network settings:
Check the security group settings of the node and the container IP range of the cluster that needs to be opened.
Check for firewall rules and check the iptables.
Check whether kube-proxy runs normally. The cluster DNS IP is the cluster IP, which is forwarded through the iptables or IPVS rules generated by kube-proxy.
3. Capture packets
If the cluster DNS runs normally and the Pod can communicate with the cluster DNS, capture packets for further checks. If the problem can be easily reproduced, you can use nsenter to enter netns to capture container packets:
tcpdump -i any port 53 -w dns.pcap
If the cause still cannot be identified, you can capture packets at multiple points along the request linkage for analysis, such as Pod container, host cbr0 bridge, primary ENI of the host (eth0), primary ENI of the host of the CoreDNS Pod, cbr0, and container. Wait for the problem to recur and locate the point where the packet is lost.
Issue and Cause
Latency of five seconds
If it often takes five seconds to return a DNS query result, packets are usually lost due to kernel conntrack conflicts. The root cause is the bug in the conntrack module, where some packets are discarded due to resource competition when netfilter is used for NAT.
It may possibly occur when multiple threads or processes send the same quintuple UDP packet through the same socket concurrently.
Both glibc and musl (Alpine Linux's libc) use "parallel query", i.e., multiple query requests are sent concurrently, which tends to cause conflicts and request discarding.
As IPVS also uses conntrack, this problem cannot be avoided in IPVS mode of kube-proxy.
Workaround
Use local DNS. DNS requests of the container are sent to the local DNS cache service (dnsmasq, nscd, etc.), without DNAT or conntrack conflicts. In addition, the DNS service will not be a performance bottleneck.
You can use local DNS in two ways:
Each container comes with a DNS cache service.
Each node runs a DNS cache service, and all containers use the DNS cache of the node as their nameserver.
Timeout of the resolution of an external domain name
Possible reasons:
The upstream DNS fails.
The ACL or firewall of the upstream DNS blocks the packet.
Timeout of all resolutions
If a cluster Pod fails to resolve both Services and external domain names, there is generally a problem with the communication between the Pod and the cluster DNS.
Possible reasons:
The node firewall doesn't open the cluster IP range; therefore, the Pod cannot communicate with that of the cluster DNS when they are on different nodes, and DNS requests cannot be received.
kube-proxy is abnormal.