tencent cloud

masukan

Cluster DNS Troubleshooting

Terakhir diperbarui:2023-10-08 17:53:41

    Troubleshooting Approaches

    1. Make sure that the cluster DNS runs normally

    Container DNS passes through cluster DNS (usually CoreDNS). First, make sure that the cluster DNS runs normally. You can see the cluster IP of the DNS in the --cluster-dns startup parameter of kubelet:

    $ ps -ef | grep kubelet
    ... /usr/bin/kubelet --cluster-dns=172.16.14.217 ...
    

    Find the DNS Service:

    $ kubectl get svc -n kube-system | grep 172.16.14.217
    kube-dns              ClusterIP   172.16.14.217   <none>        53/TCP,53/UDP              47d
    

    Check for the endpoint:

    $ kubectl -n kube-system describe svc kube-dns | grep -i endpoints
    Endpoints:         172.16.0.156:53,172.16.0.167:53
    Endpoints:         172.16.0.156:53,172.16.0.167:53
    

    Check whether the Pod of the endpoint is normal:

    $ kubectl -n kube-system get pod -o wide | grep 172.16.0.156
    kube-dns-898dbbfc6-hvwlr            3/3       Running   0          8d        172.16.0.156   10.0.0.3
    

    2. Make sure that the Pod can communicate with the cluster DNS

    Check whether the Pod can connect to the cluster DNS. You can run the telnet command in the Pod to view port 53 of the DNS:

    # Cluster IP for connecting to the DNS Service
    $ telnet 172.16.14.217 53
    
    Note:

    If there are no testing tools such as telnet in the container, you can use nsenter to enter netns for packet capturing and use telnet on the host for testing.

    If the network is found to be disconnected, check the following network settings:

    • Check the security group settings of the node and the container IP range of the cluster that needs to be opened.
    • Check for firewall rules and check the iptables.
    • Check whether kube-proxy runs normally. The cluster DNS IP is the cluster IP, which is forwarded through the iptables or IPVS rules generated by kube-proxy.

    3. Capture packets

    If the cluster DNS runs normally and the Pod can communicate with the cluster DNS, capture packets for further checks. If the problem can be easily reproduced, you can use nsenter to enter netns to capture container packets:

    tcpdump -i any port 53 -w dns.pcap
    # tcpdump -i any port 53 -nn -tttt
    

    If the cause still cannot be identified, you can capture packets at multiple points along the request linkage for analysis, such as Pod container, host cbr0 bridge, primary ENI of the host (eth0), primary ENI of the host of the CoreDNS Pod, cbr0, and container. Wait for the problem to recur and locate the point where the packet is lost.

    Issue and Cause

    Latency of five seconds

    If it often takes five seconds to return a DNS query result, packets are usually lost due to kernel conntrack conflicts. The root cause is the bug in the conntrack module, where some packets are discarded due to resource competition when netfilter is used for NAT.

    • It may possibly occur when multiple threads or processes send the same quintuple UDP packet through the same socket concurrently.
    • Both glibc and musl (Alpine Linux's libc) use "parallel query", i.e., multiple query requests are sent concurrently, which tends to cause conflicts and request discarding.
    • As IPVS also uses conntrack, this problem cannot be avoided in IPVS mode of kube-proxy.

    Workaround

    Use local DNS. DNS requests of the container are sent to the local DNS cache service (dnsmasq, nscd, etc.), without DNAT or conntrack conflicts. In addition, the DNS service will not be a performance bottleneck.
    You can use local DNS in two ways:

    • Each container comes with a DNS cache service.
    • Each node runs a DNS cache service, and all containers use the DNS cache of the node as their nameserver.

    Timeout of the resolution of an external domain name

    Possible reasons:

    • The upstream DNS fails.
    • The ACL or firewall of the upstream DNS blocks the packet.

    Timeout of all resolutions

    If a cluster Pod fails to resolve both Services and external domain names, there is generally a problem with the communication between the Pod and the cluster DNS.
    Possible reasons:

    • The node firewall doesn't open the cluster IP range; therefore, the Pod cannot communicate with that of the cluster DNS when they are on different nodes, and DNS requests cannot be received.
    • kube-proxy is abnormal.
    Hubungi Kami

    Hubungi tim penjualan atau penasihat bisnis kami untuk membantu bisnis Anda.

    Dukungan Teknis

    Buka tiket jika Anda mencari bantuan lebih lanjut. Tiket kami tersedia 7x24.

    Dukungan Telepon 7x24