Why does the Network status show an NHC006 error?

Classic infrastructure

When you check the status of your cluster's health by running the ibmcloud oc cluster health issues --cluster <CLUSTER_ID>, you see an error similar to the following example.

ID       Component   Severity   Description
NHC006   Network     Warning    One or more DNS resolvers are not reachable from certain worker nodes.

If you check the details of the issue, you will see which DNS resolvers cannot be accessed from which worker node.

ibmcloud ks cluster health issue get --cluster <CLUSTER_ID> --issue NHC006

This warning indicates that some worker nodes are unable to reach one or more DNS resolvers. This can lead to DNS failures and impact workload communication.

1. Inspect Calico GlobalNetworkPolicies (GNP) by listing all GNPs.

   kubectl get globalnetworkpolicies.crd.projectcalico.org
   

   Run the following command to review a specific policy.

   kubectl get globalnetworkpolicies.crd.projectcalico.org <policy-name> -o yaml
   

2. Look for any egress rules that block DNS traffic (UDP/TCP port 53). Also check for selector fields that might improperly include worker nodes.

3. Validate DNS reachability from worker nodes using a debug pod

   kubectl run  -i --tty debug \
     --image=us.icr.io/armada-master/network-alpine:latest \
     --restart=Never \
     --overrides='
   {
     "apiVersion": "v1",
     "spec": {
       "nodeName": "<node-name>"
     }
   }' -- sh 
   

4. Run the following commands inside the debug pod. If these fail, DNS might be blocked by policies or IaaS-level configurations.

   nslookup ibm.com
   

   dig ibm.com
   

5. Review your infrastructure (network appliances, ACLs, etc.) and allow UDP and TCP port 53 outbound traffic.

6. If the issue continues, contact support for further assistance. Open a support case. In the case details, be sure to include any relevant log files, error messages, or command outputs.