Why does the Network status show an `NHC007` error?

Virtual Private Cloud

When you check the status of your cluster's health by running the ibmcloud oc cluster health issues --cluster <CLUSTER_ID>, you see an error similar to the following example.

ID       Component   Severity   Description
NHC007   Network     Warning    One or more DNS resolvers are not reachable from certain worker nodes.

If you check the details of the issue, you will see which DNS resolvers cannot be accessed from which worker node.

ibmcloud ks cluster health issue get --cluster <CLUSTER_ID> --issue NHC007

This warning indicates that DNS traffic from certain worker nodes is being blocked, possibly due to restrictive policies or IaaS-level configurations.

Check your Calico HostEndpoint (HEP) and GlobalNetworkPolicy (GNP) resources, as well as your ACLs, security groups, and any other network appliances that may block outbound DNS traffic.

Review Calico HostEndpoint (HEP) resources to list Calico HEPs and check if any HEP configurations might incorrectly apply restrictions to your worker node interfaces.
```
kubectl get hostendpoints.crd.projectcalico.org
```
Example command to to describe a specific HEP.
```
kubectl describe hostendpoints.crd.projectcalico.org <hep-name>
```

Review Calico GlobalNetworkPolicies (GNP) to list GNPs.

kubectl get globalnetworkpolicies.crd.projectcalico.org

Inspect specific policies for restrictive DNS rules. Focus on egress rules that affect port 53 or apply to node labels/selectors.
```
kubectl get globalnetworkpolicies.crd.projectcalico.org <policy-name> -o yaml
```

Test DNS access from a debug pod, run a temporary debug pod, where use the affected worker nodes name for nodeName. If DNS fails here, it may be due to infrastructure-level blocks.

kubectl run  -i --tty debug \
  --image=us.icr.io/armada-master/network-alpine:latest \
  --restart=Never \
  --overrides='
{
  "apiVersion": "v1",
  "spec": {
    "nodeName": "<node-name>"
  }
}' -- sh

Run the following commands inside the debug pod.
```
nslookup ibm.com
```
```
dig ibm.com
```
Check ACLs (Access Control Lists).
- In the console, navigate to VPC > Access control lists and ensure outbound rules allow both UDP port 53 and TCP port 53.
- In the CLI inspect ACLs by running the following commands.
```
ibmcloud is network-acls
```
```
ibmcloud is network-acl <acl-id>
```
Inspect security group rules to find the security groups associated with your worker nodes and then run to check security group settings. Ensure there are no outbound rules blocking DNS traffic (UDP/TCP port 53).
```
ibmcloud is security-group-rules <security-group-id>
```
Review your infrastructure (network appliances, ACLs, etc.) and allow UDP and TCP port 53 outbound traffic
If DNS is still unreachable after reviewing these items, contact support. Open a support case. In the case details, be sure to include any relevant log files, error messages, or command outputs.

Why does the Network status show an NHC007 error?

Why does the Network status show an `NHC007` error?