--- name: containers-ts-pod-networking title: Debugging network connections between pods description: Review the options and strategies for debugging connection issues between pods. last-updated: 2026-07-30 --- > ## Documentation Index > The table of contents for this documentation set is at https://cloud.ibm.com/docs/containers?format=markdown > The index for all IBM Cloud docs is at: https://cloud.ibm.com/docs/llms.txt > Use these files to discover more information as needed. # Debugging network connections between pods {: #debug_pods} Review the options and strategies for debugging connection issues between pods. ## Check the health of your cluster components and networking pods {: #debug_pods_health} Follow these steps to check the health of your components. Networking issues might occur if your cluster components are not up to date or are not in a healthy state. 1. Check that your cluster master and worker nodes run on a supported version and are in a healthy state. If the cluster master or workers do not run a supported version, [make any necessary updates](https://cloud.ibm.com/docs/containers?topic=containers-update&format=markdown) so that they run a supported version. If the status of any components is not `Normal` or `Ready`, review the [cluster master health states](https://cloud.ibm.com/docs/containers?topic=containers-debug_master&format=markdown), [cluster states](https://cloud.ibm.com/docs/containers?topic=containers-cluster-states-reference&format=markdown), [worker node states](https://cloud.ibm.com/docs/containers?topic=containers-worker-node-state-reference&format=markdown), or [steps to troubleshoot `Critical` or `NotReady` worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-ts-critical-notready&format=markdown) for more information. Make sure any related issues are resolved before continuing. To check the cluster master version and health: ```sh ibmcloud ks cluster get -c CLUSTER-ID ``` {: pre} To check worker node versions and health: ```sh ibmcloud ks workers -c CLUSTER-ID ``` {: pre} 2. For each worker node, verify that the Calico and cluster DNS pods are present and running in a healthy state. 1. Run the command to get the details of your cluster's pods. ```sh kubectl get pods -A -o wide | grep -e calico -e coredns ``` {: pre} 2. In the output, make sure that your cluster includes the following pods. Make sure that each pod's status is `Running`, and that the pods do not have too many restarts. - Exactly one `calico-node` pod per worker node. - At least one `calico-typha` pod per cluster. Larger clusters might have more than one. - Exactly one `calico-kube-controllers` pod per cluster. - At least one `coredns` pod per cluster. Most clusters have three `coredns` pods; larger clusters might have more. - Exactly one `coredns-autoscaler` pod. Example output ```txt NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED READINESS GATES calico-system calico-kube-controllers-1a1a1a1 1/1 Running 0 16h 172.17.87.11 192.168.0.28 calico-system calico-node-1a1a1a1 1/1 Running 0 16h 192.168.0.28 192.168.0.28 calico-system calico-node-1a1a1a1 1/1 Running 0 16h 192.168.0.27 192.168.0.27 calico-system calico-typha-1a1a1a1 1/1 Running 0 16h 192.168.0.28 192.168.0.28 kube-system coredns-867bfb84df-1a1a1a1 1/1 Running 0 16h 172.17.87.1 192.168.0.28 kube-system coredns-867bfb84df-1a1a1a1 1/1 Running 0 16h 172.17.87.15 192.168.0.28 kube-system coredns-867bfb84df-1a1a1a1 1/1 Running 0 16h 172.17.87.14 192.168.0.28 kube-system coredns-autoscaler-1a1a1a1 1/1 Running 0 16h 172.17.87.2 192.168.0.28 ``` {: screen} 3. If any of the listed pods are not present or are in an unhealthy state, go through the cluster and worker node trouble shooting documentation included in the previous steps. Make sure any issues with the pods in this step are resolved before moving on. ## Debug with test pods {: #debug_pods_test} To determine the cause of networking issues on your pods, you can create a test pod on each of your worker nodes. Then, you can run tests and observe networking activity within the pod, which might reveal the source of the problem. ### Setting up the pods {: #debug_pods_test_setup} 1. Create a new privileged namespace for your test pods. Creating a new namespace prevents any custom policies or configurations in existing namespaces from affecting your test pods. In this example, the new namespace is called `pod-network-test`. Create the namespace. ```sh kubectl create ns pod-network-test ``` {: pre} 1. Create and apply the following daemonset to create a test pod on each node. ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: labels: name: webserver-test app: webserver-test name: webserver-test spec: selector: matchLabels: name: webserver-test template: metadata: labels: name: webserver-test app: webserver-test spec: tolerations: - operator: "Exists" containers: - name: webserver securityContext: privileged: true image: us.icr.io/armada-master/network-alpine:latest env: - name: ENABLE_ECHO_SERVER value: "true" - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name restartPolicy: Always terminationGracePeriodSeconds: 1 ``` {: codeblock} 1. Apply the daemonset to deploy a test pod on each worker node. ```sh kubectl apply --namespace pod-network-test -f ``` {: pre} 1. Verify that the pods start successfully by listing all pods in the namespace. ```sh kubectl get pods --namespace pod-network-test -o wide ``` {: pre} ### Running tests within the pods {: #debug_pods_test_run} Run `curl`, `ping`, and `nc` commands to test each pod's network connection and the `dig` command to test the cluster DNS. Review each output, then see [Identifying issues](#debug_pods_test_id) to find what the outcomes might mean. 1. List your test pods and note the name and IP of each pod. ```sh kubectl get pods --namespace pod-network-test -o wide ``` {: pre} Example output ```sh NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES webserver-test-2fv7c 1/1 Running 0 68s 172.17.36.169 10.245.0.4 webserver-test-4mktb 1/1 Running 0 68s 172.17.61.240 10.245.0.5 ``` {: screen} 1. Run the `exec` command to log in to one pod. ```sh kubectl exec -it --namespace pod-network-test -- sh ``` {: pre} 1. Run the `curl` command on the pod and note the output. Specify the IP of a test pod that you did **not** log in to. This tests the network connection between pods on different nodes. ```sh curl :8080 ``` {: pre} Example successful output. ```sh Hostname: webserver-test-4mktb Pod Information: node name: env var NODE_NAME not set pod name: webserver-test-4mktb pod namespace: env var POD_NAMESPACE not set pod IP: env var POD_IP not set Connection Information: remote address: 172.17.36.169 remote port: 56042 local address: 172.17.61.240 local port: 8080 ``` {: screen} 1. Run the `ping` command on the pod and note the output. Specify the IP of a test pod that you did **not** log in to with the `exec` command. This tests the network connection between pods on different nodes. ```sh ping -c 5 ``` {: pre} Example successful output. ```sh PING 172.17.61.240 (172.17.61.240) 56(84) bytes of data. 64 bytes from 172.17.61.240: icmp_seq=1 ttl=62 time=0.473 ms 64 bytes from 172.17.61.240: icmp_seq=2 ttl=62 time=0.449 ms 64 bytes from 172.17.61.240: icmp_seq=3 ttl=62 time=0.381 ms 64 bytes from 172.17.61.240: icmp_seq=4 ttl=62 time=0.438 ms 64 bytes from 172.17.61.240: icmp_seq=5 ttl=62 time=0.348 ms --- 172.17.61.240 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4086ms rtt min/avg/max/mdev = 0.348/0.417/0.473/0.046 ms ``` {: screen} 1. Run the `nc` command on the pod and note the output. Specify the IP of a test pod that you did **not** log in to with the `exec` command. This tests the network connection between pods on different nodes. ```sh nc -vzw 5 8080 ``` {: pre} Example successful output. ```sh nc -vzw 5 172.17.61.240 8080 172.17.61.240 (172.17.61.240:8080) open ``` {: screen} 1. Run the `dig` commands to test the DNS. ```sh dig +short kubernetes.default.svc.cluster.local ``` {: pre} Example output ```sh 172.21.0.1 ``` {: screen} ```sh dig +short ibm.com ``` {: pre} Example output ```sh 23.50.74.64 ``` {: screen} 1. Run `curl` commands to test a full TCP or HTTPS connection to the service. This example tests the connection between the pod and the cluster master by retrieving the cluster's version information. Successfully retrieving the cluster version indicates a healthy connection. ```sh curl -k https://kubernetes.default.svc.cluster.local/version ``` {: pre} Example output ```sh "major": "1", "minor": "34", "emulationMajor": "1", "emulationMinor": "34", "minCompatibilityMajor": "1", "minCompatibilityMinor": "33", "gitVersion": "v1.34.7+IKS", "gitCommit": "67bf12be5abc8e65743a243172693ad1a098a2c4", "gitTreeState": "clean", "buildDate": "2026-04-16T04:24:50Z", "goVersion": "go1.25.9", "compiler": "gc", "platform": "linux/amd64" ``` {: screen} 1. Log out of the pod. ```sh exit ``` {: pre} 1. Repeat the earlier steps with the remaining pods. ### Identifying issues {: #debug_pods_test_id} Review the outputs from the earlier section to help find the cause of your pod networking issues. This section lists some common causes that can be identified from the earlier section. - If the commands functioned normally on the test pods, but you still have networking issues with the application pods in your default namespace, there might be issues related specifically to your application. - You might have Calico or Kubernetes network security policies in place that restrict your networking traffic. If a networking policy is applied to a pod, *all traffic that is not specifically allowed by that policy is dropped*. For more information on networking policies, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/){: external}. - If you are using Istio or Red Hat OpenShift Service Mesh, there might be service configuration issues that drop or block traffic between pods. For more information, see the troubleshooting documentation for [Istio](https://istio.io/latest/docs/ops/diagnostic-tools/){: external} and [Red Hat OpenShift Service Mesh](https://docs.redhat.com/en/documentation/openshift_container_platform/4.19/html/service_mesh/service-mesh-3-x){: external}. - The issue might be related to bugs in the application rather than your cluster, and might require your own independent trouble shooting. - If the `curl`, `ping`, or `nc` commands failed for certain pods, identify which worker nodes those pods are on. If the issue exists on only some of your worker nodes, [replace those worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-cli&format=markdown#worker-replace-cli) or see additional information on [worker node trouble shooting](https://cloud.ibm.com/docs/containers?topic=containers-ts-critical-notready&format=markdown). - If the DNS lookups from the `dig` commands failed, check that the [cluster DNS is configured properly](https://cloud.ibm.com/docs/containers?topic=containers-cluster_dns&format=markdown).. If you are still unable to resolve your pod networking issue, [open a support case](https://cloud.ibm.com/docs/containers?topic=containers-get-help&format=markdown) and include a detailed description of the problem, how you have tried to solve it, what kinds of tests you ran, and [relevant logs](https://cloud.ibm.com/docs/containers?topic=containers-debug_clusters&interface=ui&format=markdown#ts-5) for your pods and worker nodes. For more information on opening a support case and what information to include, see the [general debugging guide](https://cloud.ibm.com/docs/containers?topic=containers-debug_clusters&interface=ui&format=markdown#ts-3).