Why are the ibm-cloud-provider-ip pods for the Istio ingress gateway stuck in pending?

Virtual Private Cloud Classic infrastructure

Multizone clusters only

When you run kubectl get pod -n ibm-system, the ibm-cloud-provider-ip pod that provides the external IP address for your Istio ingress gateway is stuck in the pending status.

Additionally, when you run kubectl describe pod <pod_name> -n ibm-system for the ibm-cloud-provider-ip pod, you notice a scheduling conflict error in the pod's events section of the output.

To identify the ibm-cloud-provider-ip pod for your gateway, you can run kubectl get service -n istio-system to find your gateway's load balancer service, note its EXTERNAL-IP, and look for the IP address in the ibm-cloud-provider-ip pod name.

When an istio-ingressgateway load balancer service is created, an ibm-cloud-provider-ip pod is created to provide an external IP address to the load balancer.

These ibm-cloud-provider-ip pods have a node affinity rule so that they are created in the same zone as the subnet from which the IP address is derived. However, when the ExternalTrafficPolicy setting for the istio-ingressgateway load balancer service is set to Local, then the ibm-cloud-provider-ip pod also has a pod affinity rule to be deployed to the same zone as the gateway load balancer pod. If the IP address and gateway load balancer exist in different zones in your cluster, the ibm-cloud-provider-ip pod can't properly deploy.

To verify that the ibm-cloud-provider-ip and istio-ingressgateway pods don't exist in the same zone:

1. Identify the NODE that your istio-ingressgateway pod is deployed to.

   kubectl get pod -n istio-system -o wide
   

2. Get the EXTERNAL-IP address for the gateway's load balancer service.

   kubectl get service -n istio-system
   

3. Identify the NODE that the ibm-cloud-provider-ip pod for the load balancer is deployed to. Replace <IP-with-hyphens> with the IP address that you found in the previous step. In the IP address, use hyphens (-) instead of periods (.). For example, 169.12.345.67 becomes 169-12-345-67.

   kubectl get pod -n ibm-system -o wide -l ibm-cloud-provider-ip=<IP-with-hyphens>
   

4. List the zones that the worker nodes are in. Compare the worker nodes that you found in step 1 and 3 to determine whether the pods are deployed to worker nodes in different zones.

   kubectl get node --no-headers -L ibm-cloud.kubernetes.io/zone
   

   Example output

   10.176.48.106   Ready   <none>   529d    v1.29+IKS   dal10
   10.176.48.107   Ready   <none>   196d    v1.29+IKS   dal10
   10.184.58.23    Ready   <none>   2y38d   v1.29+IKS   dal12
   10.184.58.42    Ready   <none>   529d    v1.29+IKS   dal12
   

To resolve this issue, you can either move the istio-ingressgateway pod to the zone that the ibm-cloud-provider-ip pod exists in, or vice versa.

• Moving the istio-ingressgateway pod: The gateway's load balancer retains the same external IP address after it moves. However, in version 1.9 and earlier of the add-on, the changes that you make might be overwritten when the zone labels for gateways are automatically populated during Istio patch updates.
• Moving the ibm-cloud-provider-ip pod: The gateway does not retain the same external IP address after it moves, and is assigned a new IP address. However, no changes are overwritten during add-on updates.