Classic clusters: Why does source IP preservation fail when using tainted nodes?

Classic infrastructure

You enabled source IP preservation for an Ingress ALB service by changing externalTrafficPolicy to Local in the service configuration file. However, no traffic reaches the back-end service for your app.

When you enable source IP preservation for Ingress ALB services, the source IP address of the client request is preserved. The service forwards traffic to app pods on the same worker node only to ensure that the request packet's IP address isn't changed.

Typically, Ingress ALB service pods are deployed to the same worker nodes as the app pods. However, in some situations, the service pods and app pods might not be scheduled on the same worker node. If you use Kubernetes taints on worker nodes, any pods that do not have a taint toleration are prevented from running on the tainted worker nodes. Source IP preservation might not work, depending on the type of taint that you used:

Edge node taints: You added the dedicated=edge label to two or more worker nodes on each public VLAN in your cluster to ensure that Ingress pods deploy to those worker nodes only. Then, you also tainted those edge nodes to prevent any other workloads from running on edge nodes. However, you didn't add an edge node affinity rule and toleration to your app deployment. Your app pods can't be scheduled on the same tainted nodes as the service pods, and no traffic reaches the back-end service for your app.
Custom taints: You used custom taints on several nodes so that only app pods with that taint toleration can deploy to those nodes. You added affinity rules and tolerations to the deployments of your app and Ingress service so that their pods deploy to only those nodes. However, ibm-cloud-provider-ip keepalived pods that are automatically created in the ibm-system namespace ensure that the ALB pods and the app pods are always scheduled onto the same worker node. These keepalived pods don't have the tolerations for the custom taints that you used. They can't be scheduled on the same tainted nodes that your app pods are running on, and no traffic reaches the back-end service for your app.

Resolve the issue by choosing one of the following options:

Edge node taints: To ensure that your ALB and app pods are deployed to tainted edge nodes, add edge node affinity rules and tolerations to your app deployment. Ingress ALB pods have these affinity rules and tolerations by default.
Custom taints: Remove custom taints that the keepalived pods don't have tolerations for. Instead, you can label worker nodes as edge nodes, and then taint those edge nodes.

If you completed the previous steps, but the keepalived pods are still not scheduled, gather more information about the keepalived pods:

Get the keepalived pods.
```
kubectl get pods -n ibm-system
```

In the output, look for ibm-cloud-provider-ip pods that have a Status of Pending. Example:

ibm-cloud-provider-ip-169-61-XX-XX-55967b5b8c-7zv9t     0/1       Pending   0          2m        <none>          <none>
ibm-cloud-provider-ip-169-61-XX-XX-55967b5b8c-8ptvg     0/1       Pending   0          2m        <none>          <none>

Describe each keepalived pod and review the Events section. Address any listed error or warning messages.
```
kubectl describe pod ibm-cloud-provider-ip-169-61-XX-XX-55967b5b8c-7zv9t -n ibm-system
```