Opening required ports and IP addresses in your allowlist

Allow worker nodes to communicate with cluster master

To allow worker nodes to communicate with the cluster master over the public cloud service endpoint, allow outgoing network traffic from the source <each_worker_node_publicIP> to the destination TCP/UDP port range 30000-32767 and port 443, and the following IP addresses and network groups.

This table is moving. For the latest IP lists and continued updates, see the public network isolation folder in the IBM/kube-samples repo. You can Watch pull requests in the repo for updates.

• TCP/UDP port range 30000-32767, port 443 FROM <each_worker_node_publicIP> TO <public_IPs>
• Replace <public_IPs> with the public IP addresses of the region that your cluster is located.

Allow worker nodes to communicate with IBM Cloud Container Registry

To permit worker nodes to communicate with IBM Cloud Container Registry, allow outgoing network traffic from the worker nodes to IBM Cloud Container Registry regions.

Previously Registry subnets (IP addresses) were published in the following table. As of 17 August 2021, the Registry subnets are no longer published. When you access IBM Cloud Container Registry over the public internet, you must not have any allowlist restrictions that are based on IP addresses in place. If you have any concerns about opening your allowlist, you can configure private access to IBM Cloud Container Registry by using the private IBM Cloud network, see Securing your connection to Container Registry.

In addition to the following regional subdomains, you must also allow traffic from your worker nodes to port 443 on all subdomains of icr.io in case of redirection to other subdomains for delivery optimization. You must allow: TCP port 443 FROM <each_worker_node_publicIP> TO *.icr.io. If you use the deprecated domain names, you must allow those too.

Allow outgoing network traffic from worker node to IAM

Allow outgoing network traffic from your worker node to IBM Cloud Identity and Access Management (IAM). Your allowlist must be Layer 7 to allow the IAM domain name. IAM does not have specific IP addresses that you can allow. If your allowlist does not support Layer 7, you can allow all HTTPS network traffic on port 443.

• TCP port 443 FROM <each_worker_node_publicIP> TO https://iam.bluemix.net
• TCP port 443 FROM <each_worker_node_publicIP> TO https://iam.cloud.ibm.com

Optional: Allow outgoing network traffic from the worker nodes to Monitoring and Log Analysis services

• IBM Cloud Monitoring:

  • TCP port 443, port 6443 FROM <each_worker_node_public_IP> TO <monitoring_public_IP>
  • Replace <monitoring_public_IP> with the Monitoring IP addresses.

• IBM Log Analysis:

  • TCP port 443, port 80 FROM <each_worker_node_public_IP> TO <logging_public_IP>
  • Replace <logging_public_IP> with the Log Analysis IP addresses.

Optional: Allow incoming and outgoing network traffic for the managed Istio add-on

• Allow outgoing network traffic from the istio-egressgateway load balancer through the following ports: TCP port 80, port 15443 FROM <each_worker_node_publicIP>
• Allow incoming network traffic to the istiod control plane and the istio-ingressgateway load balancer through the following ports: TCP port 443, port 853, port 15010, port 15012, port 15014 FROM <each_worker_node_publicIP>

Next steps

If you use load balancer services, ensure that all traffic that uses the VRRP protocol is allowed between worker nodes on the public and private interfaces. IBM Cloud Kubernetes Service uses the VRRP protocol to manage IP addresses for public and private load balancers.

Allow worker nodes to communicate with cluster master

To allow worker nodes to communicate with the cluster master over the private cloud service endpoint, allow outgoing network traffic from the source <each_worker_node_privateIP> to the destination TCP/UDP port range 30000-32767 and port 443, and the following IP addresses and network groups.

This table is moving. For the latest IP lists and continued updates, see the private network isolation folder in the IBM/kube-samples repo. You can Watch pull requests in the repo for updates.

• TCP/UDP port range 30000-32767, port 443 FROM <each_worker_node_privateIP> TO <private_IPs>
• Replace <private_IPs> with the private IP addresses of the region where your cluster is located.

Open ports

Open the following ports in your allowlist for your worker nodes to function properly. The following ports need should be open all destination IPs.

• Allow outbound TCP and UDP connections from the workers to ports 80 and 443 to allow worker node updates and reloads.
• Allow outbound TCP and UDP to port 2049 to allow mounting file storage as volumes.
• Allow outbound TCP and UDP to port 3260 for communication to block storage.
• Allow inbound TCP and UDP connections to port 10250 for the Kubernetes dashboard and commands such as kubectl logs and kubectl exec.
• Allow inbound and outbound connections to TCP and UDP port 53 for DNS access.

Enable worker-to-worker communication

Enable worker-to-worker communication by allowing all TCP, UDP, VRRP, and IPEncap traffic between worker nodes on the private interfaces and also allow VRRP on the public interface. IBM Cloud Kubernetes Service uses the VRRP protocol to manage IP addresses for load balancers and the IPEncap protocol to permit pod to pod traffic across subnets.

Permit worker nodes to communicate with IBM Cloud Container Registry

To permit worker nodes to communicate with IBM Cloud Container Registry, allow outgoing network traffic from the worker nodes to IBM Cloud Container Registry regions.

• TCP port 443 FROM <each_worker_node_privateIP> TO <registry_ip>
• Replace <registry_ip> with the registry IP address to which you want to allow traffic. The global registry stores IBM-provided public images, and regional registries store your own private or public images.

On 23 June 2022, only the br-sao and ca-tor regions changed. The remaining regions changed on 5 July 2022. For more information, see Container Registry private IP addresses changed on 5 July 2022.

Create persistent volume claims

To create persistent volume claims in a cluster where worker nodes are connected to private VLANs only, make sure that your cluster is set up with the following Kubernetes version or IBM Cloud storage plug-in versions. These versions enable private network communication from your cluster to your persistent storage instances.

If you must use a Kubernetes version or IBM Cloud storage plug-in version that does not support network communication over the private network, or if you want to use IBM Cloud Object Storage without HMAC authentication, allow egress access through your allowlist to IBM Cloud infrastructure and IBM Cloud Identity and Access Management:

• Allow all egress network traffic on TCP port 443.
• Allow access to the IBM Cloud infrastructure IP range for the zone that your cluster is in for both the Front-end (public) network and Back-end (private) Network. To find the zone of your cluster, run ibmcloud ks cluster ls.

Optional: Set up allowlist rules for IBM Log Analysis and IBM Cloud Monitoring services

To send logging and metric data, set up allowlist rules for your IBM Log Analysis and IBM Cloud Monitoring services.

• Log Analysis private endpoints
• Monitoring private endpoints

Getting your subnet IP addresses in the console

1. From the IBM Cloud console resource list, click on your cluster.
2. Click Worker nodes.
3. Note each Public VLAN used by the worker nodes in your cluster. Multiple worker nodes may use the same public VLAN.
4. From the IBM Cloud console menu , click Classic Infrastructure > IP Management > VLANs.
5. Click on each public VLAN to check if it is used by the worker nodes in your cluster.
6. For each public VLAN that is used by the worker nodes in your cluster, find the Subnets section and note each IP address included in the table. These are the IP addresses that you must include in your allowlist.

Getting your subnet IP addresses in the CLI

1. List the public VLANs your worker nodes use. The output is formatted as publicVLAN=<vlan_id>.

   kubectl describe nodes | grep publicVLAN | sort | uniq
   

2. For each public VLAN, find the associated public subnets. In the output, note the subnet IP addresses in the identifier column.

   ibmcloud sl subnet list | grep <vlan-id>
   

   Example output of subnets associated with public VLAN with ID 2761690.

   ID        identifier        type                 network_space   datacenter   vlan_id   IPs   hardware   virtual_servers  
   1962263   169.62.46.56      SECONDARY_ON_VLAN    PUBLIC          wdc07        2761690   8     0          0   
   2008207   169.62.39.248     SECONDARY_ON_VLAN    PUBLIC          wdc07        2761690   8     0          0   
   2342562   169.62.2.128      ADDITIONAL_PRIMARY   PUBLIC          wdc07        2761690   16    0          5 
   

3. List your worker nodes and get their public IPs. In the output, the public IPs are listed in the External-IP column. Note which of these IP addresses are contained in the subnets you listed earlier, and note those subnet IDs.

4. For each subnet ID, run the command to get the subnet details. In each output, note the IP address in the identifier column. This is the IP address you must add to the IAM allowlist.

   ibmcloud sl subnet detail <subnet_id>
   

   Example output.

   Name             Value   
   ID               2342562   
   identifier       169.62.2.128/28   
   subnet type      ADDITIONAL_PRIMARY   
   network space    PUBLIC   
   gateway          169.62.2.129   
   broadcast        169.62.2.143   
   datacenter       wdc07   
   usable ips       13   
   IP address       ID          IP address      
                   186531376   169.62.2.128      
                   186531378   169.62.2.129      
                   186531380   169.62.2.130      
                   186531382   169.62.2.131      
                   186531384   169.62.2.132      
                   186531386   169.62.2.133      
                   186531388   169.62.2.134      
                   186531390   169.62.2.135      
                   186531392   169.62.2.136      
                   186531394   169.62.2.137      
                   186531396   169.62.2.138      
                   186531398   169.62.2.139      
                   186531400   169.62.2.140      
                   186531402   169.62.2.141      
                   186531404   169.62.2.142      
                   186531406   169.62.2.143      
                       
   virtual guests   hostname                                                 domain    public_ip      private_ip      
                   kube-c8ofi5pw077drsbovf90-roks47class-default-00000187   iks.ibm   169.62.2.130   10.191.55.109      
                   kube-c8ofi5pw077drsbovf90-roks47class-default-000002a2   iks.ibm   169.62.2.133   10.191.55.108      
                   kube-c8ofi5pw077drsbovf90-roks47class-default-00000372   iks.ibm   169.62.2.135   10.191.55.121      
                   kube-c8ofh6kw0jj8l6jovf8g-iks22classi-default-00000219   iks.ibm   169.62.2.132   10.191.55.105      
                   kube-c8ofh6kw0jj8l6jovf8g-iks22classi-default-0000012f   iks.ibm   169.62.2.134   10.191.55.107