Choosing an app exposure service

Understanding load balancing for apps through Kubernetes service discovery

Kubernetes service discovery provides apps with a network connection by using network services and a local Kubernetes proxy.

All pods that are deployed to a worker node are assigned a private IP address in the 172.30.0.0/16 range and are routed between worker nodes only. To avoid conflicts, don't use this IP range on any nodes that communicate with your worker nodes. Worker nodes and pods can securely communicate on the private network by using private IP addresses. However, when a pod crashes or a worker node needs to be re-created, a new private IP address is assigned.

Instead of trying to track changing private IP addresses for apps that must be highly available, you can use built-in Kubernetes service discovery features to expose apps as services. A Kubernetes service groups a set of pods and provides a network connection to these pods. The service selects the targeted pods that it routes traffic to via labels.

A service provides connectivity between your app pods and other services in the cluster without exposing the actual private IP address of each pod. Services are assigned an in-cluster IP address, the clusterIP, that is accessible inside the cluster only. This IP address is tied to the service for its entire lifespan and does not change while the service exists. Services are assigned an IP from one of the 65,000 IPs in the 172.21.0.0/16 range.

To avoid conflicts, don't use this IP range on any nodes that communicate with your worker nodes. A DNS lookup entry is also created for the service and stored in the kube-dns component of the cluster. The DNS entry contains the name of the service, the namespace where the service was created, and the link to the assigned in-cluster IP address.

If you plan to connect your cluster to on-premises networks through IBM Cloud or a VPN service, you might have subnet conflicts with the default 172.30.0.0/16 range for pods and 172.21.0.0/16 range for services. You can avoid subnet conflicts when you create a cluster by specifying a custom subnet CIDR for pods in the --pod-subnet option and a custom subnet CIDR for services in the --service-subnet option.

To provide basic load balancing of all TCP and UDP network traffic for services, a local Kubernetes network proxy, kube-proxy, runs as a daemon on each worker node in the kube-system namespace. kube-proxy uses Iptables rules, a Linux kernel feature, to direct requests to the pod behind a service equally, independent of pods' in-cluster IP addresses and the worker node that they are deployed to.

For example, apps inside the cluster can access a pod behind a cluster service by using the service's in-cluster IP or by sending a request to the name of the service. When you use the name of the service, kube-proxy looks up the name in the cluster DNS provider and routes the request to the in-cluster IP address of the service.

If you use a service that provides both an internal cluster IP address and an external IP address, clients outside of the cluster can send requests to the service's external public or private IP address. kube-proxy forwards the requests to the service's in-cluster IP address and load balances between the app pods behind the service.

Understanding Kubernetes service types

Kubernetes supports four basic types of network services: ClusterIP, NodePort, LoadBalancer, and Ingress. ClusterIP services make your apps accessible internally to allow communication between pods in your cluster only. NodePort, LoadBalancer, and Ingress services make your apps externally accessible from the public internet or a private network.

The following table compares the features of each network service type.

* An SSL certificate for HTTPS load balancing is provided by ibmcloud ks nlb-dns commands. In classic clusters, these commands are supported for public NLBs only.

NodePort

When you expose apps with a NodePort service, a NodePort in the range of 30000 - 32767 and an internal cluster IP address is assigned to the service. To access the service from outside the cluster, you use the public or private IP address of any worker node and the NodePort in the format <IP_address>:<nodeport>. However, the public and private IP addresses of the worker node are not permanent. When a worker node is removed or re-created, a new public and a new private IP address are assigned to the worker node.

NodePorts are ideal for testing public or private access or providing access for only a short amount of time. Note: Because worker nodes in VPC clusters don't have a public IP address, you can access an app through a NodePort only if you are connected to your private VPC network, such as through a VPN connection.

LoadBalancer

The LoadBalancer service type is implemented differently depending on your cluster's infrastructure provider.

LoadBalancer services in Classic clusters

Classic infrastructure

Network load balancer (NLB). Every standard cluster is provisioned with four portable public and four portable private IP addresses that you can use to create a layer 4 TCP/UDP network load balancer (NLB) for your app. You can customize your NLB by exposing any port that your app requires. The portable public and private IP addresses that are assigned to the NLB are permanent and don't change when a worker node is re-created in the cluster. You can create a subdomain for your app that registers public NLB IP addresses with a DNS entry. You can also enable health check monitors on the NLB IPs for each subdomain.

LoadBalancer services in VPC clusters

Virtual Private Cloud

Load Balancer for VPC. When you create a Kubernetes LoadBalancer service for an app in your cluster, a layer 7 VPC load balancer is automatically created in your VPC outside of your cluster. The VPC load balancer is multizonal and routes requests for your app through the private NodePorts that are automatically opened on your worker nodes. By default, the load balancer is also created with a hostname that you can use to access your app.

Ingress

Expose multiple apps in a cluster by setting up routing with the Ingress application load balancer (ALB). The ALB uses a secured and unique public or private entry point, an Ingress subdomain, to route incoming requests to your apps. You can use one subdomain to expose multiple apps in your cluster as services. Ingress consists of three components:

• The Ingress resource defines the rules for how to route and load balance incoming requests for an app.
• The ALB listens for incoming HTTP, HTTPS, or TCP service requests. It forwards requests across the apps' pods based on the rules that you defined in the Ingress resource.
• The multizone load balancer (MZLB) for classic clusters or the VPC load balancer for VPC clusters handles all incoming requests to your apps and load balances the requests among the ALBs in the various zones. It also enables health checks for the public Ingress IP addresses.

Planning public external load balancing

Publicly expose an app in your cluster to the internet.

In classic clusters, you can connect worker nodes to a public VLAN. The public VLAN determines the public IP address that is assigned to each worker node, which provides each worker node with a public network interface. Public networking services connect to this public network interface by providing your app with a public IP address and, optionally, a public URL.

In VPC clusters, your worker nodes are connected to private VPC subnets only. However, when you create public networking services, a VPC load balancer is automatically created. The VPC load balancer can route public requests to your app by providing your app a public URL. When an app is publicly exposed, anyone that has the public URL can send a request to your app.

When an app is publicly exposed, anyone that has the public service IP address or the URL that you set up for your app can send a request to your app. For this reason, expose as few apps as possible. Expose an app to the public only when your app is ready to accept traffic from external web clients or users.

The public network interface for worker nodes is protected by predefined Calico network policy settings that are configured on every worker node during cluster creation. By default, all outbound network traffic is allowed for all worker nodes. Inbound network traffic is blocked except for a few ports. These ports are opened so that IBM can monitor network traffic and automatically install security updates for the Kubernetes master, and so that connections can be established to NodePort, LoadBalancer, and Ingress services. For more information about these policies, including how to modify them, see Network policies.

Choosing a deployment pattern for classic clusters

To make an app publicly available to the internet in a classic cluster, choose a load balancing deployment pattern that uses public NodePort, LoadBalancer, or Ingress services. The following table describes each possible deployment pattern, why you might use it, and how to set it up. For basic information about the networking services that these deployment patterns use, see Understanding Kubernetes service types.

Choosing a deployment pattern for VPC clusters

To make an app publicly available to the internet in a VPC cluster, choose a load balancing deployment pattern that uses public LoadBalancer or Ingress services. The following table describes each possible deployment pattern, why you might use it, and how to set it up. For basic information about the networking services that these deployment patterns use, see Understanding Kubernetes service types.

Planning private external load balancing

Privately expose an app in your cluster to the private network only.

When you deploy an app in a Kubernetes cluster in IBM Cloud Kubernetes Service, you might want to make the app accessible to only users and services that are on the same private network as your cluster. Private load balancing is ideal for making your app available to requests from outside the cluster without exposing the app to the general public. You can also use private load balancing to test access, request routing, and other configurations for your app before you later expose your app to the public with public network services.

As an example, say that you create a private load balancer for your app. This private load balancer can be accessed by:

• Any pod in that same cluster.
• Any pod in any cluster in the same IBM Cloud account.
• If you're not in the IBM Cloud account but still behind the company firewall, any system through a VPN connection to the subnet that the load balancer IP is on.
• If you're in a different IBM Cloud account, any system through a VPN connection to the subnet that the load balancer IP is on.
• In classic clusters, if you have VRF or VLAN spanning enabled, any system that is connected to any of the private VLANs in the same IBM Cloud account.
• In VPC clusters:
  • If you permit traffic between VPC subnets, any system in the same VPC.
  • If you permit traffic between VPCs, any system that has access to the VPC that the cluster is in.