Security for IBM Cloud Kubernetes Service

You can use built-in security features in IBM Cloud® Kubernetes Service for risk analysis and security protection. These features help you to protect your cluster infrastructure and network communication, isolate your compute resources, and ensure security compliance across your infrastructure components and container deployments.

For an analysis of security guidelines by product version, see CIS Kubernetes Benchmark.

Overview of security threats for your cluster

To protect your cluster from being compromised, you must understand potential security threats for your cluster and what you can do to reduce the exposure to vulnerabilities.

External attacks: Attackers that gain access to your cluster, deployed resources, apps, or personal information.
Vulnerable deployments: Known vulnerabilities are exploited to gain access to the cloud environment and run malicious software.
Compromised or lost data: Incorrect storage of sensitive data and missing encryption.
Insiders and third party vendors: Missing network isolation and segmentation can lead to the misuse of legitimate permissions.

Cloud security and the protection of your systems, infrastructure, and data against attacks became very important over the last couple of years as companies continue to move their workloads into the public cloud. A cluster consists of several components that each can put your environment at risk for malicious attacks. To protect your cluster against these security threats, you must make sure that you apply the latest IBM Cloud Kubernetes Service and Kubernetes security features and updates in all cluster components.

These components include:

Kubernetes API server and etcd data store
Worker nodes
Network
Persistent storage
Monitoring and Logging
Container images and registry
Container isolation and security
Personal information

Kubernetes API server and etcd

The Kubernetes API server and etcd data store are the most sensitive components that run in your Kubernetes master. You want to prevent unauthorized access to these components because they set and store the configurations for all the resources that run in your cluster, including some security settings of your applications.

Kubernetes provides security controls and limits access to protect these components and reduce the risk of cyber attacks.

How is access to my API server granted?

By default, Kubernetes requires every request to go through several stages before access to the API server is granted.

Authentication: Validates the identity of a registered user or service account.
Authorization: Limits the permissions of authenticated users and service accounts to ensure that they can access and operate only the cluster components that you want them to.
Admission control: Validates or mutates requests before they are processed by the Kubernetes API server. Many Kubernetes features require admission controllers to properly function.

What does IBM Cloud Kubernetes Service do to secure my API server and etcd data store?

The following image shows the default cluster security settings that address authentication, authorization, admission control, and secure connectivity between the Kubernetes master and worker nodes.

Describes the security stages when you access the Kubernetes API server. — Security stages when accessing the Kubernetes API server

Review the following security features for Kubernetes API server and etcd.

Fully managed and dedicated Kubernetes master

Every cluster in IBM Cloud Kubernetes Service is controlled by a dedicated Kubernetes master that is managed by IBM in an IBM-owned IBM Cloud account. The Kubernetes master is set up with the following dedicated components that are not shared with other IBM customers.

etcd data store: Stores all Kubernetes resources of a cluster, such as Services, Deployments, and Pods. Kubernetes ConfigMaps and Secrets are app data that is stored as key value pairs so that they can be used by an app that runs in a pod. Data in etcd is stored on the local disk of the Kubernetes master and is backed up to IBM Cloud Object Storage. Data is encrypted during transit to IBM Cloud Object Storage and at rest. You can choose to enable encryption for your etcd data on the local disk of your Kubernetes master by enabling IBM Key Protect encryption for your cluster. When etcd data is sent to a pod, data is encrypted via TLS to ensure data protection and integrity.
kube-apiserver: Serves as the main entry point for all cluster management requests from the worker node to the Kubernetes master. The API server validates and processes requests that change the state of cluster resources, such as pods or services, and stores this state in the etcd data store.
kube-scheduler: Decides where to deploy pods, considering the capacity and performance needs, hardware and software policy constraints, anti-affinity specifications, and workload requirements. If no worker node can be found that matches the requirements, the pod is not deployed in the cluster.
kube-controller-manager: Responsible for monitoring replica sets, and creating corresponding pods to achieve the specified state.
Konnectivity: IBM Cloud Kubernetes Service-specific component to provide secured network connectivity for all Kubernetes master to worker node communication. The Konnectivity server works with the Konnectivity agent to securely connect the master to the worker node. This connection supports apiserver proxy requests to your pods and services, and kubectl top, exec, attach, and logs requests to the kubelet. The connection from the worker nodes to the master is automatically secured with TLS certificates.

Continuous monitoring by IBM Site Reliability Engineers (SREs)

The Kubernetes master, including all the master components, compute, networking, and storage resources are continuously monitored by IBM Site Reliability Engineers (SREs). The SREs apply the latest security standards, detect and remediate malicious activities, and work to ensure reliability and availability of IBM Cloud Kubernetes Service.

CIS Kubernetes master benchmark

To configure IBM Cloud Kubernetes Service, IBM engineers follow relevant cybersecurity practices from the Kubernetes master benchmark that is published by the Center of Internet Security (CIS). The cluster master and all worker nodes are deployed with images that meet the benchmark.

Secure communication via TLS

To use IBM Cloud Kubernetes Service, you must authenticate with the service by using your credentials. When you are authenticated, IBM Cloud Kubernetes Service generates TLS certificates that encrypt the communication to and from the Kubernetes API server and etcd data store to ensure a secure end-to-end communication between the worker nodes and the Kubernetes master. These certificates are never shared across clusters or across Kubernetes master components.

Connectivity to worker nodes

Although Kubernetes secures the communication between the master and worker nodes by using the https protocol, no authentication is provided on the worker node by default. To secure this communication, IBM Cloud Kubernetes Service automatically sets up an Konnectivity connection between the Kubernetes master and the worker node when the cluster is created.

Fine-grained access control

As the account administrator you can grant access to other users for IBM Cloud Kubernetes Service by using IBM Cloud Identity and Access Management (IAM). IBM Cloud IAM provides secure authentication with the IBM Cloud platform, IBM Cloud Kubernetes Service, and all the resources in your account. Setting up proper user roles and permissions is key to limiting who can access your resources and to limiting the damage that a user can do when legitimate permissions are misused. You can select from the following pre-defined user roles that determine the set of actions that the user can perform:

Platform access roles: Determine the cluster and worker node management-related actions that a user can perform in IBM Cloud Kubernetes Service.
Service access roles: Determine the Kubernetes RBAC role that is assigned to the user and the actions that a user can run against the Kubernetes API server. With RBAC roles, users can create Kubernetes resources, such as app deployments, namespaces, or configmaps. For more information about the corresponding RBAC roles that are assigned to a user and associated permissions, see IAM service access roles.
Classic infrastructure: Enables access to your classic infrastructure resources. Example actions that are permitted by classic infrastructure roles are viewing the details of cluster worker node machines or editing networking and storage resources.
VPC infrastructure: Enables access to VPC infrastructure resources. Example actions that are permitted by VPC infrastructure roles are creating a VPC, adding subnets, changing floating IP addresses, and creating VPC Block Storage instances.

For more information about access control in a cluster, see [Assigning cluster access](/docs/containers?topic=containers-iam-platform-access-roles.

Pod access via service account token

For clusters that run Kubernetes 1.21 and later, the service account tokens that pods use to communicate with the Kubernetes API server are time-limited, automatically refreshed, scoped to a particular audience of users (the pod), and invalidated after the pod is deleted. To continue communicating with the API server, you must design your apps to read the refreshed token value on a regular basis, such as every minute. For more information, see Bound Service Account Tokens.

Admission controllers

Admission controllers are implemented for specific features in Kubernetes and IBM Cloud Kubernetes Service. With admission controllers, you can set up policies in your cluster that determine whether a particular action in the cluster is allowed or not. In the policy, you can specify conditions when a user can't perform an action, even if this action is part of the general permissions that you assigned the user by using RBAC roles. Therefore, admission controllers can provide an extra layer of security for your cluster before an API request is processed by the Kubernetes API server.

When you create a cluster, IBM Cloud Kubernetes Service automatically installs the default Kubernetes admission controllers in a particular order in the Kubernetes master, which can't be changed by the user. Review the order of default admission controllers by cluster version in the kube-apiserver component reference information.

You can install your own admission controllers in the cluster or choose an optional admission controller, such as Portieris. With Portieris, you can block container deployments from unsigned images.

If you manually installed admission controllers and you don't want to use them anymore, make sure to remove them entirely. If admission controllers are not entirely removed, they might block all actions that you want to perform on the cluster.

What else can I do to secure my API server?

You can restriction network connectivity to your cluster master in several ways

Enable only the private cloud service endpoint: The public service endpoint is only required for classic Openshift clusters. It can be disabled for all VPC clusters. It can also be disabled for classic Kubernetes clusters as long as your account has VRF and Service Endpoint enabled. This protects your cluster master from attacks on the public network.
Enable context based restrictions: You can secure network access to your cluster's private and/or public service endpoints using context based restrictions (CBR). Only authorized requests to your cluster master that originate from subnets in the CBR rules are permitted. For more information, see Using context based restrictions.

Worker node

Worker nodes carry the deployments and services that make up your app. When you host workloads in the public cloud, you want to ensure that your app is protected from being accessed, changed, or monitored by an unauthorized user or software.

Who owns the worker node and am I responsible to secure it?

The ownership of a worker node depends on the type of cluster that you create and the infrastructure provider that you choose.

Classic clusters: Worker nodes are provisioned in to your IBM Cloud account. The worker nodes are dedicated to you and you are responsible to request timely updates to the worker nodes to ensure that the worker node OS and IBM Cloud Kubernetes Service components apply the latest security updates and patches.
VPC clusters: Worker nodes are provisioned in to an IBM Cloud account that is owned by IBM to enable monitoring of malicious activities and apply security updates. You can't access your worker nodes by using the VPC dashboard. However, you can manage your worker nodes by using the IBM Cloud Kubernetes Service console, CLI, or API. The virtual machines that make up your worker nodes are dedicated to you and you are responsible to request timely updates so that your worker node OS and IBM Cloud Kubernetes Service components apply the latest security updates and patches.

For more information, see Your responsibilities by using IBM Cloud Kubernetes Service.

Use the ibmcloud ks worker update command regularly (such as monthly) to deploy updates and security patches to the operating system and to update the Kubernetes version that your worker nodes run. When updates are available, you are notified when you view information about the master and worker nodes in the IBM Cloud console or CLI, such as with the ibmcloud ks clusters ls or ibmcloud ks workers ls --cluster <cluster_name> commands. Worker node updates are provided by IBM as a full worker node image that includes the latest security patches. To apply the updates, the worker node must be reimaged and reloaded with the new image. Keys for the root user are automatically rotated when the worker node is reloaded.

How does my worker node setup look?

The following image shows the components that are set up for every worker node to protect your worker node from malicious attacks.

The image does not include components that ensure secure end-to-end communication to and from the worker node. For more information, see network security.

CIS-compliant Linux image: Every worker node is set up with an Ubuntu operating system that implements the benchmarks that are published by the Center of Internet Security (CIS). The user or the owner of the machine can't change this operating system to another operating system. To review the current Ubuntu version, run kubectl get nodes -o wide. IBM works with internal and external security advisory teams to address potential security compliance vulnerabilities. Security updates and patches for the operating system are made available through IBM Cloud Kubernetes Service and must be installed by the user to keep the worker node secure. IBM Cloud Kubernetes Service uses an Ubuntu Linux kernel for worker nodes. You can run containers based on any Linux distribution in IBM Cloud Kubernetes Service. Check with your container image vendor to verify that your container images can run on an Ubuntu kernel.
Continuous monitoring by Site Reliability Engineers (SREs): The image that is installed on your worker nodes is continuously monitored by IBM Site Reliability Engineers (SREs) to detect vulnerabilities and security compliance issues. To address vulnerabilities, SREs create security patches and fix packs for your worker nodes. Make sure to apply these patches when they are available to ensure a secure environment for your worker nodes and the apps that you run on them.
CIS Kubernetes worker node benchmark: To configure IBM Cloud Kubernetes Service, IBM engineers follow relevant cybersecurity practices from the Kubernetes worker node benchmark that is published by the Center of Internet Security (CIS). You can review the compliance of worker nodes against CIS Kubernetes benchmark standards.
Compute isolation: Worker nodes are dedicated to a cluster and don't host workloads of other clusters. When you create a standard classic cluster, you can choose to provision your worker nodes as physical machines (bare metal) or as virtual machines that run on shared or dedicated physical hardware. Worker nodes in a VPC cluster can be provisioned as virtual machines on shared infrastructure only.
Option to deploy bare metal on classic: If you create a standard classic cluster, you can choose to provision your worker nodes on bare metal physical servers (instead of virtual server instances). With bare metal machines, you have additional control over the compute host, such as the memory or CPU. This setup eliminates the virtual machine hypervisor that allocates physical resources to virtual machines that run on the host. Instead, all a bare metal machine's resources are dedicated exclusively to the worker, so you don't need to worry about "noisy neighbors" sharing resources or slowing down performance. Bare metal servers are dedicated to you, with all its resources available for cluster usage.
Encrypted disks: By default, every worker node is provisioned with two local SSD, AES 256-bit encrypted data partitions. The first partition contains the kernel image that is used to boot the worker node and is not encrypted. The second partition holds the container file system and is unlocked by using LUKS encryption keys. Each worker node in a cluster has its own unique LUKS encryption key, managed by IBM Cloud Kubernetes Service. When you create a cluster or add a worker node to an existing cluster, the keys are pulled securely and then discarded after the encrypted disk is unlocked. Encryption can impact disk I/O performance. For workloads that require high-performance disk I/O, test a cluster with encryption both enabled and disabled to help you decide whether to turn off encryption.
Expert AppArmor policies: Every worker node is set up with security and access policies that are enforced by AppArmor profiles that are loaded into the worker node during bootstrapping. AppArmor profiles can't be changed by the user or owner of the machine.
SSH disabled: By default, SSH access is disabled on the worker node to protect your cluster from malicious attacks. When SSH access is disabled, access to the cluster is forced via the Kubernetes API server. The Kubernetes API server requires every request to be checked against the policies that are set in the authentication, authorization, and admission control module before the request is executed in the cluster.; If you have a standard cluster and you want to install more features on your worker node, you can choose between the add-ons that are provided by IBM Cloud Kubernetes Service or use Kubernetes daemon sets for everything that you want to run on every worker node. For any one-time action that you must execute, use Kubernetes jobs.

Network

The classic approach to protect a company's network is to set up a firewall and block any unwanted network traffic to your apps. While this is still true, research shows that many malicious attacks come from insiders or authorized users who misuse their assigned permissions.

Network segmentation and privacy for classic clusters

To protect your network and limit the range of damage that a user can do when access to a network is granted, you must make sure that your workloads are as isolated as possible and that you limit the number of apps and worker nodes that are publicly exposed.

What network traffic is allowed for my Classic cluster by default?

All containers are protected by predefined Calico network policy settings that are configured on every worker node during cluster creation. By default, all outbound network traffic is allowed for all worker nodes. Inbound network traffic is blocked with the following exceptions:

NodePort: The Kubernetes NodePort range is opened by default so that you can expose apps with NodePort services. To block inbound network traffic on NodePorts in your cluster, see Controlling inbound traffic to NLB or NodePort services.
IBM monitoring ports: By default, IBM opens a few ports on your cluster so that network traffic can be monitored by IBM and for IBM to automatically install security updates for the Kubernetes master.

Access from the Kubernetes master to the worker node's kubelet is secured by a Konnectivity tunnel. For more information, see the IBM Cloud Kubernetes Service architecture.

What is network segmentation and how can I set it up for a Classic cluster?

Network segmentation describes the approach to divide a network into multiple subnetworks. You can group apps and related data to be accessed by a specific group in your organization. Apps that run in one subnetwork can't see or access apps in another subnetwork. Network segmentation also limits the access that is provided to an insider or third-party software and can limit the range of malicious activities.

IBM Cloud Kubernetes Service provides IBM Cloud VLANs that ensure quality network performance and network isolation for worker nodes. A VLAN configures a group of worker nodes and pods as if they were attached to the same physical wire. VLANs are dedicated to your IBM Cloud account and not shared across IBM customers. In classic clusters, if you have multiple VLANs for your cluster, multiple subnets on the same VLAN, or a multizone classic cluster, you must enable a Virtual Router Function (VRF) for your IBM Cloud infrastructure account so your worker nodes can communicate with each other on the private network. To enable VRF, see Enabling VRF. To check whether a VRF is already enabled, use the ibmcloud account show command. If you can't or don't want to enable VRF, enable VLAN spanning. To perform this action, you need the Network > Manage Network VLAN Spanning infrastructure permission, or you can request the account owner to enable it. To check whether VLAN spanning is already enabled, use the ibmcloud ks vlan spanning get --region <region> command.

When you enable VRF or VLAN spanning for your account, network segmentation is removed for your clusters.

Review the following table to see your options for how to achieve network segmentation when you enable VRF or VLAN spanning for your account.

Network segmentation options
Security feature	Description
Set up custom network policies with Calico	You can use the built-in Calico interface to set up custom Calico network policies for your worker nodes. For example, you can allow or block network traffic on specific network interfaces, for specific pods, or services. To set up custom network policies, you must install the `calicoctl` CLI.
Support for IBM Cloud network firewalls	IBM Cloud Kubernetes Service is compatible with all IBM Cloud firewall offerings. For example, you can set up a firewall with custom network policies to provide dedicated network security for your standard cluster and to detect and remediate network intrusion. For example, you might choose to set up a Virtual Router Appliance to act as your firewall and block unwanted traffic. When you set up a firewall, you must also open up the required ports and IP addresses for each region so that the master and the worker nodes can communicate.

What else can I do to reduce the surface for external attacks for Classic clusters?

The more apps or worker nodes that you expose publicly, the more steps you must take to prevent external malicious attacks. Review the following table to find options for how to keep apps and worker nodes private.

Private services and worker node options
Security feature	Description
Limit the number of exposed apps	By default, your apps and services that run within the cluster are not reachable over the public internet. You can choose if you want to expose your apps to the public, or if you want your apps and services be reachable on the private network only. When you keep your apps and services private, you can leverage the built-in security features to assure secured communication between worker nodes and pods. To expose services and apps to the public internet, you can leverage the NLB and Ingress ALB support to securely make your services publicly available. Ensure that only necessary services are exposed, and revisit the list of exposed apps regularly to ensure that they are still valid.
Keep worker nodes private	When you create a cluster, every cluster is automatically connected to a private VLAN. The private VLAN determines the private IP address that is assigned to a worker node. You can choose to keep your worker nodes private by connecting them to a private VLAN only. Keep in mind that to communicate with the Kubernetes master from your local machine and for IBM Cloud Kubernetes Service to connect to IBM Cloud services that don't support a private cloud service endpoint, you must configure public connectivity to specific URLs and IP addresses. If the IBM Cloud services that you want to connect to have a private cloud service endpoint and your account is enabled for VRF, network traffic to and from these services is automatically routed over the private network and no public network connection is required. To set up public connectivity for cluster that is connected to a private VLAN only, you can configure a firewall, such as a Virtual Router Appliance, in front of your worker nodes and enable network traffic to these URLs and IP addresses.
Limit public internet connectivity with edge nodes	If you don't create a cluster with a gateway enabled, every worker node is configured to accept app pods and associated load balancer or ingress pods. You can label worker nodes as edge nodes to force load balancer and Ingress pods to be deployed to these worker nodes only. In addition, you can taint your worker nodes so that app pods can't schedule onto the edge nodes. With edge nodes, you can isolate the networking workload on fewer worker nodes in your cluster and keep other worker nodes in the cluster private.

What if I want to connect my cluster to an on-prem data center?

To connect your worker nodes and apps to an on-prem data center, you can configure a VPN IPSec endpoint with a strongSwan service, a Virtual Router Appliance, or with a Fortigate Security Appliance.

Network segmentation and privacy for VPC clusters

What network traffic is allowed for my VPC cluster by default?

By default, worker nodes are connected to VPC subnets on the private network only and don't have a public network interface. All public ingress to your worker nodes is blocked. Public egress from your worker nodes is only allowed if the workers are connected to a VPC subnet that has a public gateway.

If your worker nodes must access a public endpoint outside of the cluster, you can attach a public gateway to the VPC subnet that the worker nodes are deployed to. For example, your VPC cluster can automatically connect to other IBM Cloud services that support private cloud service endpoints, such as IBM Cloud Container Registry. However, if you need to access IBM Cloud services that support only public cloud service endpoints, you can attach a public gateway to the subnet so that your pods can send requests over the public network. All egress is permitted for worker nodes on a subnet with an attached public gateway, but all ingress is still blocked.

If you deploy apps in your cluster that must receive traffic requests from the internet, you can create a VPC load balancer to expose your apps. To allow ingress network traffic to your apps, you must configure your VPC load balancer for the ingress network traffic that you want to receive.

Security groups are applied to your VPC instance and VPC ALBs by default. For more information, see Controlling traffic with VPC security groups.

What is network segmentation and how can I set it up for a VPC cluster?

IBM Cloud Kubernetes Service provides IBM Cloud VPC subnets that ensure quality network performance and network isolation for worker nodes. A VPC subnet consists of a specified private IP address range (CIDR block) and configures a group of worker nodes and pods as if they were attached to the same physical wire. VPC subnets are dedicated to your IBM Cloud account and not shared across IBM customers.

VPC subnets provide a channel for connectivity among the worker nodes within the cluster. Any system that is connected to any of the private subnets in the same VPC can communicate with workers. For example, all subnets in one VPC can communicate through private layer 3 routing with a built-in VPC router. If your clusters don't need to communicate, you can achieve the best network segmentation by creating the clusters in separate VPCs. If you have multiple clusters that must communicate with each other, you can create the clusters in the same VPC. Although subnets within one VPC can be shared by multiple clusters in that VPC, you can achieve better network segmentation by using different subnets for clusters within one VPC.

To achieve further private network segmentation between VPC subnets for your account, you can set up custom network policies with VPC access control lists (ACLs). When you create a VPC, a default ACL is created in the format allow-all-network-acl-<VPC_ID> for the VPC. Any subnet that you create in the VPC is attached to this ACL by default. The ACL includes an inbound rule and an outbound rule that allow all traffic between your worker nodes on a subnet and any system on the subnets in the same VPC. If you want to specify which private network traffic is permitted to the worker nodes on your VPC subnets, you can create a custom ACL for each subnet in the VPC. For example, you can create a set of ACL rules to block most inbound and outbound private network traffic of a cluster, while allowing communication that is necessary for the cluster to function.

What else can I do to reduce the surface for external attacks for VPC clusters?

VPC network security options
Security feature	Description
Limit the number of exposed apps	By default, your apps and services that run within the cluster are not reachable over the public internet. You can choose if you want to expose your apps to the public, or if you want your apps and services be reachable on the private network only. When you keep your apps and services private, you can leverage the built-in security features to assure secured communication between worker nodes and pods. To expose services and apps to the public internet, you can leverage the VPC load balancer and Ingress ALB support to securely make your services publicly available. Ensure that only necessary services are exposed, and revisit the list of exposed apps regularly to ensure that they are still valid.
Limit public network egress to one subnet with a public gateway	If pods on your worker nodes need to connect to a public external endpoint, you can attach a public gateway to the subnet that those worker nodes are on. You can isolate this network traffic in your cluster by attaching a public gateway to only one subnet in your cluster. Then, you can set affinity rules to deploy app pods that require access to external endpoints to only the subnet with an attached public gateway.

Depending on the network that you want to connect your worker nodes to, you can choose a VPN solution.

Securely expose apps with LoadBalancer and Ingress services

You can use network load balancer (NLB) and Ingress application load balancer (ALB) networking services to connect your apps to the public internet or to external private networks. Review the following optional settings for NLBs and ALBs that you can use to meet back-end app security requirements or encrypt traffic as it moves through your cluster.

Can I use security groups to manage my cluster's network traffic?

Classic clusters: IBM Cloud security groups are applied to the network interface of a single virtual server to filter traffic at the hypervisor level. If you want to manage traffic for each worker node, you can use security groups. When you create a security group, you must allow the VRRP protocol, which IBM Cloud Kubernetes Service uses to manage NLB IP addresses. To uniformly manage traffic for your cluster across all your worker nodes, use Calico and Kubernetes policies.

VPC clusters: VPC security groups are applied to the network interface of a single virtual server to filter traffic at the hypervisor level. You can add inbound and outbound rules to the default security group for your cluster to manage inbound and outbound traffic to a VPC cluster. For more information about the security groups, including the default settings, see Controlling traffic with VPC security groups.

Because the worker nodes of your VPC cluster exist in a service account and are not listed in the VPC infrastructure dashboard, you can't create a security group and apply it to your worker node instances. You can only modify existing security groups that are created for you.

How can I secure the source IP within the cluster?

In version 2.0 NLBs, the source IP address of the client request is preserved by default. However, in version 1.0 NLBs and in all Ingress ALBs, the source IP address of the client request is not preserved. When a client request to your app is sent to your cluster, the request is routed to a pod for the NLB 1.0 or ALB. If no app pod exists on the same worker node as the load balancer service pod, the NLB or ALB forwards the request to an app pod on a different worker node. The source IP address of the package is changed to the public IP address of the worker node where the app pod runs.

Preserving the client’s IP is useful, for example, when app servers have to apply security and access-control policies. To preserve the original source IP address of the client request, you can enable source IP preservation for version 1.0 NLBs or Ingress ALBs.

How can I do TLS termination with LoadBalancer and Ingress services?

The Ingress service offers TLS termination at two points in the traffic flow:

Decrypt package upon arrival: By default, the Ingress ALB load balances HTTP network traffic to the apps in your cluster. To also load balance incoming HTTPS connections, you can configure the ALB to decrypt the network traffic and forward the decrypted request to the apps that are exposed in your cluster. If you use the IBM-provided Ingress subdomain, you can use the IBM-provided TLS certificate. If you use a custom domain, you can use your own TLS certificate to manage TLS termination.
Re-encrypt package before you forward it to upstream apps: The ALB decrypts HTTPS requests before forwarding traffic to your apps. If you have apps that require HTTPS and need traffic to be encrypted before it is forwarded to those upstream apps, you can use the ssl-services annotation. If your upstream apps can handle TLS, you can optionally provide a certificate that is contained in a one-way or mutual-authentication TLS secret.

To secure the service-to-service communication, you can use Istio's mutual TLS authentication. Istio is an open source service that gives developers a way to connect, secure, manage, and monitor a network of microservices, also known a service mesh, on cloud orchestration platforms like Kubernetes.

Persistent storage

Review supported options for encrypting and protecting your data on persistent storage in IBM Cloud.

By default, all IBM Cloud storage solutions automatically encrypt your data at rest with an IBM-managed encryption key at no additional cost. For more information, see the following links.

Depending on the type of storage that you choose, you can set up additional encryption with IBM Key Protect to protect your data in transit and at rest with your own encryption key.

You can also use an IBM Cloud database service, such as IBM Cloudant NoSQL DB, to persist data in a managed database outside the cluster. Data that is stored with a cloud database service can be accessed across clusters, zones, and regions. For security-related information, see the database service-specific IBM Cloud documentation.

Monitoring and logging

The key to detect malicious attacks in your cluster is the proper monitoring and logging of metrics and all the events that happen in the cluster. Monitoring and logging can also help you understand the cluster capacity and availability of resources for your app so that you can plan accordingly to protect your apps from a downtime.

Does IBM monitor my cluster?

Every cluster master is continuously monitored by IBM to control and remediate process level Denial-Of-Service (DOS) attacks. IBM Cloud Kubernetes Service automatically scans every node where the master is deployed for vulnerabilities that are found in Kubernetes and OS-specific security fixes. If vulnerabilities are found, IBM Cloud Kubernetes Service automatically applies fixes and resolves vulnerabilities on behalf of the user to ensure master node protection.

What information is logged?

By default, IBM Cloud Kubernetes Service automatically collects logs for the following cluster components:

Containers: Logs that are written to STDOUT or STDERR.
Apps: Logs that are written to a specific path inside your app.
Workers: Logs from the Ubuntu operating system that are sent to /var/log/syslog and /var/log/auth.log.
Kubernetes API server: Every cluster-related action that is sent to the Kubernetes API server is logged for auditing reasons, including the time, the user, and the affected resource. For more information, see Kubernetes audit logs. You can access these logs by using IBM Cloud Activity Tracker. For more information, see the getting started tutorial.
Ingress: Logs for an Ingress application load balancer (ALB) that manages inbound network traffic.
Kubernetes system components: Logs from the kubelet, the kube-proxy, and other components that run in the kube-system namespace.

To access cluster component logs, you can choose to forward your logs to IBM Log Analysis, an external server, or a third-party logging solution. For more information, see Choosing a logging solution.

How can I monitor the health and performance of my cluster?: You can verify the health, capacity, and performance of your apps, services, and worker nodes by monitoring your cluster components and compute resources from the IBM Cloud Kubernetes Service console or CLI, such as the CPU and memory usage. To view more in-depth metrics for a standard cluster or your apps, you can configure a monitoring agent in your cluster to send metrics to IBM Cloud Monitoring. You can also install third-party monitoring solutions, such as Prometheus, or use the metrics that are provided in the Kubernetes dashboard. For more information, see Choosing a monitoring solution.

To set up a host-based intrusion detection system (HIDS) and security event log monitoring (SELM), install third-party tools that are designed to monitor your cluster and containerized apps to detect intrusion or misuse, such as Twistlock or the Sysdig Falco project.

How can I audit events that happen in my cluster?: You can set up IBM Cloud® Activity Tracker in your IBM Cloud Kubernetes Service cluster. For more information, view the Activity Tracker documentation.
What are my options to enable trust in my cluster?: By default, IBM Cloud Kubernetes Service provides many features for your cluster components so that you can deploy your containerized apps in a security-rich environment. Extend your level of trust in your cluster to better ensure that what happens within your cluster is what you intended to happen. You can implement trust in your cluster in various ways, as shown in the following diagram.

Deploying containers with trusted content. — Deploying containers with trusted content

Content Trust for your images: Ensure the integrity of your images by enabling content trust in your IBM Cloud Container Registry. With trusted content, you can control who can sign images as trusted. After trusted signers push an image to your registry, users can pull the signed content so that they can verify the source of the image. For more information, see Signing images for trusted content.
Container Image Security Enforcement: Use an admission controller with custom policies so that you can verify container images before you deploy them. With a container image security enforcement project like Portieris, you control where the images are deployed from and ensure that they meet content trust requirements. If a deployment does not meet the policies that you set, security enforcement prevents modifications to your cluster.
Image Vulnerability Scanner: By default, Vulnerability Advisor scans images that are stored in IBM Cloud Container Registry to find potential security vulnerabilities. For more information, see Managing image security with Vulnerability Advisor.
IBM Cloud Security and Compliance Center: When you enable IBM Cloud Security and Compliance Center, you can view reports about suspicious incoming and outgoing network traffic. For more information, see What is IBM Cloud Security and Compliance Center?.
IBM Cloud® Secrets Manager: You can store your Ingress and Kubernetes secrets in IBM Cloud® Secrets Manager. When you integrate Secrets Manager into your cluster, you set a default Secrets Manager instance where all Ingress subdomain secrets are uploaded. For more information, see Setting up Secrets Manager in your Kubernetes Service cluster.

Image and registry

Every deployment is based on an image that holds the instructions for how to spin up the container that runs your app. These instructions include the operating system inside the container and extra software that you want to install. To protect your app, you must protect the image and establish checks to ensure the image's integrity.

Should I use a public or a private registry to store my images?: Public registries, such as Docker Hub, can be used to get started with Docker images and Kubernetes to create your first containerized app in a cluster. But when it comes to enterprise applications, avoid registries that you don't know or don't trust to protect your cluster from malicious images. Keep your images in a private registry, like the one provided in IBM Cloud Container Registry and make sure to control access to the registry and the image content that can be pushed.
Why is it important to check images against vulnerabilities?: Research shows that most malicious attacks leverage known software vulnerabilities and weak system configurations. When you deploy a container from an image, the container spins up with the OS and extra binaries that you described in the image. Just like you protect your virtual or physical machine, you must eliminate known vulnerabilities in the OS and binaries that you use inside the container to protect your app from being accessed by unauthorized users.

To protect your apps, consider to address the following areas:

Automate the build process and limit permissions: Automate the process to build your container image from your source code to eliminate source code variations and defects. By integrating the build process into your CI/CD pipeline, you can ensure that your image is scanned and built only if the image passes the security checks that you specified. To avoid that developers apply hot fixes to sensitive images, limit the number of people in your organization who have access to the build process.
Scan images before they deploy into production: Make sure to scan every image before you deploy a container from it. For example, if you use IBM Cloud Container Registry, all images are automatically scanned for vulnerabilities when you push the image to your namespace. If vulnerabilities are found, consider eliminating the vulnerabilities or block deployment for those images. Find a person or team in your organization who is responsible for monitoring and removing vulnerabilities. Depending on your organizational structure, this person might be part of a security, operations, or deployment team. Enable content trust so that images must be approved by a trusted signer before they can be pushed to the container registry. Then, install the open source Portieris project admission controller to block container deployments from unsigned images.
Regularly scan running containers:. Even if you deployed a container from an image that passes the vulnerability check, the operating system or binaries that run in the container might get vulnerable over time. To protect your app, you must ensure that running containers are regularly scanned so that you can detect and remediate vulnerabilities. Depending on the app, to add extra security, you can establish a job that takes down vulnerable containers after they are detected.

Security for images and deployments
Security feature	Description
Secured Docker private image repository in IBM Cloud Container Registry	Set up your own Docker image repository in a multi-tenant, highly available, and scalable private image registry that is hosted and managed by IBM. By using the registry, you can build, securely store, and share Docker images across cluster users. /n Learn more about securing your personal information when you work with container images.
Push images with trusted content only	Ensure the integrity of your images by enabling content trust in your image repository. With trusted content, you can control who can sign images as trusted and push images to a specific registry namespace. After trusted signers push an image to a registry namespace, users can pull the signed content so that they can verify the publisher and the integrity of the image.
Automatic vulnerability scans	When you use IBM Cloud Container Registry, you can leverage the built-in security scanning that is provided by Vulnerability Advisor. Every image that is pushed to your registry namespace is automatically scanned for vulnerabilities against a database of known CentOS, Debian, Red Hat, and Ubuntu issues. If vulnerabilities are found, Vulnerability Advisor provides instructions for how to resolve them to ensure image integrity and security.
Block deployments from vulnerable images or untrusted users	Create an admission controller with custom policies so that you can verify container images before you deploy them. With the open source Portieris project, you control where the images are deployed from and ensure that they meet content trust requirements. If a deployment does not meet the policies that you set, the admission controller blocks the deployment in your cluster.

What options do I have to scan running containers for vulnerabilities?: You can install third-party solutions in your cluster, such as Twistlock or StackRox to scan running containers and block malicious activities when they are detected.

Container isolation and security

When you run multiple apps in your cluster, you want to make sure that your workloads run isolated from each other and that you restrict the permissions of your pods within the cluster to avoid noisy neighbors or denial-of-service attacks.

What is a Kubernetes namespace and why should I use it?: Kubernetes namespaces are a way to virtually partition a cluster and provide isolation for your deployments and the groups of users that want to move their workload onto the cluster. With namespaces, you can organize resources across worker nodes and also across zones in multizone clusters.; Every cluster is set up with a set of default Kubernetes namespaces that include the deployments and services that are required for IBM Cloud Kubernetes Service to run properly and manage the cluster. For more information, see the service architecture.; Cluster administrators automatically have access to these namespaces and can set up additional namespaces in the cluster.

For every namespace that you have in the cluster, make sure to set up proper RBAC policies to limit access to this namespace, control what gets deployed, and to set proper resource quotas and limit ranges.

Should I set up a single-tenant or a multi-tenant cluster?

In a single-tenant cluster, you create one cluster for every group of people that must run workloads in a cluster. Usually, this team is responsible to manage the cluster and to properly configure and secure it. Multi-tenant clusters use multiple namespaces to isolate tenants and their workloads.

Deciding between a single tenant or a multi-tenant cluster. — Single tenant versus multi-tenant cluster

Deciding between single-tenant and multi-tenant clusters depends on the number of teams that must run workloads in a cluster, their service requirements, the size of the service, and the level of isolation that you want to achieve for your workloads.

A single-tenant cluster might be your option if you have many teams with complex services that each must have control over the lifecycle of the cluster. This includes having the freedom to decide when a cluster is updated or what resources can be deployed to the cluster. You can also configure a single-tenant cluster to allow privileged pods without putting other tenants at risk of being compromised. Keep in mind that managing a cluster requires in-depth Kubernetes and infrastructure knowledge to ensure cluster capacity and security for your deployments.

Multi-tenant clusters use Kubernetes namespaces to isolate tenants and are usually managed by a separate team that does not belong to one of the tenants. A multi-tenant cluster might be your option if you have multiple teams that must run small workloads in a cluster, and where creating a single-tenant cluster that is highly available across multiple zones does not bring the cost benefits that you want. While multi-tenant clusters usually require fewer people to manage and administer the cluster, they might not provide the level of isolation that you need and add more complexity in the following areas:

Access: When you set up multiple namespaces, you must configure proper RBAC policies for each namespace to ensure resource isolation. RBAC policies are complex and require in-depth Kubernetes knowledge.
Privileged pods: If one tenant in a multi-tenant cluster requires to run privileged pods, this pod can access other namespaces in the cluster or damage the shared compute host. Controlling privileged pods is a complex task that requires effort and deep technical expertise. Use pod security policies (PSPs) to control what resources your tenants can deploy in the cluster.
Network policies: Because your worker nodes are connected to the same private network, you must make sure that you have strict network policies in place to prevent pods from accessing pods in other namespaces.
Compute resource limitation: To ensure that every team has the necessary resources to deploy services and run apps in the cluster, you must set up resource quotas for every namespace. Resource quotas determine the deployment constraints, such as the number of Kubernetes resources that you can deploy, and the amount of CPU and memory that can be consumed by those resources. After you set a quota, users must include resource requests and limits in their deployments.
Shared cluster resources: If you run multiple tenants in one cluster, some cluster resources, such as the Ingress application load balancer (ALB) or available portable IP addresses are shared across tenants. Smaller services might have a hard time using shared resources if they must compete against large services in the cluster.
Updates: You can run one Kubernetes API version at a time only. All apps that run in a cluster must comply with the current Kubernetes API version independent of the team that owns the app. When you want to update a cluster, you must ensure that all teams are ready to switch to a new Kubernetes API version and that apps are updated accordingly. This also means that individual teams have less control over the Kubernetes API version they want to run.
Changes in cluster setup: If you want to change the cluster setup or reschedule workloads onto new worker nodes, you must roll out this change across tenants. This roll out requires more reconciliation and testing than in a single-tenant cluster.
Communication process: When you manage multiple tenants, consider setting up a communication process so that tenants know where to go when an issue with the cluster exists, or when they need more resources for their services. This communication process also includes informing your tenants about all changes in the cluster setup or planned updates.

Although single-tenant and multi-tenant clusters come with roughly the same costs, single-tenant clusters provide a higher level of isolation than the namespaces in a multi-tenant cluster. For better workload isolation, use single-tenant clusters.

Kubernetes network policies protect pods from internal network traffic. For example, if most or all pods don't require access to specific pods or services, and you want to ensure that pods by default can't access those pods or services, you can create a Kubernetes network policy to block ingress traffic to those pods or services. Kubernetes network policies can also help you enforce workload isolation between namespaces by controlling how pods and services in different namespaces can communicate.

How can I control pod permissions?: By default, every cluster enables the Kubernetes pod security policy admission controller that you can use to define what requirements a pod must meet to get deployed in a namespace. With pod security policies, you can control the usage of privileged containers, root namespaces, host networking and ports, volume types, host file systems, and Linux permissions such as read-only or group IDs. For more information, see Configuring pod security policies.
What else can I do to protect my container?: Limit the number of privileged containers. Containers run as a separate Linux process on the compute host that is isolated from other processes. Although users have root access inside the container, the permissions of this user are limited outside the container to protect other Linux processes, the host file system, and host devices. Some apps require access to the host file system or advanced permissions to run properly. You can run containers in privileged mode to allow the container the same access as the processes running on the compute host.; Keep in mind that privileged containers can cause huge damage to the cluster and the underlying compute host if they become compromised. Try to limit the number of containers that run in privileged mode and consider changing the configuration for your app so that the app can run without advanced permissions.

If you want to block privileged containers from running in your cluster, consider setting up custom pod security policies.

Apply OS security settings to pods: You can add the securityContext section to your pods to control the user ID and group ID that can run inside the container, or the user ID and group ID that owns the volume mount path. Setting a specific user ID helps facilitate a least privilege model. If the security context does not specify a user, Kubernetes automatically uses the user that is specified in the container image.; If you want to use securityContext to set the runAsUser user ID or fsGroup group ID, consider using block storage when you create persistent storage. NFS storage does not support fsGroup, and runAsUsermust be set at the container level, not the pod level.
Set CPU and memory limits for containers: Every container requires a specific amount of CPU and memory to properly start and to continue to run. You can define Kubernetes resource requests and resource limits for your containers or pods to limit the amount of CPU and memory that they can consume. If no limits for CPU and memory are set, and the container is busy, the container uses all the resources that are available. This high consumption of resources might affect other containers on the worker node that don't have enough resources to properly start or run, and puts your worker node at risk for denial-of-service attacks.
Enforce policy-driven authentication: You can add an Ingress annotation to your deployments to control access to your services and APIs. By using App ID and declarative security, you can ensure user authentication and token validation.

Storing personal information

You are responsible for ensuring the security of your personal information in Kubernetes resources and container images. Personal information includes your name, address, phone number, email address, or other information that might identify, contact, or locate you, your customers, or anyone else.

Use a Kubernetes secret to store personal information: Store personal information only in Kubernetes resources that are designed to hold personal information. For example, don't use your name in the name of a Kubernetes namespace, deployment, service, or config map. For proper protection and encryption, store personal information in secrets instead.; For centralized management of all your secrets across clusters and injection at application runtime, try IBM Cloud Secrets Manager.
Use a Kubernetes imagePullSecret to store image registry credentials: Do not store personal information in container images or registry namespaces. For proper protection and encryption, store registry credentials in Kubernetes imagePullSecrets and other personal information in secrets instead. Remember that if personal information is stored in a previous layer of an image, deleting an image might not be sufficient to delete this personal information.

To set up encryption for your secrets, see Encrypting Kubernetes secrets by using a key management service (KMS) provider.

Kubernetes security bulletins

If vulnerabilities are found in Kubernetes, Kubernetes releases CVEs in security bulletins to inform users and to describe the actions that users must take to remediate the vulnerability. Kubernetes security bulletins that affect IBM Cloud Kubernetes Service users or the IBM Cloud platform are published in the IBM Cloud security bulletin.

Some CVEs require the latest patch update for a Kubernetes version that you can install as part of the regular cluster update process in IBM Cloud Kubernetes Service. Make sure to apply security patches in time to protect your cluster from malicious attacks. For more information about what is included in a security patch, refer to the Kubernetes version information.