Learning about Code Engine architecture and workload isolation

IBM Cloud® Code Engine is the IBM Cloud platform that unifies container images, 12-factor-apps, functions, and batch jobs as a one-stop-shop. It's a multi-tenant system that consists of three major building blocks: A control plane, a (set of) shard (or shards), and a routing layer. The control plane and the shards are realized as separate multi-zone Kubernetes clusters. The following diagram gives a graphical overview of the architecture.

Figure 1. Code Engine architecture diagram

Code Engine is based on IBM Cloud Kubernetes Service clusters and depends on the components and workload isolation of the IBM Cloud Kubernetes Service. For more information, see IBM Cloud Kubernetes Service VPC cluster architecture.

All components are managed and owned by IBM and run in the IBM Cloud account. Each cluster is running in its own VPC and separated from other clusters.

The Code Engine control plane runs the components that are shared among all Code Engine users and make the Kubernetes cluster a true multi-tenant system. The control-plane consists of four microservices that are deployed on it.

Table 1. Code Engine control-plane microservices
Component	Purpose
Resource broker	Creates and deletes Code Engine project resources in the IBM Cloud resource controller and requests the placement of the project on a shard.
Project placement controller	Selects a shard and requests the creation, deletion, and isolation of the project on the shard.
API server	Provides the target information (`KUBECONFIG` file) for the selected project. It also performs IAM access policy checks and writes audit records.
`Kube API proxy`	Proxies each API request to the proper shard cluster, perform IAM policy checks, and writes audit records.

The shards are running the customer workload, such as builds, batch jobs, or apps. Therefore, the shard cluster runs the following microservices to control the customer workloads.

Table 2. Shard cluster microservices
Component	Purpose
Project isolation controller	Manages and isolates the Kubernetes namespace corresponding to the Code Engine project resource. It monitors and ensures the isolation aspects like role-based-access-control (RBAC), pod security policies, resource quota, and network policies are enforced.
Project domain and cert controller	Manages the domain and certificates for the route endpoint of the project. The endpoint consists of a DNS entry and a wildcard certificate.
Knative and Istio	Manage the lifecycle of applications. Knative is responsible for scaling the application. Istio is responsible for routing the traffic to the proper revision and container of the application.
Batch controller	Manages the lifecycle and containers for jobs and job runs.
Build controller	Manages the lifecycle and containers for builds and build runs.
Service binding and IBM Cloud operator	Manage the lifecycle of secrets that are associated to bindings of IBM Cloud services to applications and jobs.
IBM Cloud Object Storage event source controller	Manage the lifecycle of event subscriptions from the IBM Cloud Object Storage service.
Cluster node autoscaler	Scales the shard by adding and removing worker nodes based on capacity demand.
IBM Cloud Monitoring	Sends service metrics to IBM Cloud Monitoring. For more information about these metrics, see Monitoring for Code Engine.
IBM® Log Analysis	Forward platform logs and metrics to IBM Log Analysis. For more information, see Auditing events for Code Engine.

Code Engine workload isolation

Code Engine is a multi-tenant, regional service where tenants share the same network and compute infrastructure. In particular, the network and compute infrastructure are shared resources and some management components are common to all tenants. Code Engine implements the following controls to separate tenants:

Code Engine project resources are separated within a secured Kubernetes environment that is running in an IBM Cloud multi-zone region.
Code Engine projects and its containing resources, such as application, builds, and jobs that run on shared clusters that use shared management components.
To separate the access to project resources, Code Engine performs several levels of authentication and authorization checks within the apiserver and kube-api-proxy components (see previous table),
- IAM authentication and access policies checks are performed on a project level.
- To manage multi-tenant access to the underlying Kubernetes API, direct access to the API server is not allowed. Instead, use the Code Engine custom Kube-api-proxy API for access.
- Role-based access control checks are performed on a resource level to allow only authorized users to perform certain operations on project resources.
The authorization is controlled by the customer by assigning manager, reader, or writer roles to users for a Code Engine project resource within IAM.
To restrict customer workload, Code Engine enforces the following concepts,
- Container isolation through various Linux isolation techniques. These techniques ensure multiple layers of security to prevent the privilege escalation of containers and to restrict containers to use a limited set of system privileges.
- Resource quota and LimitRange to prevent excessive resource consumption.
- Network policies to control the network flows within the environment.
Shared multi-tenant components are secured, for example, by disabling reverse lookup in KubeDNS.
To limit the blast radius, each shard cluster is running in its own VPC, which is isolated from other shard VPCs.
DDoS protection is enabled automatically for all endpoints, including customer applications, through Cloud Internet Services (CIS) with no additional cost to you. DDoS protection covers System Interconnection (OSI) Layer 3 and Layer 4 (TCP/IP) protocol attacks, but not Layer 7 (HTTP) attacks. See DDoS protection.