Working with Delivery Pipeline Private Workers

The Delivery Pipeline uses public and private workers to run pipeline jobs. By default, pipeline jobs are run by using public workers on IBM-managed public shared infrastructure.

In certain scenarios, your Delivery Pipeline might require access to internal or on-premises resources. In these situations, you can connect to and integrate a Delivery Pipeline Private Worker to run on your own Kubernetes infrastructure.

Prerequisites

Before you set up a private worker, make sure that you have the following resources in place:

A Kubernetes cluster. You must have a cluster to install a private worker. You can provide your own cluster or you can set up a cluster from the IBM Cloud Kubernetes Service.
Optional. A toolchain with a pipeline that contains at least one stage. You can create a toolchain by using the Delivery Pipeline Private Worker tool integration. You can also create a toolchain by using the Develop a Kubernetes app toolchain template:
1. Log in to IBM Cloud.
2. Go to https://cloud.ibm.com/devops/create and select the Develop a Kubernetes app toolchain template.
3. Complete the fields for the Delivery Pipeline.
4. Click Create to create your Kubernetes app toolchain.

Setting up a Delivery Pipeline Private Worker

Toolchains provide an integrated set of tools to build, deploy, and manage your apps. Tool integrations are the building blocks of a toolchain. A pipeline that is contained within a toolchain that has one or more private worker integrations has extra options available. For Classic pipelines, these options are available on the Workers tab of the Stage Configuration page. For Tekton pipelines, these options are available within the settings that are specified on the Worker page.

For Tekton pipelines, you can also specify one worker per trigger, which overrides the worker that is configured at the pipeline level on the Worker page.

Complete the following steps to set up a private worker:

Configure the Delivery Pipeline Private Worker tool integration for your toolchain.
Configure your Kubernetes cluster with a private worker.
Use the private worker in your pipeline.

Configuring the Delivery Pipeline Private Worker tool integration by using the console

Complete the following steps to configure the Delivery Pipeline Private Worker tool integration for your toolchain:

From the IBM Cloud console, click the Menu icon > Platform Automation > Toolchains. On the Toolchains page, click a toolchain to open its Overview page. Alternatively, on your app's Overview page, on the Continuous delivery card, click View toolchain. Then, click Overview.

a. Click Add tool.

b. In the Tool Integrations section, click Delivery Pipeline Private Worker.
Type a name for the tool integration. This name identifies a pool of private workers in the Workers tab of the pipeline stage.
Type your service ID API key to authenticate access to the work queue where one or more private workers can look for work. If you don't have a service ID API key, click Create to generate one for this private worker.
Click Create Integration.
From your toolchain's Overview page, on the Delivery pipelines card, click Delivery Pipeline Private Worker. A list of all of the workers that are registered by using an API key that is associated with this service ID is displayed.

The list of workers is empty until you use Delivery Pipeline private workers for the first time.

For more information about the Delivery Pipeline Private Worker tool integration, see Configuring Delivery Pipeline Private Worker.

Configuring the Delivery Pipeline Private Worker tool integration with the API

Obtain an IAM bearer token.
Determine the region and ID of the toolchain to which you want to add the Delivery Pipeline tool integration.

Add the Delivery Pipeline Private Worker tool integration to the toolchain.

curl -X POST \
  https://api.{region}.devops.cloud.ibm.com/toolchain/v2/toolchains/{toolchain_id}/tools \
  -H 'Authorization: Bearer {token}' \
  -H 'Accept: application/json` \
  -H 'Content-Type: application/json' \
    -d '{
    "tool_type_id": "private_worker",
    "parameters": {
      "name":"{tool_integration_name}", "workerQueueCredentials":"{sid_apikey}"
      }
  }'

The following table lists and describes each of the variables that are used in the previous step.

Variables for adding the Delivery Pipeline Private Worker tool integration with the API
Variable	Description
`{region}`	The region in which the toolchain resides, for example, `us-south`.
`{tool_integration_name}`	A name for your tool integration, for example, `dev-worker`.
`{toolchain_id}`	The ID of the toolchain to which to add the tool integration.
`{token}`	A valid IAM bearer token.
`{sid_apikey}`	A service ID API key that is used by the private worker to authenticate with the work queue.

Configuring the Delivery Pipeline Private Worker tool integration with Terraform

To install the Terraform CLI and configure the IBM Cloud provider plug-in for Terraform, follow the tutorial for Getting started with Terraform on IBM Cloud®.
Create a Terraform configuration file that is named main.tf. In this file, add the configuration to create a pipeline by using the HashiCorp Configuration Language. For more information about using this configuration language, see the Terraform documentation.

A private worker must belong to a toolchain. You can also create toolchains by using Terraform.

The following example creates a toolchain and a private worker by using the specified Terraform resources.
```
data "ibm_resource_group" "group" {
  name = "default"
}

resource "ibm_cd_toolchain" "my_toolchain" {
  name              = "terraform_toolchain"
  resource_group_id = data.ibm_resource_group.group.id
}

resource "ibm_cd_toolchain_tool_privateworker" "my_private_worker" {
  parameters {
     name = "terraform-private-worker"
     worker_queue_credentials = "{my_sid_apikey}"
  }
  toolchain_id = ibm_cd_toolchain.my_toolchain.id
}
```
For more information about ibm_cd_toolchain_tool_privateworker, see the argument reference details in the Terraform Registry Documentation.
Initialize the Terraform CLI, if required.
```
terraform init
```
Create a Terraform execution plan. This plan summarizes all of the actions that must run to create a toolchain with a private worker.
```
terraform plan
```
Apply the Terraform execution plan. Terraform takes all of the required actions to create the toolchain and the private worker.
```
terraform apply
```

Configuring your Kubernetes cluster

Configure your Kubernetes cluster with a private worker:

From the IBM Cloud console, click the menu icon and select DevOps. On the Toolchains page, click a toolchain to open its Overview page. Alternatively, on your app's Overview page, on the Continuous delivery card, click View toolchain.
On the Delivery pipelines card, click the Delivery Pipeline Private Worker tool integration that you want to configure.
Click Getting Started and then follow the steps to install and register a private worker on your Kubernetes cluster.

Using the Delivery Pipeline Private Worker in your pipeline

Complete the following steps to use the private worker in your pipeline:

From the IBM Cloud console, click the Menu icon > Platform Automation > Toolchains. On the Toolchains page, click a toolchain to open its Overview page. Alternatively, on your app's Overview page, on the Continuous delivery card, click View toolchain.
On the Delivery pipelines card, click the pipeline that you want to use the private worker with.
If you are using a Classic pipeline, on the Pipeline page, on the stage, click the Stage Configuration icon. Then, click Configure Stage and click the Workers tab.
If you are using a Tekton pipeline, from the trigger for which you want to specify a worker, select the Workers page.
Select the private worker that you want to use in your pipeline.

By default, pipeline jobs and pipeline runs are run by using a pool of IBM-managed shared workers in the region where the pipeline is defined.
Click Save.
You can manually run your Classic pipeline stage or Tekton pipeline, or you can wait for a trigger to start the pipeline run. The pipeline run is completed by using the specified private worker on the associated Kubernetes cluster. You can view the log file output for the jobs or the Run Details page in Tekton pipelines to determine which worker was used.

Verifying the Delivery Pipeline Private Worker output

If you are assigned the Toolchain Editor role, you can view the pipeline execution logs in the Stage History page of the pipeline.

If you are assigned the Toolchain Administrator role, or you have worker cluster access, you can also verify the Kubernetes resources in the installation, by way of the kubectl CLI.

Private Worker output
Action	Command
View the list of namespaces in the cluster	`$ kubectl get ns` (while the pipeline is running). At least one namespace, prefixed with `pw-` is returned during the pipeline job execution. For example, `pw-f0fb3cdb-5173-4785-ae92-bb05268e041e`.
View the pod that is running the pipeline job	`$ kubectl get pods -n pw-f0fb3cdb-5173-4785-ae92-bb05268e041e`
Start a bash session within a pod that is named `pw-f0fb3cdb-5173-4785-ae92-bb05268e041e`	`kubectl -n pw-f0fb3cdb-5173-4785-ae92-bb05268e041e exec -it job-pod-e56e78 bash`

Modifying the Delivery Pipeline Private Worker tool integration credentials

After you set up your private worker, you can update the credentials that are used for the tool integration. You might want to change these credentials if they are deleted, expired, or compromised. You can update the credentials by creating a new service ID or by updating the API key.

Creating a service ID

A service ID identifies a service or an application in the same way that a user ID identifies a user. You can use a service ID to enable an application outside of IBM Cloud® to access your IBM Cloud® service. For more information about service IDs, see Creating and working with service IDs.

Complete the following steps to create a service ID:

From the IBM Cloud console, click the Menu icon > Platform Automation > Toolchains. On the Toolchains page, click a toolchain to open its Overview page. Alternatively, on your app's Overview page, on the Continuous delivery card, click View toolchain.
On the Delivery pipelines card, click the private workers tool integration that you want to modify, click the menu, and then click Configure to access the configuration options.
Click Create.
Enter a name and description for the service ID.
Click Save Integration.
On the DevOps dashboard, on the Toolchains page, click a toolchain to open its Overview page. Alternatively, on the app's Overview page, on the Continuous delivery card, click View toolchain.
On the Delivery pipelines card, click the Delivery Pipeline Private Worker tool integration that you want to specify new user credentials for.
Click Getting Started, and complete steps 2 and 3 to register and verify the private worker on your cluster.

Updating the API key

An API key is a unique code that is passed to an API to identify the application or user that is calling it. To prevent malicious use of an API, you can use API keys to track and control how that API is used. For more information about API keys, see Understanding API keys.

Complete the following steps to update the API key for use with the Delivery Pipeline Private Worker tool integration:

From the IBM Cloud console, click the Menu icon > Platform Automation > Toolchains. On the Toolchains page, click a toolchain to open its Overview page. Alternatively, on your app's Overview page, on the Continuous delivery card, click View toolchain.
On the Delivery pipelines card, go to the Delivery Pipeline Private Worker that you want to modify, click the menu, and then click Configure to access the configuration options.
Specify your new API key. It is recommended that you do not use the API key directly. Instead, use a Secrets Manager to store the API key and add a reference to that stored key in your private worker configuration.
Click Save Integration.

You can revoke access to a private worker by deleting the corresponding API key. Due to caching policies and invalidation, it might take up to 60 minutes for a deleted API key to prevent jobs from running. For more information about deleting an API key, see Deleting an API key for a service ID.

Deleting a Delivery Pipeline Private Worker

Complete the following steps to delete a private worker:

Delete the private worker from the pool of workers.
Delete the private worker from your Kubernetes cluster.

Deleting a Delivery Pipeline Private Worker from the pool of workers

Complete the following steps to delete the private worker from the pool of workers:

From the IBM Cloud console, click the Menu icon > Platform Automation > Toolchains. On the Toolchains page, click a toolchain to open its Overview page. Alternatively, on your app's Overview page, on the Continuous delivery card, click View toolchain.
On the Delivery pipelines card, click the Delivery Pipeline Private Worker tool integration that you want to configure.
Click Overview.
Click the menu for the private worker that you want to delete to access the configuration options.
Click Remove and then click Confirm.

If you delete all of the private workers from a pool of workers, subsequent Classic pipelines stage runs fail and the No workers currently registered message is displayed in the Stage History. For Tekton pipelines, subsequent stage run requests are placed in a queue and do not start to run until a worker is available in the selected worker pool.

Although deleting a private worker tool integration from a toolchain prevents that toolchain from using the pool of workers, other toolchains might still use the worker pool. You can delete a pool of workers from all toolchains by deleting the service ID that is associated with the Delivery Pipeline Private Worker tool integration.

Deleting a Delivery Pipeline Private Worker from your Kubernetes cluster

Complete the following steps in the console to delete the private worker from your cluster:

Click the Delivery Pipeline Private Worker tool integration that you want to configure.
Click Getting Started and follow the steps to delete the private worker from your cluster.

Run the following command to delete the private worker from your cluster by using the CLI:

kubectl delete --filename "https://private-worker-service.{REGION}.devops.cloud.ibm.com/install/worker?serviceId={SERVICE_ID}&apikey={APIKEY}&name={WORKER_NAME}"

Deleting the Delivery Pipeline Private Worker tool integration by using the console

If you delete the Delivery Pipeline Private Worker tool integration from your toolchain, the deletion cannot be undone.

Complete the following steps to delete a Delivery Pipeline Private Worker tool integration:

From the IBM Cloud console, click the Menu icon > Platform Automation > Toolchains. On the Toolchains page, click a toolchain to open its Overview page. Alternatively, on your app's Overview page, on the Continuous delivery card, click View toolchain.
On the Delivery pipelines card, click the Delivery Pipeline Private Worker tool integration that you want to delete, and then click the menu to access the configuration options.
To delete the tool integration from your toolchain, click Delete.
Confirm by clicking Delete. The Delivery Pipeline Private Worker tool integration is removed from the toolchain and is no longer available in the Workers tab in the delivery pipeline Stage Configuration page.

If you delete the Delivery Pipeline Private Worker tool integration from a toolchain and the private worker is configured for a pipeline stage, it is still listed in the Workers tab of the Stage Configuration page. However, the private worker is disabled and labeled REMOVED. You must select a different private worker (if one exists) or use a public worker instead.

Deleting the Delivery Pipeline Private Worker tool integration with the API

Obtain an IAM bearer token.
Determine the region and ID of the toolchain to which you want to add the Delivery Pipeline tool integration.

Remove the Delivery Pipeline Private Worker tool integration from the toolchain.

curl -X DELETE \
  https://api.{region}.devops.cloud.ibm.com/toolchain/v2/toolchains/{toolchain_id}/tools/{worker_id} \
  -H 'Authorization: Bearer {token}'

The following table lists and describes each of the variables that are used in the previous step.

Variables for deleting the Delivery Pipeline Private Worker tool integration with the API
Variable	Description
`{region}`	The region in which the toolchain resides, for example, `us-south`.
`{worker_id}`	The ID of the private worker tool to delete.
`{toolchain_id}`	The ID of the toolchain to which to add the tool integration.
`{token}`	A valid IAM bearer token.

Deleting the Delivery Pipeline Private Worker tool integration with Terraform

Locate the Terraform file (for example, main.tf) that contains the resource block for the existing pipeline.

The resource in the following example describes an existing pipeline.

data "ibm_resource_group" "group" {
  name = "default"
}

resource "ibm_cd_toolchain" "my_toolchain" {
  name              = "terraform_toolchain"
  resource_group_id = data.ibm_resource_group.group.id
}

resource "ibm_cd_toolchain_tool_privateworker" "my_private_worker" {
  parameters {
     name = "terraform-private-worker"
     worker_queue_credentials = "{my_sid_apikey}"
  }
  toolchain_id = ibm_cd_toolchain.my_toolchain.id
}

Remove the ibm_cd_toolchain_privateworker resource block from your terraform file.
Initialize the Terraform CLI, if required.
```
terraform init
```
Create a Terraform execution plan. This plan summarizes all of the actions that must run to create a toolchain with a private worker.
```
terraform plan
```
Apply the Terraform execution plan. Terraform takes all of the required actions to remove the private worker from your toolchain.
```
terraform apply
```

Because the private worker agent might be used in other toolchains, it remains on your clusters until you delete it.

Viewing the Delivery Pipeline Private Worker status

You can view the private worker status on a cluster to verify that the private worker is registered, and that it uses a valid API key. Private worker registration and API key assignment is set up during the initial private worker registration. These settings allow the private worker to communicate with the IBM Cloud® Continuous Delivery framework and be considered Active.

Complete the following steps to view the private worker status on the cluster where it is installed:

Log in as an authorized user to the cluster that hosts the worker.
Run the following command to list all of the agents that are registered on the cluster:
```
kubectl get workeragents
```
The following information is listed for each agent:
- Name: The name that is registered for this worker.
- ServiceID: The serviceId of this worker.
- Agent: Indicates whether the worker agent is operating correctly on the cluster by specifying OK, or displaying an error message.
- Registered: The registration state of the worker, such as Succeeded or Unregistered.
- Version: The version of this worker for the current version that is used by the framework, such as OK, Outdated, or Unsupported.
- Auth: The validity of the API key that is used by the worker, such as OK or Error.
- Constrained: Indicates whether any cluster issues exist that affect the worker by specifying OK, or displaying an error message.
- Paused: Indicates whether the agent is actively searching for new tasks by specifying true or false.

To verify the overall status of a registered private worker pool, check the Overview page of the Delivery Pipeline private worker from the toolchain where it is used.

From the IBM Cloud console, click the Menu icon > Platform Automation > Toolchains. On the Toolchains page, click a toolchain to open its Overview page. Alternatively, on your app's Overview page, on the Continuous delivery card, click View toolchain.
On the Delivery pipelines card, click the private workers tool integration for which you want to view a list of all of the registered workers. This worker pool contains one or more private workers that are installed on different clusters, but share a ServiceId. When work is assigned to a private worker pool, any of the private workers that are listed in the pool are eligible to run the workload.

The following information is provided for each worker that is registered in this pool:
- Name: The name of the registered worker.
- Status: The current status of the worker.
- Version: The current version of the worker. If the current version is out of date, a notification is displayed.
- Tekton Version: The Tekton release version.
- Cluster: The name of the cluster that the worker is registered on, if available.
- Last Active: The last time that the agent reported in.
Private workers can have one of the following statuses:
- active: The private worker is operating normally.
- inactive: The private worker is offline. Check your cluster. You might need to register the private worker again.
- unsupported: The private worker version in use is no longer supported. The private worker cannot run and you must update to the latest version.

Updating a Delivery Pipeline Private Worker

Complete the following steps to update a private worker to use the latest version:

Log in as an authorized user to the cluster that hosts the worker.
Run the following command:

kubectl apply --filename "https://private-worker-service.{REGION}.devops.cloud.ibm.com/update"

Delivery Pipeline Private Worker images

The private worker installation script pulls required images from the global IBM Cloud® Container Registry. It pulls the most recent images of pipeline private workers and respective Tekton framework images that include fixes for any vulnerability found.

The image URL for the pipeline private worker is icr.io/continuous-delivery/pipeline/pipeline-private-worker:<agent version>.

Private Worker images
Pipeline private worker image version	IBM Cloud® Container Registry version	Known vulnerabilities
0.18.0	`icr.io/continuous-delivery/pipeline/pipeline-private-worker:0.18.0`

The private worker installation also pulls the following supporting open source images to the cluster:

icr.io/continuous-delivery/pipeline/tekton/controller
icr.io/continuous-delivery/pipeline/tekton/entrypoint
icr.io/continuous-delivery/pipeline/tekton/nop
icr.io/continuous-delivery/pipeline/tekton/sidecarlogresults
icr.io/continuous-delivery/pipeline/tekton/events
icr.io/continuous-delivery/pipeline/tekton/workingdirinit
icr.io/continuous-delivery/pipeline/tekton/powershell
icr.io/continuous-delivery/pipeline/tekton/resolvers
icr.io/continuous-delivery/pipeline/tekton/webhook

The private worker agent also uses these internal images:

icr.io/continuous-delivery/pipeline/pipeline-private-worker-util
icr.io/continuous-delivery/pipeline/tekton/kubectl-jq
icr.io/continuous-delivery/pipeline/tekton/ubi
icr.io/continuous-delivery/pipeline/tekton/base

Support for previous Delivery Pipeline Private Worker images

To take advantage of the current features and fixes, and to maintain full compliance, it is recommended that you keep your private worker images up to date.

For each subsequent release of a private worker image, the version number is updated.

Agent changes result in a minor version increment, for example version 14.10 changes to version 14.11.
A new Tekton framework change results in a major version increment, for example version 14.11 changes to version 15.0.

IBM Cloud® Continuous Delivery supports the n-1 major version of the current private worker image. For example, if the current image is version 14.x, previous version 13.x images are also allowed to run. After the major version of the pipeline worker image is incremented to 15.x, version 14.x images are allowed, but any version 13.x or earlier images are considered outdated.