Setting up Tekton continuous deployment pipelines with DevSecOps

With this tutorial, you can set up a continuous deployment (CD) pipeline and toolchain that adhere to compliance best practices.

Before you begin

Create a Kubernetes cluster on IBM Cloud® Kubernetes Service to deploy your application.
Create toolchain secrets to access different integrations and secure them.
Compliance continuous integration (CI) toolchain configured with IBM Cloud® DevOps Insights integrations.
Optional. Configure IBM Cloud® Object Storage as the compliance evidence locker to store pipeline run evidence.
Validate your IBM Cloud® Identity and Access Management (IAM) permissions that are assigned to the corresponding integrations.
View the following video to get an overview of the CD toolchain setup process.

Start the CD toolchain setup

The Continuous Delivery service provides templates that guide you through the toolchain setup and create processes in a logical order. A progress indicator shows the steps to complete the configuration. Follow the steps to access the template for the CI toolchain.

In the IBM Cloud console, click the Menu icon > Platform Automation > Toolchains.
On the Toolchains page, click Create toolchain.
Click CD-Develop with DevSecOps practices tile.

Set up the CI toolchain settings

The Welcome page summarizes the purpose of the toolchain along with pointers to the documentation and related materials.

Click Start.
Enter a Toolchain name within your toolchain for the same region and resource group in the IBM Cloud. Make sure that the toolchain's name is unique within your toolchains for the same region and resource group in IBM Cloud.
Select a region from the dropdown list.

The toolchain region can differ from the cluster and registry region.
Select a resource group from the dropdown list.
Click Continue.

You can advance to the next step only when the configuration for the current step is complete and valid. You can always click Back to view previous steps in the guided installer. The toolchain installer retains all the configuration settings from the successive steps.

Some steps include a Switch to advanced configuration toggle button. These steps by default present you with the minimum configuration. However, advanced users that need finer grained control can click the Switch to advanced configuration toggle to reveal the options for the underlying integration.

Set up the CD tool integrations

The CD tool integration covers the setup of the various tool and service integrations that the CD toolchain uses. If you use CI pipeline toolchain template to set up your CI process, refer to your CI toolchain and copy the names of the repositories that are used in that toolchain.

Application related repositories

If you want to set up your CI toolchain from scratch, configure these repositories during CI toolchain creation.

Inventory: Deployment information is tracked using this repository. For each successful CI pipeline run, the CD pipeline creates a new branch that is named as the created CR number, and merges to the master after deployment is concluded. For example, https://<region>.git.cloud.ibm.com/myorg/my-compliance-ci-inventory.
Issues: Stores the issues about incidents that happen during the build and deployment process. For example, https://<region>.git.cloud.ibm.com/myorg/my-compliance-ci-issues.
Evidence: Collects all the raw compliance evidence that belongs to the application. For example, https://<region>.git.cloud.ibm.com/myorg/my-compliance-ci-evidence.
Toolchain: The tekton pipeline definitions, such as, pipelines, triggers, and listeners are stored in this repository. For example, https://<region>.git.cloud.ibm.com/myorg/my-compliance-ci-toolchain.

After you capture the names of repositories from CI toolchain, proceed with the Guided Setup to begin CD toolchain creation. During the setup process, for each repository you can provide the URL to an existing IBM hosted Git Repos and Issue Tracking repository. Currently, the toolchain supports creating only Git Repos and Issue Tracking repositories. Future releases provide support to create repositories with GitHub, GitHub Enterprise (GHE), and other SCM providers.

Inventory

The inventory repository records details of artifacts that are built by the CI toolchains. The default behavior of the toolchain is to Use existing inventory to link an existing inventory repository for the toolchain. The toolchain currently supports linking only to existing Git Repos and Issue Tracking repositories. For more information, see Application related repositories.

You can accept the default configuration that is provided in the template.
Click Continue.

Issues

The issues repository records issues that are found while the CI pipeline is running.

You can accept the default configuration that is provided in the template.
Click Continue

Pipeline configuration

The pipeline configuration repository contains YAML files and scripts that are needed for deployment, testing, and other custom tasks. For more information about Git repos, see Configuring your Git Repos and Issue Tracking. For more information about customizable scripts, see Custom scripts.

Accept the default settings for Source Provider and Create a new deployment configuration repository.

The toolchain clones the sample hello-compliance-deployment in your Git organization. For more information about Git repos, see Configuring your Git Repos and Issue Tracking. For more information about customizable scripts, see Custom scripts.
Enter a New repository name.
Click Continue.

Secrets

Several tools in this toolchain, and possibly in your customizable scripts, require secrets to access privileged resources. An IBM Cloud API key is an example of such a secret. Store these secrets securely in a secrets management tool, such as IBM Key Protect for IBM Cloud, IBM Cloud® Secrets Manager, or HashiCorp Vault. The secrets management tool can be integrated into the toolchain so that you can easily reference the secrets in your Tekton pipeline.

You can use the AppRole authentication method to read secret values.

This tutorial uses IBM Cloud® Secrets Manager as the vault for secrets. The Region, Resource group, and Service name fields are automatically populated based on available choices. Click the drop-down indicators to see the other choices.
Type your Secrets Manager instance name.
Select the Authorization type from the dropdown list.
Click Continue.

Evidence Storage

The evidence repository stores all the evidence and artifacts that are generated by the DevSecOps CI pipeline.

Select a Use existing evidence locker repository.
Select the Repository URL that was created when you configured the continuous integration (CI) toolchain. For more information about evidence storage, see Evidence.
Toggle the IBM Cloud Object Storage bucket slider to store all the evidence in the IBM Cloud Object Storage bucket.
Click Continue.

Cloud Object Storage Bucket

You must have the IBM Cloud Object Storage instance and a bucket to act as a compliance evidence locker.

The Cloud Object Storage instance, Bucket name, and Cloud Object Storage endpoint fields are automatically populated.
Enter your Service ID API key.
- Preferred: An existing key can be imported from a secrets vault by clicking the key icon.
- An existing key can be copied and pasted.
Click Continue.

The endpoint field is optional. It is recommended to select or provide the endpoint during the setup of the toolchain or during the pipeline run.

Deploy

Configure the inventory target and Kubernetes cluster where the application is deployed:

The default App name is hello-compliance-app.
Type your IBM Cloud API Key. The API key is used to interact with the IBM Cloud CLI tool in several tasks.
- Preferred: An existing key can be imported from a secrets vault by clicking the key icon.
- An existing key can be copied and pasted.
- A new key can be created from here by clicking the New +.
The newly generated API key can be immediately saved to a secrets vault.
If the API key is valid and has sufficient access, the Container Registry, Container Registry namespace, Dev cluster region, Resource group, Cluster name, Cluster namespace are automatically populated. You can change any of these fields to match your configuration.
Click Continue.

IBM Cloud API Key

The API key is used to interact with the ibmcloud CLI tool in several tasks.

An existing API key can be imported from an existing secret provider intance created as prerequisites (Recommended)
An existing key can be copy and pasted (Not Recommended)
A new key can be created by clicking New +. Generate a api-key if you don’t have one. The newly generated API key can be immediately saved to an existing Key Protect instance

When the API Key field is filled, the registry and cluster related fields are filled automatically.

Inventory target and source branches

Accept the Single cluster (push based deployment) to deploy your application to targets such as Virtual Server Instance or customize the deployment process, use the Custom option.
Click Continue to view the cluster page.
Enter the IBM Cloud API Key field to interact with the CLI tool in several tasks. Note an existing key can be imported from a secrets vault by clicking the key icon.
Verify your Cluster region, Resource group, Cluster name, and Cluster namespace fields where your target cluster is created.
Click Continue.

Change Request Management

Select IBM Cloud hosted Git Repos and Issue Tracking repository to manage the change requests. For more information, see Automating change management.
Click Continue.

DevOps Insights toolchain

Link an existing DevOps Insights instance from another toolchain to this pipeline, so that all the build, deploy, and test records from the CI and CD toolchain pipelines can be collected in the same place.

The CD toolchain can publish the deployment records to an existing DevOps Insights instance. To enable this feature, provide the ID of the toolchain that contains the existing DevOps Insights instance by selecting in the DevOps Insights toolchain ID list.

Provide your DevOps Insights IBM Cloud API Key.
Accept the default configuration.
Verify your Region, Resource group, DevOps Insights toolchain ID, and DevOps Insights target environment fields where your target environment is created.
Click Continue.

Link to existing DevOps Insights toolchain

DevOps Insights can optionally be included in the created toolchain and after each compliance check evidence is published. The toolchain can use an existing DevOps Insights instance to publish the deployment records to insights. You can link DevOps Insights integration from another toolchain by providing the integration ID.

Accept the Current Toolchain.
Click Continue.

You can copy the toolchain ID from the URL of your toolchain. A toolchain's URL follows this pattern: https://cloud.ibm.com/devops/toolchains/<toolchain-ID-comes-here>?env_id=ibm:yp:us-south. If the URL is: https://cloud.ibm.com/devops/toolchains/aaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee?env_id=ibm:yp:us-south then the toolchain's ID is: aaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee. Include the ID only, not the full URL.

You can also set a target environment for the DevOps Insights interactions. This parameter is optional, which is used instead of the target environment from the inventory.

Optional tools

Slack

Configure the Slack to receive notifications about your pull requests, or CI pipeline events. You can also add the Slack tool after the toolchain creation.

Enter your Slack webhook. For more information, see Slack webhook.
Enter your Slack channel to post message.
Enter the Slack team name. For example, if your team URL https://team.slack.com, the team name is team.
Choosing the events for which you want to receive notifications for Automated Slack Notifications.
Click Continue.

Security and Compliance

To integrate the toolchain with the Security and Compliance Center service, you need to provide a name and the evidence locker repository name for the Security and Compliance Center data collector. For more information, see the Security and Compliance Center and the tool integration configuration process.

Enter a Name to verify the Security and Compliance Center posture of the toolchain.
Enter an Evidence repository name or URL.
Accept Evidence context as Continuous Deployment.
Accept Use profile with attachment as Disabled. For more information, see Security and Compliance Center and the tool integration configuration process.
Click Continue to view the Summary page.

Private worker

The delivery pipeline private worker tool integration connects with one or more private workers that can run delivery pipeline workloads in isolation. For more information, see Working with private workers.

Enter a Name for the delivery pipeline private worker.
Enter your Service ID API key.
- Preferred: An existing key can be imported from a secrets vault by clicking the key icon.
- An existing key can be copied and pasted.
Click Continue.

Create the CD toolchain

On the Summary page, click Create.
Wait for the toolchain creation to view the summary page as shown in the screen capture. This can take a few minutes.

You can configure the individual toolchain integrations after the pipeline is created.

Explore the CD toolchain

Run the promotion pipeline

Make sure that the CI pipeline ran successfully before you run the promotion pipeline.

Click cd-pipeline.
Click Run for Manual Promotion Trigger pipeline.
Click Run to trigger the pipeline.
Click Manual Promotion Trigger > #1promotion-pipelinerun pipeline. Wait for the promotion pipeline run to complete and check the execution log.

The promotion pipeline creates a pull request with the content of the inventory on the inventory source environment such as master branch targeting the inventory target environment branch such as staging or production.
After the promotion pipeline finishes successfully, the promote task log provides a link to the pull request in the inventory repository. The pull request name is of the format promote <inventory source environment> to <inventory target environment>, for example, promote master to prod
Open the pull request in your browser with the link provided in the log. Complete the details in the following sections:
- Priority: (mandatory) Set as Critical, High, Moderate, Low, or Planning.
- Change Request assignee: (mandatory) Email-ID of the assignee.
- Additional Description: Description about the changes in the application.
- Purpose/Goal: Purpose of the changes that are made to the application.
- Explanation of Impact: Impact of the change to the application behavior or environment.
- Backout Plan: Steps to back out if there is a deployment failure.
Complete the fields in the Pull Request and click save.
Add the EMERGENCY label to your pull request if any compliance checks in CI failed and you want to continue with deployment
Merge the pull request from Git Repos and Issue Tracking.

The details of the pull request are used during the CD pipeline that is run to create and update in the change request repository.

Run the CD pipeline

Run promotion pipeline

Make sure that CI pipeline ran successfully before you run the Promotion Pipeline.
The promotion pipeline creates a Pull Request with the content of the inventory on the inventory source environment. For example, master branch targeting the inventory target environment branch, for example, staging or prod. An intermediary branch for the PR is created which can be discarded after the PR is merged.

Running the promotion pipeline
After the Promotion Pipeline finishes successfully, the promote task provides you with a link to the previously mentioned Pull Request in the inventory repository. The pull request name is in the following format:
```
promote <Inventory Source Environment> to <Inventory Target Environment>
```
Open the Pull Request in your browser with the link provided in the log. Complete the sections as follows:
- Priority: (mandatory) One of Critical, High, Moderate, Low, Planning
- Change Request assignee: (mandatory) Email-id of the assignee
- Additional Description: Description about the changes in the application
- Purpose: Purpose of the changes that are made to the application
- Explanation of Impact: Impact of the change to the application behavior or environment
- Backout Plan: Steps to backout if the deployment fails
Complete the fields in the Pull Request and save.
Merge the Pull Request from the Git Repos and Issue Tracking.

The details of the Pull Request are used by the CD pipeline to create a change request in the change request management repository during the CD pipeline run.

Run the CD pipeline

You can trigger a CD pipeline in manual or automatic way. You can trigger the CD pipeline manually anytime, but if no changes since the last successful deployment, the CD pipeline stops early as nothing to deploy. A Git Repos and Issue Tracking trigger is set up to trigger automatic CD pipeline, but is disabled by default and can be enabled after the first promotion.You can add and use force-redeploy variable to rerun the CD with no code changes. View the successful screen capture of the DevSecOps CD pipeline.

DevSecOps CD pipeline automatic promotion

You can also trigger the CD Pipeline manually anytime, but if there are no changes since the last successful deployment, the CD Pipeline doesn't deploy anything new.

A successful CD pipeline run looks like this:

Flow of the tasks in the CD pipeline:

After a successful CD pipeline run, you can find the sample app that is running on the prod namespace. The app URL can be found under run stage substep of prod deployment step of CD pipeline run. Use that URL to check that the app is running.