Part 2: Set up a CI toolchain for Infrastructure as Code (Terraform)
This tutorial is part 2 of a 3-part tutorial series where you learn IBM Cloud® DevSecOps best practices by using a complete reference implementation that is available as a service and powered by IBM Cloud continuous delivery. In part 2 of this tutorial series, you use the toolchain template for continuous integration (CI) with security and compliance-related best practices in DevSecOps for Terraform. It is preconfigured for continuous deployment with inventory integration, change management with Git Repos and Issue Tracking, evidence collection, and deployment of the infrastructure to IBM Cloud.
Before you begin
Before you begin part 2 of this tutorial series, ensure that you complete the following prerequisites:
- Complete Part 1: Set up prerequisites.
- View the Getting started with DevSecOps in IBM Cloud - Part 1 video.
- Start the IBM Cloud console. The console is the starting point of this tutorial.
Continuous Integration (CI) toolchain for Infrastructure as Code (IaC) introduction
The CI toolchain implements the following best practices:
- Runs a static code scanner on the infrastructure code repositories and performs Terraform linting checks on the Infrastructure code.
- Runs compliance checks on the infrastructure code to detect secrets and security vulnerabilities.
- Builds artifacts on every Git commit.
- Stores the built artifacts metadata in inventory repository.
- Automatically builds and validates any code that is merged into the target Git repository branch.
Guided setup overview for the CI toolchain
The Continuous Delivery service provides templates to guide you through the toolchain setup process in a logical order, presenting you with the recommended configuration options that you need to create your toolchain.
A progress indicator shows the steps to complete the configuration. Use the progress indicator to navigate to previous steps. The configuration options for the current step are displayed in the main area of the page.
To advance to the next step, click Start. You can advance to the next step only when the configuration for the current step is complete and valid. Navigate to the previous step by clicking Back.
Some steps include a Switch to advanced configuration toggle. These steps by default present you with the minimum recommended configuration needed. However, advanced users that need finer grained control can click the Switch to advanced configuration toggle to reveal all options for the underlying integration.
When all the steps are successfully completed, create the toolchain by clicking Create on the Summary step.
You can always go back to previous steps in the guided installer. The toolchain installer retains all the configuration settings from the successive steps.
Start the CI toolchain setup
Use one of the following options to access the template for the CI toolchain:
-
Click Create toolchain.
-
From the IBM Cloud console, click Menu
and select DevOps. On the Toolchains page,
click Create toolchain. On the Create a Toolchain page, click CI - Develop secure infrastructure as code with DevSecOps practices.
Set up the CI toolchain name and region
The Welcome page summarizes the purpose of the toolchain that is created, along with links to documentation and related materials. When you are ready to proceed, click Start to begin configuring the toolchain.
Review the default information for the toolchain settings:
- The toolchain's name identifies it in IBM Cloud. Make sure that the toolchain's name is unique within your toolchains for the same region and resource group in IBM Cloud.
- The toolchain region is the region where the toolchain itself and related resources (Git repos, pipelines, for example) are created. Toolchains can deploy to any region, regardless of where they are created.
Set up CI tool integrations
Configure multiple repositories during the guided setup, as described in the following sections.
You can clone the sample repositories or you can use your own, but the toolchain supports Git Repos and Issue Tracking repositories only.
Infrastructure code
The infrastructure source code repository is hosted on GitHub Enterprise. For consistency, this tutorial refers to Git Repos and Issue Tracking.
The recommended options are displayed by default, but you can click the Switch to advanced configuration toggle to see all of the configuration options available for the underlying Git integration. The default behavior of the toolchain is Use default sample infrastructure code, which clones the sample application into an IBM-hosted Git Repos and Issue Tracking repository.
Enter a name for the IBM-hosted Git Repos and Issue Tracking repository that is created by the toolchain as your infrastructure code repository.
The region of the repository remains the same as the region of the toolchain.
To link an existing infrastructure code repository for the toolchain, select Bring your own infrastructure code, and provide it as input to the Repository URL field. The toolchain currently supports linking only to existing Git Repos and Issue Tracking repositories. For more information, see Bringing your own infrastructure code to DevSecOps.
Inventory
The inventory repository records details of artifacts that are built by the CI toolchains.
Issues
The issues repository records issues that are found while the CI pipeline is running.
Secrets
Specify the secret vault integrations to be added to your toolchain by using the toggles provided. For more information, see Managing IBM Cloud secrets.
This tutorial uses IBM Cloud® Secrets Manager as the vault for secrets. The Region, Resource group, and Service name fields are automatically populated based on available choices. Click the drop-down indicators to see the other choices.
Evidence Storage
The evidence repository stores all the evidence and artifacts that are generated by the DevSecOps CI pipeline.
Toggle the Cloud Object Storage bucket slider to also store all the evidence in a Cloud Object Storage bucket, which you can configure on the next page.
Cloud Object Storage bucket
To use this feature, you must have a Cloud Object Storage instance and a bucket. Click Using IBM Cloud Object Storage buckets as an evidence locker to create a Cloud Object Storage instance. See Configuring IBM Cloud Object Storage for storing evidence to configure a bucket that can act as a compliance evidence locker.
- Cloud Object Storage instance, Bucket name, and Cloud Object Storage endpoint fields are automatically populated.
- Enter the Service ID API key:
- Preferred: An existing key can be imported from a secrets vault by clicking the key icon.
- You can also copy and paste an existing key.
The endpoint field is optional. Select or provide the endpoint during the setup of the toolchain or during the pipeline run.
Deploy
- IBM Cloud API Key: The API key interacts with the IBM Cloud CLI tool in several tasks, it also clones the Git repository.
- Preferred: Import an existing key from a secrets vault by clicking the key icon.
- You can also copy and paste an existing key.
- Create a key by clicking New +.
Save the newly generated API key to a secrets vault.
If the API key is valid and has sufficient access, the schematic workspace, Git repos, and resource group are automatically populated. You can change any of these fields to match your configuration if needed.
Configure the target schematic workspace where the terraform states are maintained:
- Schematic workspace name: The name of the schematic workspace.
- Default:
toolchainName-toolchainId-pipelineID
- Default:
Signing
Any code that is built by this toolchain and recorded in the inventory must be signed before the code is delivered to production environment.
To enable signing, provide a GPG key. You can select an existing key from a secret store or input the value directly. Alternatively you can create a new GPG key by clicking NEW.
Ensure that the key follows the appropriate encoding as required by the chosen tool to store secrets.
DevSecOps for IaC creates a detached signature file for the tar.gz file that is created containing the Terraform configuration.
You can also sign the Git release tag that is created by using this GPG key. Add the GPG public key to the Git user profile to verify the signed commits. Use gpg --armor --output <public key file> --export <Email Address> to get the public key content if the gpg --list-keys shows the appropriate GPG key entry.
Optional tools
Slack
Configure Slack to receive notifications about your PR or CI pipeline events:
- Slack webhook
- Preferred: Import an existing webhook from a secrets vault by clicking the Key icon.
- Copy and paste an existing webhook.
- Slack channel: an existing Slack channel to post messages to.
- Slack team name: the phrase before .slack.com in our team URL. For example, if your team URL https://team.slack.com, the team name is team.
- Automated Slack notifications: Customize your selection by choosing the events for which you want to receive notifications.
You can add the Slack Tool after toolchain creation.
Create the CI toolchain
On the Summary page, click Create, and wait for the toolchain to be created.
Other toolchain integrations can be configured after the pipeline is created.
Explore the CI toolchain
The DevSecOps CI IaC toolchain looks like this:
The toolchain contains two pipelines:
- pr-pipeline (PR): triggers when a new Merge Request (MR) is submitted in the infrastructure code repository.
- ci-pipeline (CI): triggers when an MR is merged to main in the infrastructure code repository, but can be manually triggered.
The sample that is used in this tutorial expects a resource_group to create the KeyProtect instance. The default resource_group name is Default. If the default is not appropriate for your account, replace
the default resource_group variable in the variables.tf file or specify the resource_group variable by setting the TF_VAR_resource_group environment property in the PR pipeline and CI pipeline
to the appropriate resource group name.
Run the pr pipeline
To start the pr pipeline, create a merge request in your application repository:
- From the CI toolchain page, click the application repository tile. By default it gets created with a name
compliance-app-<timestamp>. - Create a branch from main branch.
- Update some code such as an app or readme file and save changes.
- Submit merge request.
- On the CI toolchain page, click the pr-pipeline tile. Verify that the PR pipeline is triggered (by the creation of the Merge Request).
- Wait for the PR pipeline run to complete.
The corresponding Merge Request that is in your application repository is in Pending state until all the stages of PR pipeline finish successfully.
When the PR pipeline run is successful, click it to explore numerous steps completed.
- Navigate back to the merge request.
- Merge the request so that your changes are copied to the main branch of your application repository: the CI pipeline is automatically triggered.
Run the CI pipeline
Start the CI pipeline in one of the following ways:
- Automatically: after a successful PR pipeline, by approving and merging the PR to the main branch.
- Manually: select the Delivery Pipeline card, click Run Pipeline, and select Manual Trigger.
In this tutorial, the CI Pipeline is triggered after you merge your code changes to the main branch of your application repository
- On the CI toolchain page, click the ci-pipeline tile.
- Observe: a pipeline-run is running. Wait for the pipeline-run to complete.
When the CI pipeline run completes, click it to explore the completed steps.
Evidence from all compliance checks in the CI pipeline is uploaded to the evidence-locker repository, but also to the Cloud Object Storage bucket if you enabled it during setup.
To evaluate if you have any failures in your pipeline run, check the final step of your pipeline, which has a pipeline evaluator.
Customizing the pipeline
Custom scripts are extension points in the pipeline where adopters, teams, and users can provide scripts to run custom tasks that are required by their continuous integration and continuous deployment strategies.
Custom scripts control the pipeline stages. Use the pipeline-config.yaml configuration file to configure the behavior of stages, script content, and the base image that runs the scripts. The scripts and configuration for pipeline
stages are loaded from a Git repository (repo) that can either be the application (app) repo (similar to .travis.yml or Jenkinsfile) or a custom repo.
For more information on customizing the CI pipelines, see Custom scripts.
Wrapping up
By completing the CI part of this tutorial, you:
- Created a DevSecOps CI toolchain for Terraform.
- Modified some code in the infrastructure code repository.
- Ran the pr pipeline before you merge your changes.
- Ran the ci-pipeline to build, test, and deploy your changes to your dev environment.
Next steps
Continue to Part 3: Set up a CD toolchain for Infrastructure as Code.