DevSecOps Pipelines stages and custom scripts explained

Pull Request (PR) pipeline stages

The PR pipeline runs compliance checks on a pull request for a given application repository and acts as a safeguard for merging into the main branch. The PR pipeline is triggered by opening or updating a pull request against a given branch.

For more information, see Pull Request (PR) pipeline documentation.

Continuous Integration (CI) pipeline stages

The CI pipeline builds deployable artifacts from application repositories. It scans, tests, and signs built artifacts while also collecting evidence on each stage to be released in the inventory and tracked through the deployment and change management.

For more information, see Continuous Integration (CI) pipeline documentation.

Continuous Deployment (CD) pipeline stages

The CD pipeline compiles all the content for evidence and the change request summary. It deploys the build to an environment, such as staging or production, while uploading all evidence to the evidence locker.

For more information, see Continuous Deployment (CD) pipeline documentation.

Continuous Compliance (CC) pipeline stages

The CC pipeline performs periodic scans on deployed artifacts and their source repositories. It scans the latest deployed state in the inventory and creates, updates or auto-closes incident issues in the incident repository.

For more information, see Continuous Compliance (CC) pipeline documentation.

Pipeline compliance checks

The following scans and checks are performed during the code-compliance-checks task within the PR, CI, and CC pipelines.

Stage descriptions

• setup: Run scripts to complete setup tasks that are done outside the pipeline run context. These tasks persist for the entire pipeline run. For example, setting up an external environment, setting up branch protection rules, performing health checks, checking environments, or initializing external services.
• test: Run unit tests, include all of your tests that run on your codebase, and pre-deploy time.
• static-scan: Run a static code analyzer tool.
• peer-review: Run scripts to check whether the pull request is reviewed and has the minimum number of required approvals.
• dynamic-scan: Run dynamic scan on the application.
• containerize: Create the artifacts that are required by your app, such as Docker images, RPMs, and Java Archive (JARS) files.
• scan-artifact: Scan the artifacts, for example, by using Container Registry Vulnerability Advisor for Docker images.
• sign-artifact: Sign the artifacts.
• release: Add artifacts to the inventory by marking them available for deployment.
• deploy: Deploy artifacts to an environment such as test and dev or staging and prod.
• acceptance-test: Run tests on deployed artifacts. You can also include your post-deployment tests in this stage.
• verify-artifact: Run scripts to check whether the artifacts that are signed in the CI pipeline have valid signatures.
• finish: This stage is customizable stage and is always executed, irrespective of pipeline status.

Configuration in pipeline-config.yaml

You can use a .pipeline-config.yaml configuration file to extend pipeline behavior.

File location

For pull request and continuous integration pipelines, store the .pipeline-config.yaml configuration file in an app repo in the same manner as the .travis.yml, or Jenkinsfile files. For continuous deployment pipelines, store this file in a dedicated repo.

For all pipelines, you can customize the location and source of the .pipeline-config.yaml file with the following pipeline UI parameters:

• pipeline-config to set the path of the configuration file. The default value is .pipeline-config.yaml.
• pipeline-config-repo to set the repo to pull the configuration and scripts from. The default value is the continuous integration app repo.
• pipeline-config-branch to use as the branch for the configuration in the config repo. The default value is the continuous integration app repo branch, and the master branch in continuous deployment.

Configuration parameters

The configuration in the .pipeline-config.yaml file supports the following parameters:

These settings are not pipeline parameters, they must be part of your .pipeline-config.yaml file.

• image: You can use any Docker image that is accessible by the pipeline worker as a base image for the task.

• script: The script to run in the stage. Because this field is used as a script file, ensure that the contents act like a script file in the base image that you provided. You can include other scripts next to this configuration file, and refer to them from this entry point. For example:

test:
  image: ibmcom/pipeline-base-image:2.7
  script: |
    #!/bin/sh
    scripts/lint.sh
    scripts/unit-test.sh

• dind: Specify whether to enable docker-in-docker for the script context. The default setting is false.

• abort_on_failure: By default, the pipeline stops when the script fails. Setting this to false marks the stage in a warning (amber state) (it passed with warning), allows the pipeline to continue, and lets the failed job be referenced in the evidence later.

• image_pull_policy: Set the pod ImagePullPolicy setting that is provided in the image, for the base image. Possible values are the same as valid values for Kubernetes: Always or IfNotPresent. The default value is IfNotPresent.

• configmap: Pull key-value pairs from a provided configmap that is accessible by PipelineRun. The following values are supported:

  • config-name: The direct configmap name syntax. The stage runner tries to access and mount the configmap named config-name.
  • $prop-name: The indirect configmap name syntax. The configmap name is looked up from the environment-properties configmap, which contains every environment value that is set on the pipeline UI. For example, if there is a prop: my-config entry in the environment properties, then my-config is used for $prop.

• secret: Pull key-value pairs from a provided configmap that is accessible by PipelineRun. The following values are supported:

  • secret-name: The direct Secret name syntax. The stage runner tries to access and mount the Secret named secret-name.

  • $prop-name: The indirect Secret name syntax. The Secret name is located in the environment-properties configmap that contains every environment value that is set on the pipeline UI. For example, if the environment-properties configmap contains the my-secret entry property, my-secret is used for $prop.

• runAfter: Run the current stage after the stage that specified in this property completes. Set the value with the stage name that you used in .pipeline-config.yaml. Use this property sparingly and confirm that the stage that is specified by this property exists in the pipeline. If the stage does not exist, the pipeline can go into a deadlock.

• skip: If set to true, bypass the current stage, if possible, in v10 pipelines. Not all stages can be bypassed. The following table lists the stages that can be skipped during the pipeline run.

version: '1' # fixed, this value is used to track schema changes

# `setup` runs right after the app repo is cloned
setup:

  # the pipeline will break in case setup fails (default is true)
  abort_on_failure: true

  # any docker image can be used which is accessible by the private worker
  image: ibmcom/pipeline-base-image:2.7

  # you can reference configmaps, each key in the configmap is going to be available as `/config/{key}`
  configmap: my-config

    # `$prop` is the indirect config map syntax, the concrete configmap is looked
    # up from the `environment-properties` configmap
    #
    # Eg. if there's a `prop: my-config` entry in the environment properties,
    # then `my-config` is going to be used for `$prop`
  configmap: $prop

  # the mechanism described works for secrets as well!
  secret: $my-secrets

  # the script is executed inside the checked out app repo
  script: |
    #!/bin/sh
    ...

# `test` runs after `setup`, but before building the docker image
test:
  image: ibmcom/pipeline-base-image:2.7
  script: |
    #!/bin/sh
    ...

# `static-scan` runs after `test`, but before building the docker image
static-scan:
  image: ibmcom/pipeline-base-image:2.12
  script: |
    #!/bin/sh
    ...


# `deploy` runs after building the docker image
deploy:
  image: ibmcom/pipeline-base-image:2.7

  # the script has access to the built docker image, which is available at `/config/image`
  script: |
    #!/bin/sh

    cat /config/image

# `dynamic-scan` runs after `deploy`, but before the acceptance test run
dynamic-scan:
  image: ibmcom/pipeline-base-image:2.12
  script: |
    #!/bin/sh
    ...

# `acceptance-test` runs after `deploy`
acceptance-test:
  image: ibmcom/pipeline-base-image:2.7
  script: |
    #!/bin/sh
    ...

Accessing pipeline environment secrets and properties

Environment properties and secrets (defined on the pipeline UI) are accessible through pipelinectl:

# if you set my_config_value on the UI, it's accessible via `get_env`
my_config_value=$(get_env my_config_value)

You can set other variables by using set_env:

set_env my_config_value "some value"

# later on, even in another stage
my_config_value=$(get_env my_config_value)

These values are not persisted, they're scoped for a single pipeline run.

my-artifactory-email
my-artifactory-user
my-artifactory-password
my-artifactory-url

kubectl create secret docker-registry regcred \
  --o json \
  --dry-run=client \
  --docker-server="$(get_env my-artifactory-url)" \
  --docker-username="$(get_env my-artifactory-user)" \
  --docker-email="$(get_env my-artifactory-email)" \
  --docker-password="$(get_env my-artifactory-password)" \
   | jq -r '.data[".dockerconfigjson"]'

{"auths":{"us.icr.io":{"username":"user@ibm.com","password":"secret-password","email":"user@ibm.com","auth":"dXNlckBpYm0uY29tOnNlY3JldC1wYXNzd29yZA=="}}}

Parameters for custom scripts

In addition to the pipeline environment variables and secrets, these parameter values are provided for user script stages.

You can access repos, artifacts, files, and environment variables by using the pipelinectl command in the user scripts. Use the following methods: list_repos, load_repo, load_artifact, list_artifacts, and get_env.

Accessing toolchain data from custom scripts

You can read toolchain data from the toolchain.json file that is provided by the pipeline at the path that is stored in the TOOLCHAIN_DATA_PATH environment variable. You can also use the toolchain integration and jsonpath, and access them in /config in the same way as any other variable. Or, you can use get_env, which has the same effect.

TOOLCHAIN_CONFIG_JSON="$(get_env TOOLCHAIN_CONFIG_JSON)"
toolchain_id=$(jq -r '.toolchain_guid' "$TOOLCHAIN_CONFIG_JSON")

Shared workspaces

Because these custom scripts operate in the same workspace, you can pass data between them by writing it into a file and then reading it in the other task. This workspace is mounted on a path that is provided in the $WORKSPACE variable.

echo -n "$DIGEST" > "${WORKSPACE}"/image-digest

image_digest=$(cat "${WORKSPACE}"/image-digest)

Custom finish stages

The finish stage has three steps:

This pipeline status can be determined in two ways:

1. You can get the overall status of the pipeline run from the environment variable PIPELINE_STATUS. The value can be "Succeeded" , "Failed" or "" (case sensitive). PIPELINE_STATUS comes from the underlying Tekton framework.
2. Because the evaluate step evaluates a pipeline run from the perspective of evidence, this evaluation might differ from the value set in PIPELINE_STATUS. To see the pipeline status based on evidence evaluation, use the environment variable finish_evaluate_exit_code.

Stage results API

The result states of these custom stages are saved with pipelinectl. The result state of a stage is either success, failure, or skipped. These results can be used later to evaluate whether the custom stage ran successfully.

The following tasks and stages are available:

# List saved stage results
$ get_data result
detect-secrets
branch-protection

# Get stage result
$ get_data result detect-secrets
success