Understanding inventory for DevSecOps

Branches

The inventory is implemented in a Git repository (repo). Git is self-sufficient in tracking and auditing changes.

Branches are used as environments. The main branch (master) is written and updated by the continuous integration pipeline. Other environments are updated from the master branch by using promotions. For more information about promotions, see the Promotion section.

Inventory content

The inventory contains an inventory Entry for every artifact that participates in deployment. One inventory Entry points to a single artifact. Inventory entries are JSON files that can be structured in folders and are named by using the entry name.

Inventory folders are part of the entry name.

/
├── auth
│   ├── service
│   └── db
├── ui
│   └── service
├── helm-charts
│   └── main_service
└ main_service

Example

Inventory entry for the image type of asset

{
  "repository_url": "https://github.com/test-org/compliance-app-20201211", # source code repository url of the image artifact
  "artifact": "us.icr.io/namespace/hello-compliance-app:20201217081811-master-b85e3d472e9cc35b429c39e8c3f9eb282738c20a@sha256:da36831d5154307ac9ca4b8d900df2da0c6c14754977c32479dc62994b5722d0", # image artifact name, should be in a format like <static_name>:<version>@sha256:<sha256> OR <static_name>@sha256:<sha256>
  "build_number": 21, # pipeline build number
  "commit_sha": "b85e3d472e9cc35b429c39e8c3f9eb282738c20a",  # should be a proper git commit sha
  "name": "hello-compliance-app", # name of the inventory entry file name
  "pipeline_run_id": "f21321bf-9054-4bf3-80a8-4fb34743b7d9", # pipeline run id where artifact was built
  "version": "v1",  # version of the artifact
  "app_artifacts": {}, # any additional information can be stored here
  "type": "image",
  "sha256": "sha256:da36831d5154307ac9ca4b8d900df2da0c6c14754977c32479dc62994b5722d0",
  "provenance": "us.icr.io/namespace/hello-compliance-app:20201217081811-master-b85e3d472e9cc35b429c39e8c3f9eb282738c20a@sha256:da36831d5154307ac9ca4b8d900df2da0c6c14754977c32479dc62994b5722d0", # The fully qualified URL where artifact is stored, in case of image artifact, it is same as the artifact field
  "signature": "owFNUX1IE3EY3vwAEy0VyoSJdkVWtu0+d3eTjNIIi0jETCSU3353cz+83V23myi2BlFgYiUFqSmZLciUzCJkRYJp9IEWFWXB0hAyrBQxEjIIuhFSf70vL8/7vM/zPi3JsaYEc+TnbqG8qrnTPDZ8w2WqiqwtacCghnQEgYQ5GzAkiLKO9PpoLyiwRtSsmugWNVGGIubE/D4bgpoNKXag60gCdo8oSYoVKl5VQsDAWIGqOkmcxAmSYHGO4AjC6gU+3eBxcYxICTRLijyEFOOiSR5SvMhBys2LLpIjWYqDJA6wwHYMeUG1+J8GL5CRW/TpVgFVG8VQ4vMAknE4BUA5OIoQGIKhKZwFkIeAdnECj+OCmxQADh2QoFmG5lkWUiTN8gJ0kDxPuxiWJAU8ekyvV6PegK54EcyGiqwDJItatg9Vy0D3a2IUpKg6UuS/T4KaaIC1fzuMDbfhmMGEvIY64FUxJ+Ew3PMU5SADgdNmKs5kTjBlrtsQ99iyY49nns8o6jsXWgkjPiYahClxVcrK5Flngumz2j7YdmD0ecutytsvxxNPHflKlRV2RyqD69NjC6Smji9XuukFyZ+lvM18gUr7F4NXge9YyZPik411tc1n9bKO/hDz7oGaX/ur94OnNHiwOG6n5ZsrsCsvtyvmkH4zOWlrRWuL7fzgj9zlcCTAhtu2LbeGLNfv3Ju0Jc9PZPk7Pm3Ov2CW+2xj8ReX6ooGamZ610yqRM/+8Qo2XnOeyZzbVKgs+f2+9unpJOve4Tmla+PdqftDWkHPm9Vq3nsyZ2DUkmP/nTb7qNw5l2SfWtiSemJx9nLaYOpx6amlOT18aeIwJQhHg1XOjJF9r18NXQvPuL9rH5tSQqbGhyN/AA=="  # The signature of the artifact
}

Inventory entry for the nonimage type of asset, like deployment, helm charts

{
  "repository_url": "https://github.com/test-org/compliance-app-20201211",  # source code repository url of the artifact
  "artifact": "deployment.yaml", # artifact name, must be contant every build
  "build_number": 21, # pipeline build number
  "commit_sha": "b85e3d472e9cc35b429c39e8c3f9eb282738c20a", # must be a proper git commit sha
  "name": "hello-compliance-app-deployment", # name of the inventory entry file name
  "pipeline_run_id": "f21321bf-9054-4bf3-80a8-4fb34743b7d9", # pipeline run id where artifact was built
  "version": "v1", # version of the artifact
  "app_artifacts": {}, # any additional information can be stored here
  "type": "deployment", # type of the artifact
  "sha256": "sha256:da36831d5154307ac9ca4b8d900df2da0c6c14754977c32479dc62994b5722d0",  # sha256 of the artifact
  "provenance": "https://raw.github.ibm.com/org/my-app/commit-1/deployment.yaml", # The fully qualified URL where artifact is stored
  "signature": "owFNUX1IE3EY3vwAEy0VyoSJdkVWtu0+d3eTjNIIi0jETCSU3353cz+83V23myi2BlFgYiUFqSmZLciUzCJkRYJp9IEWFWXB0hAyrBQxEjIIuhFSf70vL8/7vM/zPi3JsaYEc+TnbqG8qrnTPDZ8w2WqiqwtacCghnQEgYQ5GzAkiLKO9PpoLyiwRtSsmugWNVGGIubE/D4bgpoNKXag60gCdo8oSYoVKl5VQsDAWIGqOkmcxAmSYHGO4AjC6gU+3eBxcYxICTRLijyEFOOiSR5SvMhBys2LLpIjWYqDJA6wwHYMeUG1+J8GL5CRW/TpVgFVG8VQ4vMAknE4BUA5OIoQGIKhKZwFkIeAdnECj+OCmxQADh2QoFmG5lkWUiTN8gJ0kDxPuxiWJAU8ekyvV6PegK54EcyGiqwDJItatg9Vy0D3a2IUpKg6UuS/T4KaaIC1fzuMDbfhmMGEvIY64FUxJ+Ew3PMU5SADgdNmKs5kTjBlrtsQ99iyY49nns8o6jsXWgkjPiYahClxVcrK5Flngumz2j7YdmD0ecutytsvxxNPHflKlRV2RyqD69NjC6Smji9XuukFyZ+lvM18gUr7F4NXge9YyZPik411tc1n9bKO/hDz7oGaX/ur94OnNHiwOG6n5ZsrsCsvtyvmkH4zOWlrRWuL7fzgj9zlcCTAhtu2LbeGLNfv3Ju0Jc9PZPk7Pm3Ov2CW+2xj8ReX6ooGamZ610yqRM/+8Qo2XnOeyZzbVKgs+f2+9unpJOve4Tmla+PdqftDWkHPm9Vq3nsyZ2DUkmP/nTb7qNw5l2SfWtiSemJx9nLaYOpx6amlOT18aeIwJQhHg1XOjJF9r18NXQvPuL9rH5tSQqbGhyN/AA=="  # The signature of the artifact
}

Use cocoa inventory add command to write to the inventory.

CI writes to inventory

The master branch is populated from continuous integration builds. The last commit in the target (in this case named staging) contains a tag that shows that it was the last concluded deployment.

If the default branch for the inventory is switched out to a different branch, you must rebase the commits from the previous default branch into the new default branch to make the Git commit history linear.

You can skip writing to inventory by customizing the release script. For more information, see Release to inventory.

Promotion

A pull request is created when you promote to a target branch happens. Pull request contents populate the Change Request fields. After the pull request is reviewed, you can merge it.

Delta and deployment

After the promotion pull request is merged, the Deployment pipeline can start. The Deployment delta is the difference between the contents of the last concluded deployment and the current deployment. The Deployment delta lists the inventory items that are being deployed.

Inventory conclusion

When the deployment finishes, you can move the latest tag ahead.

During the dev mode trigger, the tags will not be advanced. The purpose of the dev mode trigger is solely for testing the CD pipeline only.

Promote to further environments

You can promote and deploy from any branch to another one.

Inventory landscape

The current deployed state contains the content to deploy to an environment. Every promoted commit in the target branch contains the relevant Pipeline run ID and Change Request ID as a tag. Some commits can have multiple tags, for example, when you are retrying a failed deployment or deploying again. The Inventory holds every piece of information to replay the deployments.

Setup for a single target with multiple regions

Multiple latest tags are introduced for a single target environment so that multiple continuous deployment pipelines can work on the same target, for different types of use cases. You can use, for example, the same target environment (such as us-south or eu-de) for multiple regions in the prod target environment and the inventory branch.

You do not need to set up a different branch for each region by using the region property, such as us-south-prod and eu-de-prod, and run the promotion redundantly. Instead, specify these additional targets for the same inventory branch, and then use them as Git tags.

In this setup, the prod branch has multiple latest tags on the same branch, such as us-south_prod_latest and eu-de_prod_latest. Each continuous deployment pipeline that is responsible for each region can use those tags to deploy.

For example, a set of changes that you plan to deploy everywhere might be released to a single region first, and then gradually deployed to other regions by using continuous deployment pipelines to target those regions.

CLI commands

1. Create a promotion pull request from master to the target branch in staging.

   cocoa inventory promote \
     --source="master" \
     --target="staging" \
     --priority="moderate" \
     --assigned-to="assignee@ibm.com" \
     --description="Change description" \
     --purpose="Change purpose" \
     --impact="Change impact" \
     --backout-plan="Details on backout and rollback")
   

2. Conclude a deployment by moving the target_latest tag to the same commit as the pipeline-run-id tag.

   cocoa inventory label move \
     --to-label="${PIPELINE_RUN_ID}" \
     "target_latest"
   

Git and GitHub CLI

1. Create a promotion pull request from master to the target branch in staging.

   promote() {
   
     if [ -z $1 ] || [ -z $2 ]; then
       echo "Missing source and target"
       exit 1
     fi
   
     local source="$1"
     local target="$2"
   
     if ! git show-ref "refs/remotes/origin/$target"; then
       # Create a new target branch, from the beginning of master
       git checkout master
       git checkout -b "$target" $(git rev-list --max-parents=0 HEAD)
       git_push
     fi
   
     git checkout "$source"
     git pull --rebase
   
     # Create a promotion branch for the PR
     # this can be discarded after the Promotion PR merge
     git checkout -b "promote-$source-to-$target"
     git push --set-upstream origin "promote-$source-to-$target"
   
     # Create PR from promotion branch to target branch
     gh pr create \
       --base "$target" \
       --head "promote-$source-to-$target" \
       --title "Promote $source to $target" \
       --body "" \
       --repo "https://github.com/org/inventory-repository"
   
     # promotion branch can be deleted once the PR was merged
   }
   
   $ promote master staging
   

2. Conclude a deployment by moving the target-latest tag to the same commit as the pipeline-run-id tag.

   conclude () {
     local target="$1"
     local tag="$2"
   
     latest="$1-latest"
   
     # remove the latest tag
     git push origin ":refs/tags/$latest"
     # find the commit hash of the target tag
     sha=$(git rev-list -n 1 $tag)
   
     # add the latest tag to the same commit of the target tag
     git tag -fa "$latest" -m "" $sha
     git push --tags --force
   }
   
   $ conclude staging pipeline-run-fe33b05c
   

3. Revert staging to an earlier state by using Git and the GitHub CLI.

   revert () {
     local branch="$1"
     local commit="$2"
   
     # create a revert branch from the target branch
     git checkout "$branch"
     git pull --rebase
     git checkout -b "$branch-revert"
   
     # revert commits since the target commit, then commit and push
     git revert -n $(git rev-list --no-merges HEAD...$commit)
     git commit -m "revert $branch to $commit"
     git push --set-upstream origin "$branch-revert"
   
     # create PR from revert branch to the target branch
     gh pr create \
       --base "$branch" \
       --head "$branch-revert" \
       --title "Revert $branch to $commit" \
       --body "" \
       --repo "$REPO"
   
     # revert branch can be deleted once the PR was merged
   }
   
   $ revert staging ba3b8e5ed3320e6b4981077e1a1627f08de4f511