IBM Cloud Docs
Preparing to deploy DevSecOps toolchains in a firewalled environment

Preparing to deploy DevSecOps toolchains in a firewalled environment

In a secure enterprise setting, the DevOps team builds toolchains in a private hosted environment to make sure compliance, control, and automation are fully functional in a toolchain. The team streamlines the software delivery to maintain the complete ownership of their infrastructure.

During the toolchain creation and execution in a private hosted environment, the DevOps team faces integrating challenges, security management issues through the pipeline workloads.

To overcome the challenges. Follow the steps that outlines the prerequisites, configuration and permissions to create the DevSecOps solution for the Apps Stack in an IBM Cloud® account when the customers source code is hosted in a firewalled environment.

Planning and setting infrastructure

Identify your hosting environment such as on-premises data center, private cloud, or hybrid cloud, where to deploy, and how to network and secure your environment.

Installing a private worker on your infrastructure

To install the private worker, see installation of delivery pipeline private worker documentation.

Securing the environment

You need to secure the environment by using IAM, secrets manager, API keys, or encryption. Create an IBM Cloud® service API key with right permissions for Deployable Architecture (DA) creation. Refer to the permissions outlined in DevSecOps Solutions for Apps Stack.

  • Create an IBM Cloud® service API key for private worker.

  • Create an access token for connecting the source repository.

Configuring the network

The firewall must enable network connectivity between the customer environment and IBM Cloud endpoints. For more information about IBM Cloud access endpoints, see IBM Cloud endpoints.

Use Case Scenarios and application contexts

Configuring Git triggering

You need to create and configure pipeline triggers so that events on the firewalled repositories run the correct pipeline triggers.

Following are the three triggers:

CI pipeline trigger

  1. Log in to IBM Cloud console.

  2. Click the Menu icon hamburger icon > Platform Automation > Toolchains.

  3. Click your CI toolchain > <your CI pipeline>.

  4. Get the pipelineID from the URL https://cloud.ibm.com/devops/pipelines/tekton/<CI_PIPELINE_ID>?env_id=ibm:yp:<REGION>

  5. Create the following JSON file, data_push.json :

    {
    "enabled": true,
    "event_listener": "ci-listener-gitlab",
    "filter": "header['x-gitlab-event'] == 'Push Hook' && body.ref == 'refs/heads/main'",
    "max_concurrent_runs": 1,
    "name": "Git Trigger",
    "secret": {
        "key_name": "X-Gitlab-Token",
        "source": "header",
        "type": "token_matches",
        "value": "<WEBHOOK_SECRET>"
    },
    "type": "generic",
    "worker": {
        "id": "inherit"
    }
}

  6. Get IBM account credentials

    • Login to the IBM Cloud CLI with ibmcloud login.
    • Obtain an IAM Bearer token as IAM_TOKEN=$(ibmcloud iam oauth-tokens)

  7. Set Environment variables

    IAM_TOKEN=${IAM_TOKEN:33}
CI_PIPELINE_ID=36127d0b-1858-404b-b3ee-7a68aefec994
REGION=us-south

  8. Run the following CURL command.

    `curl -X POST --location --header "Authorization: Bearer ${IAM_TOKEN}" --header "Accept: application/json" --header "Content-Type: application/json" --data @data_push.json "https://api.${REGION}.devops.cloud.ibm.com/pipeline/v2/tekton_pipelines/${CI_PIPELINE_ID}/triggers"`

    The JSON response from the CURL command looks as shown in the code block.

    {
  "type": "generic",
  "name": "Git Trigger",
  "event_listener": "ci-listener-gitlab",
  "id": "ef110ae8-2395-4928-9dba-f80b49cab276",
  "filter": "header['x-gitlab-event'] == 'Push Hook' && body.ref == 'refs/heads/main'",
  "secret": {
    "type": "token_matches",
    "value": "hash:SHA3-512:7a4f5b0701a921c9054eb41d97258979389922592801d21d213bdb5aa99cb8aec71368153fe0a084e7c9e5be119976a8279cfa0e794de90036fe138a57e4fd31",
    "source": "header",
    "key_name": "X-Gitlab-Token"
  },
  "properties": [],
  "webhook_url": "https://devops-api.us-south.devops.cloud.ibm.com/v1/tekton-webhook/f6e201e0-ba61-4e88-9a1b-6fc81febb952/run/ef110ae8-2395-4928-9dba-f80b49cab276",
  "max_concurrent_runs": 1,
  "enabled": true,
  "href": "https://api.us-south.devops.cloud.ibm.com/pipeline/v2/tekton_pipelines/f6e201e0-ba61-4e88-9a1b-6fc81febb952/triggers/ef110ae8-2395-4928-9dba-f80b49cab276"
}

Copy the href field from the response and add the private keyword in the endpoint, as shown in the example - https://api.private.us-south.devops.cloud.ibm.com/pipeline/v2/tekton_pipelines/f6e201e0-ba61-4e88-9a1b-6fc81febb952/triggers/ef110ae8-2395-4928-9dba-f80b49cab276

Creating source repository webhook for CI pipeline

You need to create a repository webhook to invoke the created trigger.

  1. Navigate to the webhooks page of your firewalled source code repository, for example https://my-private-gitlab/test-admin/code-engine-compliance-app/-/hooks
  2. Click the Add new webhook.
  3. Complete the URL field by using the modified href field from the trigger created earlier. For example,https://api.private.us-south.devops.cloud.ibm.com/pipeline/v2/tekton_pipelines/f6e201e0-ba61-4e88-9a1b-6fc81febb952/triggers/ef110ae8-2395-4928-9dba-f80b49cab276
  4. Choose a meaningful name. For example, CI Pipeline webhook.
  5. Complete the Secret token with the <WEBHOOK_SECRET> value used when creating the pipeline trigger
  6. Select Push events > All branches as the Trigger options.
  7. Click Add webhook to persist your changes.

The value for the URL field can be retrieved from the trigger Webhook URL field value through the pipeline UI after the CI pipeline creation. The secret used by the webhook and trigger pair can be modified.

PR pipeline trigger

  1. Log in to IBM Cloud console.

  2. Click the Menu icon hamburger icon > Platform Automation > Toolchains.

  3. Click your PR toolchain > <your CR pipeline>.

  4. Get the pipelineID from the URL. For example, https://cloud.ibm.com/devops/pipelines/tekton/<PR_PIPELINE_ID>?env_id=ibm:yp:<REGION>.

  5. Create the following JSON file, data_merge.json.

    {
    "enabled": true,
    "event_listener": "pr-listener-gitlab",
    "filter": "header['x-gitlab-event'] == 'Merge Request Hook' && (body.object_attributes.action == 'open' || body.object_attributes.action == 'update' || body.object_attributes.action == 'reopen') && body.object_attributes.target_branch == 'main'",
    "max_concurrent_runs": 1,
    "name": "PR Git Trigger",
    "secret": {
        "key_name": "X-Gitlab-Token",
        "source": "header",
        "type": "token_matches",
        "value": "<WEBHOOK_SECRET>"
    },
    "type": "generic",
    "worker": {
        "id": "inherit"
    }
}

  6. Get IBM account credentials

    • Login to the IBM Cloud CLI with ibmcloud login.
    • Obtain an IAM Bearer token as IAM_TOKEN=$(ibmcloud iam oauth-tokens)

  7. Set Environment variables

    IAM_TOKEN=${IAM_TOKEN:33}
CI_PIPELINE_ID=36127d0b-1858-404b-b3ee-7a68aefec994
REGION=us-south

  8. Run the following CURL command.

    `curl -X POST --location --header "Authorization: Bearer ${IAM_TOKEN}" --header "Accept: application/json" --header "Content-Type: application/json" --data @data_merge.json "https://api.${REGION}.devops.cloud.ibm.com/pipeline/v2/tekton_pipelines/${PR_PIPELINE_ID}/triggers"`

    The JSON response from the CURL command looks as shown in the code block.

    {
  "type": "generic",
  "name": "PR Git Trigger",
  "event_listener": "ci-listener-gitlab",
  "id": "ef110ae8-2395-4928-9dba-f80b49cab276",
  "filter": "header['x-gitlab-event'] == 'Merge Request Hook' && (body.object_attributes.action == 'open' || body.object_attributes.action == 'update' || body.object_attributes.action == 'reopen') && body.object_attributes.target_branch == 'main'",
  "secret": {
    "type": "token_matches",
    "value": "hash:SHA3-512:7a4f5b0701a921c9054eb41d97258979389922592801d21d213bdb5aa99cb8aec71368153fe0a084e7c9e5be119976a8279cfa0e794de90036fe138a57e4fd31",
    "source": "header",
    "key_name": "X-Gitlab-Token"
  },
  "properties": [],
  "webhook_url": "https://devops-api.us-south.devops.cloud.ibm.com/v1/tekton-webhook/f6e201e0-ba61-4e88-9a1b-6fc81febb952/run/ef110ae8-2395-4928-9dba-f80b49cab276",
  "max_concurrent_runs": 1,
  "enabled": true,
  "href": "https://api.us-south.devops.cloud.ibm.com/pipeline/v2/tekton_pipelines/f6e201e0-ba61-4e88-9a1b-6fc81febb952/triggers/ef110ae8-2395-4928-9dba-f80b49cab276"
}

Copy the href field from the response and add the private keyword in the endpoint. For example, https://api.private.us-south.devops.cloud.ibm.com/pipeline/v2/tekton_pipelines/f6e201e0-ba61-4e88-9a1b-6fc81febb952/triggers/ef110ae8-2395-4928-9dba-f80b49cab276

Creating source repository webhook for PR pipeline

You need to create a repository webhook to invoke the created trigger.

  1. Navigate to the webhooks page of your firewalled source code repository. For example, https://my-private-gitlab/test-admin/code-engine-compliance-app/-/hooks.
  2. Click the Add new webhook.
  3. Complete the URL field using the modified href field from the trigger created earlier. For example, https://api.private.us-south.devops.cloud.ibm.com/pipeline/v2/tekton_pipelines/f6e201e0-ba61-4e88-9a1b-6fc81febb952/triggers/ef110ae8-2395-4928-9dba-f80b49cab276.
  4. Choose a meaningful name. For example, PR Pipeline webhook.
  5. Complete the Secret token with the <WEBHOOK_SECRET> value used when creating the pipeline trigger
  6. Select Push events > All branches as the Trigger options.
  7. Click Add webhook to persist your changes.

The value for the URL field can be retrieved from the trigger Webhook URL field value in the pipeline UI at any time post creation. The secret that is used by the webhook and trigger pair can be modified.

CD pipeline trigger

  1. Log in to IBM Cloud console.

  2. Click the Menu icon hamburger icon > Platform Automation > Toolchains.

  3. Click your CD toolchain > <your CD pipeline>.

  4. Get the pipelineID from the URL. For example, https://cloud.ibm.com/devops/pipelines/tekton/<CD_PIPELINE_ID>?env_id=ibm:yp:<REGION>.

  5. Create the following JSON file, data_cd_merge.json :

    
    "enabled": true,
    "event_listener": "promotion-validation-listener-gitlab",
    "filter": "header['x-gitlab-event'] == 'Merge Request Hook' && (body.object_attributes.action == 'open' || body.object_attributes.action == 'reopen') && body.object_attributes.target_branch == 'prod'",
    "max_concurrent_runs": 1,
    "name": "Git Promotion Validation Trigger",
    "secret": {
        "key_name": "X-Gitlab-Token",
        "source": "header",
        "type": "token_matches",
        "value": "<WEBHOOK_SECRET>"
    },
    "type": "generic",
    "worker": {
        "id": "inherit"
    }

  6. Get IBM account credentials

    • Login to the IBM Cloud CLI with ibmcloud login.
    • Obtain an IAM Bearer token as IAM_TOKEN=$(ibmcloud iam oauth-tokens)

  7. Set Environment variables

    IAM_TOKEN=${IAM_TOKEN:33}
CI_PIPELINE_ID=36127d0b-1858-404b-b3ee-7a68aefec994
REGION=us-south

  8. Run the following CURL command.

    `curl -X POST --location --header "Authorization: Bearer ${IAM_TOKEN}" --header "Accept: application/json" --header "Content-Type: application/json" --data @data_cd_merge.json "https://api.${REGION}.devops.cloud.ibm.com/pipeline/v2/tekton_pipelines/${CD_PIPELINE_ID}/triggers"`

    The JSON response from the CURL command looks as shown in the code block.

    {
  "type": "generic",
  "name": "Git Promotion Validation Trigger",
  "event_listener": "ci-listener-gitlab",
  "id": "ef110ae8-2395-4928-9dba-f80b49cab276",
  "filter": "header['x-gitlab-event'] == 'Merge Request Hook' && (body.object_attributes.action == 'open' || body.object_attributes.action == 'reopen') && body.object_attributes.target_branch == 'prod'",
  "secret": {
    "type": "token_matches",
    "value": "hash:SHA3-512:7a4f5b0701a921c9054eb41d97258979389922592801d21d213bdb5aa99cb8aec71368153fe0a084e7c9e5be119976a8279cfa0e794de90036fe138a57e4fd31",
    "source": "header",
    "key_name": "X-Gitlab-Token"
  },
  "properties": [],
  "webhook_url": "https://devops-api.us-south.devops.cloud.ibm.com/v1/tekton-webhook/f6e201e0-ba61-4e88-9a1b-6fc81febb952/run/ef110ae8-2395-4928-9dba-f80b49cab276",
  "max_concurrent_runs": 1,
  "enabled": true,
  "href": "https://api.us-south.devops.cloud.ibm.com/pipeline/v2/tekton_pipelines/f6e201e0-ba61-4e88-9a1b-6fc81febb952/triggers/ef110ae8-2395-4928-9dba-f80b49cab276"
}

Copy the href field from the response and add the private keyword. For example, https://api.private.us-south.devops.cloud.ibm.com/pipeline/v2/tekton_pipelines/f6e201e0-ba61-4e88-9a1b-6fc81febb952/triggers/ef110ae8-2395-4928-9dba-f80b49cab276

Creating inventory repository webhook for CD pipeline

You need to create a repository webhook to invoke the created trigger.

  1. Navigate to the webhooks page of your firewalled source code repository. For example, https://my-private-gitlab/test-admin/compliance-inventory2/-/hooks.
  2. Click the Add new webhook.
  3. Complete the URL field by using the modified href field from the trigger created earlier. For example, https://api.private.us-south.devops.cloud.ibm.com/pipeline/v2/tekton_pipelines/f6e201e0-ba61-4e88-9a1b-6fc81febb952/triggers/ef110ae8-2395-4928-9dba-f80b49cab276.
  4. Choose a meaningful name. For example, Promotion Validation webhook.
  5. Complete the Secret token with the <WEBHOOK_SECRET> value used when creating the pipeline trigger
  6. Select Push events > All branches as the Trigger options
  7. Click Add webhook to persist your changes.

The value for the URL field can be retrieved from the trigger Webhook URL field value through the pipeline UI after the CD pipeline creation. The secret that is used by the webhook and trigger pair can be modified.

Scenario 1: Fully Air Gapped prerequisites

In this scenario, you have self hosted an application repository in a private GitLab instance and all the DevSecOps compliance repositories required to operate the toolchains. For these steps the air gapped GitLab instance is referred to as https://my-private-gitlab/ and the Git user, or owner as test-user

The required repositories can be downloaded from the following URLs. Us-south is the given example but can be replaced with any of the following: us-south, eu-gb, au-syd, eu-de, jp-tok, jp-osa, eu-es, or ca-tor.

The required repositories for a fully air gapped configuration
URL Description
https://us-south.git.cloud.ibm.com/open-toolchain/code-engine-compliance-app The sample Code Engine application repository.
https://us-south.git.cloud.ibm.com/open-toolchain/hello-compliance-app The sample Kubernetes application repository.
https://us-south.git.cloud.ibm.com/open-toolchain/compliance-evidence-locker The compliance evidence repository.
https://us-south.git.cloud.ibm.com/open-toolchain/compliance-incident-issues The compliance issues repository.
https://us-south.git.cloud.ibm.com/open-toolchain/compliance-inventory The compliance inventory repository.
https://us-south.git.cloud.ibm.com/open-toolchain/compliance-change-management The compliance change management repository.
https://us-south.git.cloud.ibm.com/open-toolchain/hello-compliance-deployment The sample repository containing deployment scripts. Not required when using the Code Engine sample that contains all the necessary scripts.
https://us-south.git.cloud.ibm.com/open-toolchain/compliance-pipelines The compliance repository containing all the pipeline definitions.

The repositories or the contents of the repositories need to be mirrored into the air gapped environment. For example, the URL to the sample code application is https://my-private-gitlab/test-user/code-engine-compliance-app

In this example, the repositories are set up as follows for the Code Engine application.

The repositories after being setup in the air gapped private GitLab
URL
https://my-private-gitlab/test-admin/code-engine-compliance-app
https://my-private-gitlab/test-admin/compliance-evidence
https://my-private-gitlab/test-admin/compliance-issues
https://my-private-gitlab/test-admin/compliance-inventory
https://my-private-gitlab/test-admin/compliance-change-management
https://my-private-gitlab/test-admin/compliance-pipelines

A Git personal access token is required to configure the repository tool integrations that are stood up as part of the DevSecOps toolchains and the service API key for the pipeline tool integration.

Scenario 1: Fully Air Gapped configuration

You are now ready to configure the security, required, and optional variables for an air gapped deployment by using the sample Code Engine application.

The repository URLs are specific to this example and the URLs used should reflect your environment.

  1. In the Configure section, select your authentication method. You can use an existing secret in Secrets Manager, add your API key directly, or set a Trusted Profile. For more information, see Using an API key or secret to authorize a project to deploy an architecture.
  2. In the Required tab, enter values for required fields.
  3. There are three region fields:
    • The first declares the region in which all the services get are deployed.
    • The second is for the region in which the Security and Compliance service is deployed.
    • The third is for Event Notifications. By default these are set to us-south. If you change the default, ensure that the three regions are all set the same.
  4. In the app_repo_existing_url set https://my-private-gitlab/test-admin/code-engine-compliance-app.
  5. In the Optional tab, find the following variables and set the values as stated.
    • create_git_token - toggle to true.
    • repo_git_token_secret_name - set a name for your token. This is the name that appears for the Git token secret in Secrets Manager.
    • repo_git_token_secret_value - set the Git personal access token to access the team's repositories.
    • repo_group - set the name of the repositories group, or owner name.
    • repo_apply_settings_to_compliance_repos - true
    • repo_git_provider - gitlab
    • repo_title - set a repository title name. For example development server.
    • repo_root_url - set according to your own private GitLab instance. For example, https://my-private-gitlab
    • repo_blind_connection - true
    • repo_git_id - gitlabcustom
    • evidence_repo_existing_url - https://my-private-gitlab/test-admin/compliance-evidence
    • issues_repo_existing_url - https://my-private-gitlab/test-admin/compliance-issues
    • inventory_repo_existing_url - https://my-private-gitlab/test-admin/compliance-inventory
    • change_management_existing_url - https://my-private-gitlab/test-admin/compliance-change-management
    • compliance_pipeline_existing_repo_url - https://my-private-gitlab/test-admin/compliance-pipelines
    • app_repo_branch - Make sure that the branch name matches the branch of an example. For the Code Engine sample set as main.
    • create_triggers - true
    • create_git_triggers - false
    • compliance_pipeline_repo_use_group_settings - true
    • create_privateworker_secret - true
    • enable_privateworker - true
    • privateworker_credentials_secret_name - set a name that you would like to see the private worker secret appear as in Secrets Manager.
    • privateworker_secret_value - Provide private worker service API key.
  6. Click the Save button.

Scenario 1: (Optional) Fully Air Gapped configuration using a pre-existing resource group

  1. Navigate to the projects level and find the current stack. Expand the stack and locate the Compliance Manager member. Click the Options or Kebab menu.
  2. Select Edit > Optional tab, and find the following variables and set the values as stated.
    • resource_group_name - Provide the name of an existing resource group in which the stack provisions the resources.
    • use_existing_resource_group - true
  3. Click the Save button. Do not validate at this point.
  4. Navigate back up to the Projects level and find the current stack. Expand the stack and find the Compliance Manager member. Click the Options or Kebab menu.
  5. Select Optional tab to view the configuration of the Compliance Manager.
  6. Edit the resource_groups_scope variable, and set the name of the existing resource group in quotes within the square brackets. For example ["my-resource-group"].
  7. Click the Save button.

Scenario 1: Fully Air Gapped running of the stack

  1. Navigate to the top level of the project.
  2. Click the project level Manage > Settings option.
  3. Set Auto-deploy on to deploy all the configurations of the stack without requiring each configuration deployment to start manually.
  4. Click the Configurations tab.
  5. Click the options or Kebab menu of the stack configuration to expand.
  6. Click **Validate and deploy to start the deployment after the validation completes.

Scenario 2: Fully Air Gapped apart from compliance pipelines

In this scenario, you can link to an existing compliance pipelines repository and allows you to always keep up to date with the latest pipeline definitions.

Otherwise, the user must periodically check the state of the repository at https://us-south.git.cloud.ibm.com/open-toolchain/compliance-pipelines and manually copy the changes to the air gapped private GitLab.

  1. Follow the steps as stated in Scenario 1: Fully Air Gapped configuration.
  2. When you get to the step of setting compliance_pipeline_existing_repo_url, leave the value empty.
  3. Find the input compliance_pipeline_repo_use_group_settings and set as false.
  4. Continue the remaining steps of Scenario 1: Fully Air Gapped configuration.

Scenario 3: Hybrid air gapped configuration

In this scenario, only the repositories containing application code and scripts are stored in the private GitLab instance. The remaining compliance repositories use the default IBM Git hosted solution.

  1. Follow the steps as stated in Scenario 1: Fully Air Gapped configuration.

  2. When you get to the step of setting compliance_pipeline_existing_repo_url, leave the value empty.

  3. Find the input compliance_pipeline_repo_use_group_settings and set as false, and leave the following variable inputs as empty.

    • evidence_repo_existing_url
    • issues_repo_existing_url
    • inventory_repo_existing_url
    • change_management_existing_url

  4. Find the input repo_apply_settings_to_compliance_repos and set as false.

  5. Continue the remaining steps of Scenario 1: Fully Air Gapped configuration.

    Configuration of the Code Engine compliance app scenario 1
    Name Value
    app_repo_branch main
    app_repo_existing_url https://my-private-gitlab/test-admin/code-engine-app
    use_app_repo_for_cd_deploy false
    cd_deployment_repo_existing_url https://my-private-gitlab/test-admin/hello-compliance-deployment
    create_git_token true
    create_triggers true
    create_git_triggers false
    compliance_pipeline_repo_use_group_settings true
    add_pipeline_definitions true
    create_privateworker_secret true
    enable_privateworker true
    privateworker_credentials_secret_name private-worker-key
    privateworker_secret_value <Service-API-Key>
    repo_apply_settings_to_compliance_repos false
    repo_title My Git Enterprise Server
    repo_root_url https://my-private-gitlab/
    repo_blind_connection true
    repo_git_id gitlabcustom
    repo_group test-user
    repo_git_provider gitlab
    repo_git_token_secret_name gitlab-token
    repo_git_token_secret_value <Access-Token>

Following are the important tips to configure the parameter in the application

  • If you are using an application repository that contains the deployment scripts such as the Code Engine then you can set use_app_repo_for_cd_deploy to true and leave cd_deployment_repo_existing_url unset.

  • If you already have an existing Secrets Manager instance that you need, then use existing_secrets_manager_crn.