Managing incident issues

Incident issue processing in PR pipeline

PR pipeline with evidence collection enabled can create incident issues when any vulnerabilty or CVE is encountered. To enable evidence collection in PR pipeline refer to:PR pipeline evidence collection

The issue management in PR pipeline for incident issues is similar to that of CI pipeline. The only difference lies in the autoclosing logic.

An incident issue created by PR pipeline can be autoclosed only by a PR pipeline running on the same PR. The issue can't be autoclosed by a PR pipeline when:

• The issue is found by a PR pipeline running on any other PR.
• The issue is found by any CI or CC pipelines.

Incident issue processing

Even though the CI and CC pipelines have common steps, the issue processing of these pipelines has some differences:

• Incident issues that are created during the CI pipeline do not carry a due date, while incident issues that are created during the CC pipeline do.
• Incident issues that are created during the CI pipeline are found during the build, while incident issues that are created during the CC pipeline are found in the production environment.

Figures 1 - 6 show the possible use cases that are based on these differences.

Issue lifecycle

The issue lifecycle extends from code PRs to production scans.

Use case 1: vulnerability found in the build

The new build introduces a vulnerability, which is not accepted. Deployment is blocked unless the change request is an emergency CR that is manually approved.

This build pipeline flow incase of a vulnerability found, and the corresponding user actions has been explained below.

Use case 2: vulnerability found in the build that is also in production

The new build contains a vulnerability that is also in the currently deployed production. Teams have a timeline to fix the issue, but new features or fixes can still be deployed.

CC pipeline sets the timeline to fix any such vulnerabilities found in production. It will only fail when the timeline to fix the vulnerability has expired. The flow has been outlined below.

Use case 2A: vulnerability that is found in production allows PRs

The vulnerability in production does not prevent PRs from merging.

Use case 3: false positives

If the team categorizes an issue as false positive, the issue can be labeled as Exempted. The issue can then be handled as a nonblocking issue. To maintain an audit trail, change requests keep the issues visible.

Use case 4: automatically closing fixed issues

Periodically running the CC pipeline can close issues that are open and have a due date set. Also, the relevant vulnerability cannot be found in scans.

The following diagram explains the flowchart of CC pipeline detecting which issues to close, and which issues to create fresh.

Setting the due date for incident issues

If the incident issues are found in production, the Due Date property might be added to the issue to specify the grace period in which it must be fixed. The duration of the grace period is determined by the severity of the found vulnerability.

For more information about customizing the grace periods, see Configuring custom grace periods on the CC pipeline.

Labeling incident issues

Incident issues that are created by the continuous integration (CI) or continuous compliance (CC) pipelines can have default labels.

Incident issues with due date

When you use the collect-evidence script, incident issues are created and attached to the collected evidence. If issues are found in production, they can have a specified time period in which they must be fixed so that deployments are not blocked. The time frame that is given to fix the problem in production is called the grace period. However, for better readability, Due Date is now available in incident issues so that users know the date when the fix is due without calculating it from the grace period.

If an issue such as vulnerability or CVE is found in production, and the same problem is also found in a build, the build does not make the situation worse. The feature can be deployed, and the team can focus on fixing the issue in production.

Configuring custom grace periods on the CC pipeline

The Continuous Compliance (CC) pipeline calculates the due dates of incident issues based on the severity of an issue. You can change the default grace period values and replace them with custom values.

The pipeline calculates the grace period according to the following table:

To change the default configuration, create a new property in the CC pipeline's environment properties named grace-period-configuration. This environment property must be a JSON string and match the following format:

{
  "informational": 50,
  "low": 40,
  "medium": 30,
  "high": 20,
  "critical": 10
}

If the environment property does not match the expected format or is not a valid JSON string, the pipeline uses the default values.

The environment property grace-period-configuration sets due dates for issues that do not have a due date set already. For issues that have a due date set, reconfiguring the grace-period-configuration does not update those due dates.

<due date> = <date of finding the issue in prod> + <grace period in days (determined by severity)>

Postponing the due date of an incident issue

If you want to postpone the due date of an incident issue, you can ask for a review from a security focal. Depending on the review, you can postpone the due date by modifying the Due date field in the Git Repos and Issue Tracking incident issues meta fields.

Be sure to reference the security-focal review in the issue, such as providing a link to it in a comment.

Slack alerts for pending and overdue issues for CC pipeline

The Continuous Compliance (CC) pipeline can set due dates for the incident issues. The pipeline can also notify users about issues that have approaching due dates and overdue due dates by using Slack, if Slack integration is enabled.

For more information, see Set up a Continuous Integration (CI) toolchain. For more information about due dates, see [Incident issues with due date](/docs/devsecops?topic=devsecops-incident-issues#devsecops-devsecops-issues-due-date.

The issues that are notified are classified as follows:

• Issues having pending due dates within a specific timeframe: issues that are open, have due dates, and are due within a timeframe.
• Issues having past due dates: issues that are open, have due dates, and the date is passed.

The timeframe durations are overdue, due in 1 day, due in 2 days, due in 5 days, and due in 10 days.

An example of this capability is as follows:

Overdue issues:
- <issue url#1>
- <issue url#2>
- <issue url#3>

Issues due in 1 day:
- <issue url#4>

Issues due in 2 days:
- <issue url#5>

Issues due in 5 days:
- <issue url#6>
- <issue url#7>

Issues due in 10 days
- <issue url#8>
- <issue url#9>
- <issue url#10>
- <issue url#11>
- <issue url#12>

The aggregated list is ordered with the issues that are overdue first, and the issues with nearest due dates are listed before those with a later due date.

The cc-finish stage in the CC pipeline queries for the issues based on the criteria and triggers a Slack alert.