Validating evidence

Understanding evidence declaration levels

1. Recommended mode:

When compliance checks are marked as "recommended," they serve as warnings rather than blockers. This means:

• The pipeline execution continues even when checks fail
• The pipeline provides visibility into compliance status but doesn't enforce automatic failure
• You are responsible for investigating and resolving all flagged issues
• All failing or missing compliance checks must be addressed and fixed during your CI process

This approach emphasizes a shared responsibility model: the pipeline provides compliance visibility and recommendations, but teams must actively ensure their CI processes generate required evidence and address any gaps before production deployment.

2. Required mode:

Evidence must be present in success state. Deployment is blocked if evidence is missing, pending, or failed.

3. Required if present mode:

Evidence must be in success state if it exists. This mode:

• Fails only if evidence status is failure
• Passes if evidence is success, pending, or missing
• Useful for optional scans that must pass when they run

Evidence aggregation

Evidence aggregation is the process of collecting and evaluating compliance evidence across multiple service environments (dev, stage, prod) to determine overall application-level compliance. During CI/CD pipeline runs, various compliance tools collect evidence for different asset types (images, commits, artifacts). Evidence is tracked separately for each service environment, and the system aggregates results based on rules defined in your configuration file. Overall compliance status is determined by evaluating all collected evidence against configured requirements.

Key benefits:

• Environment-Specific Validation: Different environments can have different evidence requirements (e.g., stage may require fewer checks than prod)
• Flexible Configuration: Customize which evidence types are required, recommended, or conditionally required for each environment
• Promotion Gating: Block deployments to production if required evidence is missing or failed in previous environments
• Audit Trail: Maintain complete evidence history across all environments for compliance reporting

The following diagram illustrates how evidence flows through environments and how aggregation determines deployment readiness:

Config file Sections

Pre-deployment checks:

• Validate evidence produced by the CI pipeline
• Gate deployment based on these checks
• Auto-approve change request if all checks pass
• Block deployment if checks fail (except for emergency change requests)
• Add Config file as part of the change request

Post-deployment checks:

• Validate evidence produced by the CD pipeline
• Evaluate the pipeline based on these checks

Configuration file rule structure

The configuration file uses the following JSON structure to define validation rules. Each evidence type can have multiple rules based on asset types and environments:

Structure:

• Evidence type ID - Example: com.ibm.static-scan
  • rules
  • Asset type - Examples: image, commit, artifact
  • source_environments - Where evidence originates
    • name - Environment name (e.g., ci-pipeline, stage, prod)
    • region - Geographic region (optional)
  • target_environments - Where evidence is validated
    • name - Environment name (e.g., stage, prod)
    • region - Geographic region (optional)
  • required - Evidence must be present in success state; blocks deployment if missing, pending, or failed
    • Tool - Tool type for evidence collection (e.g., SonarQube, owasp-zap, *)
  • required_if_present - Evidence must be in success state if it exists; fails only on failure status; passes if success, pending, or missing
    • Tool - Tool type for evidence collection (e.g., SonarQube, owasp-zap, *)
  • recommended - Pipeline logs warnings if evidence is missing/pending/failed; does not block deployment
    • Tool - Tool type for evidence collection (e.g., SonarQube, owasp-zap, *)

Evidence collected in the CI pipeline should have its origin field set to ci-pipeline in the configuration file.

For detailed explanations of evidence declaration levels (required, required_if_present, recommended), see the Config file mode section.

Evidence validation using config file

To implement configuration file mode in your toolchain, follow these configuration steps:

Step 1: Enable evidence validation

In your CD and CC pipeline environment properties, set the following variable to enable the evidence validation feature in your deployment pipelines:

opt-in-evidence-checks=1

Step 2: Configure the validation file path

Set the path to your configuration file in the pipeline-config-repo:

evidence-checks-config-path=validation.json

If evidence-checks-config-path is not defined, the system automatically searches for a file with the name <region>.<target>.validation.json, <target>.validation.json, or validation.json in the pipeline-config-repo.

Step 3: Verify configuration

Once the pipeline is running, verify that evidence validation based on the config file is enabled by following these steps:

1. Access Pipeline - Navigate to the specific CD pipeline run associated with your change
2. Locate Task - Find the task labeled prod-change-request and select the change-request step
3. Review Tables - Locate two critical tables by searching for these keywords:
   • Asset Scope Table: Search for Current pipeline scope (first table)
   • Evidence Check Table: Search for PRE-DEPLOYMENT (second table)

If you can see these tables, it confirms that evidence validation based on the config file is enabled.

Note: Pipeline evaluation will also occur when the check is enabled in both the CD and CC pipelines, utilizing the configuration file.

Various deployment topologies

The following examples demonstrate how evidence validation works across different deployment topologies. Each topology shows how evidence collected in one environment is validated when promoting to subsequent environments.

Service environments:

• master or dev - Development
• stage - Pre-production (recently supported)
• prod - Production

{
  "evidence_type_id": "com.ibm.acceptance_tests",
  "rules": [
    {
      "asset_type": "image",
      "source_environments": [
        {
          "name": "ci-pipeline"
        }
      ],
      "target_environments": [
        {
          "name": "prod"
        }
      ],
      "required": [],
      "required_if_present": [
        {
          "name": "tool",
          "values": [
            "*"
          ],
          "description": "Acceptance tests must pass if they are run"
        }
      ],
      "recommended": []
    }
  ]
}

{
  "evidence_type_id": "com.ibm.acceptance_tests",
  "rules": [
    {
      "asset_type": "image",
      "source_environments": [
        {
          "name": "ci-pipeline"
        },
        {
          "name": "stage"
        }
      ],
      "target_environments": [
        {
          "name": "stage"
        },
        {
          "name": "prod"
        }
      ],
      "required": [],
      "required_if_present": [
        {
          "name": "tool",
          "values": [
            "*"
          ],
          "description": "Acceptance tests must pass if they are run"
        }
      ],
      "recommended": []
    }
  ]
}

{
  "evidence_type_id": "com.ibm.acceptance_tests",
  "rules": [
    {
      "asset_type": "image",
      "source_environments": [
        {
          "name": "ci-pipeline"
        },
        {
          "name": "stage",
          "region": "us-south"
        }
      ],
      "target_environments": [
        {
          "name": "stage",
          "region": "us-south"
        },
        {
          "name": "stage",
          "region": "us-east"
        },
        {
          "name": "prod",
          "region": "us-south"
        }
      ],
      "required": [],
      "required_if_present": [
        {
          "name": "tool",
          "values": [
            "*"
          ],
          "description": "Acceptance tests must pass if they are run"
        }
      ],
      "recommended": []
    }
  ]
}

Default tool values for DevSecOps stages

Different asset types (such as container images, source code commits, or deployment artifacts) require different types of evidence. The configuration file allows you to specify which evidence types must be collected for each asset type, and the system validates that the appropriate evidence has been gathered by the corresponding tools.

| com.ibm.cloud.slsa | tekton-chains | Artifact (helm, image or chart) |

My deployment is blocked - how do I find out why?

If your deployment is blocked, follow these steps to identify the issue:

1. Access Pipeline - Navigate to the specific CD pipeline run associated with your change
2. Locate Task - Find the task labeled prod-change-request and select the change-request step
3. Review Tables - Locate two critical tables by searching for these keywords:
   • Asset Scope Table: Search for Current pipeline scope (first table)
   • Evidence Check Table: Search for PRE-DEPLOYMENT (second table)

The Evidence Check Table shows which evidence items are missing or failed, causing your deployment to be blocked.

Understanding the Evidence Check Table

The Evidence Check Table displays the validation status of evidence for each asset. Understanding its structure helps you troubleshoot compliance issues.

Asset Table Structure:

• ID Column: Primary key identifying each asset
• Full_id: Complete asset id.
• type: Type of the asset
• URI Column: The actual asset (image, artifact, etc.) being validated
• Relationship: Asset ID serves as the foreign key (AssetId) in the Evidence Check Table

Evidence Check Table Structure:

• AssetId Column: foreign key identifying each asset
• EvidenceTypeId: Specifies which compliance evidence is required (e.g., vulnerability scan, unit tests, peer review)
• Evidence Origin: Indicates where evidence should come from (typically ci-pipeline)
• Status: Shows whether evidence is present, missing, or failed for each asset

Evidence Summary Table Structure:

This table displays the compliance evidence summary for all assets and shows how many evidence items are missing, successful, failed, or pending.

How do I fix missing or failed evidence?

Once you've identified the missing or failed evidence from the Evidence Check Table, use this decision tree to resolve the issue. Note the specific evidence type (e.g., com.ibm.static_scan) and asset type (e.g., image, commit).

Question: "Have I configured my CI pipeline to collect [evidence type] for [asset type]?"

If NO (you haven't collected it):

• Ask: "Is this a valid compliance requirement for my service?"
  • YES, it's valid: Implement evidence collection in your CI pipeline
  • Add necessary scanning tools or scripts
  • Configure proper evidence reporting mechanisms
  • Ensure evidence is linked to the correct asset
  • See the Asset Evidence Collection Guide for implementation details
  • NO, it's not applicable: Document justification and escalate
  • Contact your security focal point
  • Explain why this check doesn't apply to your service
  • Provide clear justification
  • Remove the check from your configuration file after approval

If YES (you have collected this evidence previously):

• Investigate why it's showing as missing in the Evidence Check Table
• Potential issues to check:
  • Is evidence being generated correctly in your CI pipeline?
  • Is it being reported to the correct location?
  • Is the asset ID properly linked to the evidence?
  • Are there timing or synchronization issues between CI and CD pipelines?
  • Is the evidence format compatible with the validation requirements?
  • Check pipeline logs for error messages related to evidence collection or reporting

Sample configuration file

{
  "version": "2.0",
  "pre-deployment": [
    {
      "evidence_type_id": "com.ibm.prod_change_request",
      "rules": [
        {
          "asset_type": "image",
          "source_environments": [
            {
              "name": "stage"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Change request must be successful if present"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.acceptance_tests",
      "rules": [
        {
          "asset_type": "commit",
          "source_environments": [
            {
              "name": "ci-pipeline"
            },
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Acceptance tests must pass if they are run"
            }
          ],
          "recommended": []
        },
        {
          "asset_type": "image",
          "source_environments": [
            {
              "name": "ci-pipeline"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Acceptance tests must pass if they are run"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.branch_protection",
      "rules": [
        {
          "asset_type": "commit",
          "source_environments": [
            {
              "name": "ci-pipeline"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Branch protection must be successful if present"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.cloud.slsa",
      "rules": [
        {
          "asset_type": "image",
          "source_environments": [
            {
              "name": "ci-pipeline"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "SLSA attestation must be successful if present"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.dynamic_scan",
      "rules": [
        {
          "asset_type": "image",
          "source_environments": [
            {
              "name": "ci-pipeline"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "owasp-zap"
              ],
              "description": "Dynamic scan must pass if OWASP ZAP is run"
            }
          ],
          "recommended": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Other dynamic scan tools are recommended"
            }
          ]
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.cloud.verify_signature",
      "rules": [
        {
          "asset_type": "image",
          "source_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Signature verification must pass if present"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.cloud.image_vulnerability_scan",
      "rules": [
        {
          "asset_type": "image",
          "source_environments": [
            {
              "name": "ci-pipeline"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Image vulnerability scan must pass if run"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.peer_review",
      "rules": [
        {
          "asset_type": "commit",
          "source_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Peer review must be successful if present"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.unit_tests",
      "rules": [
        {
          "asset_type": "commit",
          "source_environments": [
            {
              "name": "ci-pipeline"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Unit tests must pass if they are run"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.static_scan",
      "rules": [
        {
          "asset_type": "commit",
          "source_environments": [
            {
              "name": "ci-pipeline"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Static scan must pass if run"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.detect_secrets",
      "rules": [
        {
          "asset_type": "commit",
          "source_environments": [
            {
              "name": "ci-pipeline"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Detect secrets scan must pass if run"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.code_vulnerability_scan",
      "rules": [
        {
          "asset_type": "commit",
          "source_environments": [
            {
              "name": "ci-pipeline"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Code vulnerability scan must pass if run"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.code_cis_check",
      "rules": [
        {
          "asset_type": "commit",
          "source_environments": [
            {
              "name": "ci-pipeline"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "CIS check must pass if run"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.code_bom_check",
      "rules": [
        {
          "asset_type": "commit",
          "source_environments": [
            {
              "name": "ci-pipeline"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "BOM check must pass if run"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.network_compliance",
      "rules": [
        {
          "asset_type": "commit",
          "source_environments": [
            {
              "name": "ci-pipeline"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Network compliance check must pass if run"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.pipeline_run_data",
      "rules": [
        {
          "asset_type": "generic",
          "source_environments": [
            {
              "name": "ci-pipeline"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Pipeline run data must be successful if present"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.pipeline_logs",
      "rules": [
        {
          "asset_type": "generic",
          "source_environments": [
            {
              "name": "ci-pipeline"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Pipeline logs must be successful if present"
            }
          ],
          "recommended": []
        }
      ]
    }
  ],
  "post-deployment": [
    {
      "evidence_type_id": "com.ibm.prod_change_request",
      "rules": [
        {
          "asset_type": "image",
          "source_environments": [
            {
              "name": "stage"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Change request must be successful if present"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.acceptance_tests",
      "rules": [
        {
          "asset_type": "commit",
          "source_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Acceptance tests must pass if they are run"
            }
          ],
          "recommended": []
        },
        {
          "asset_type": "image",
          "source_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Acceptance tests must pass if they are run"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.pipeline_run_data",
      "rules": [
        {
          "asset_type": "generic",
          "source_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Pipeline run data must be successful if present"
            }
          ],
          "recommended": []
        }
      ]
    },
    {
      "evidence_type_id": "com.ibm.pipeline_logs",
      "rules": [
        {
          "asset_type": "generic",
          "source_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "target_environments": [
            {
              "name": "stage"
            },
            {
              "name": "prod"
            }
          ],
          "required": [],
          "required_if_present": [
            {
              "name": "tool",
              "values": [
                "*"
              ],
              "description": "Pipeline logs must be successful if present"
            }
          ],
          "recommended": []
        }
      ]
    }
  ]
}