Publishing test results
Test results inform DevOps Insights about the tests that are run during the build process. These tests must generate a JSON or XML-based file. The generated test file is published as a test record. Along with the test file, this record contains the application name, build number, types of test, and other fields.
Before you begin
Uploading test results
The application name and build ID in the test record must match the ones that are used in the build record for a specific build. The following example script uploads functional verification test (FVT) results to DevOps Insights:
# Run tests and generate a test results file here.
#install the DevOps Insights plugin
ibmcloud plugin install -f doi
# Login to IBMCloud if you are not already logged in. Assumes that $API_KEY environment variable has been set as a secured property
ibmcloud login --apikey $API_KEY --no-region
ibmcloud doi testrecord-publish --logicalappname="$MY_APP_NAME" --buildnumber="$MY_BUILD_NUMBER" --filelocation fvttest.json --type fvt
In the example script, testrecord-publish specifies that the command uploads test records. The --filelocation flag indicates the location of the test results file relative to the root directory of the job. The --type flag indicates the type of test result.
The following example script runs tests and then uploads Mocha results to DevOps Insights:
# Run tests and generate a test results file here.
istanbul cover --report lcov --report json-summary _mocha -- -R xunit -O output=./test/results/mocha.xml 'test/**/*.js'
#install the DevOps Insights plugin
ibmcloud plugin install -f doi
# Login to IBMCloud if you are not already logged in. Assumes that $API_KEY environment variable has been set as a secured property
ibmcloud login --apikey $API_KEY --no-region
#Publish test results. Assumes that MY_APP_NAME and MY_BUILD_NUMBER environment variables are already set
ibmcloud doi testrecord-publish --logicalappname="$MY_APP_NAME" --buildnumber="$MY_BUILD_NUMBER" --filelocation /test/results/mocha.xml --type unittest
ibmcloud doi testrecord-publish --logicalappname="$MY_APP_NAME" --buildnumber="$MY_BUILD_NUMBER" --filelocation ./coverage/coverage-summary.json --type code
The command supports the following type values:
| Type | Description |
|---|---|
unittest |
Unit test results |
fvt |
Functional verification test results |
code |
Code coverage results |
sonarqube |
SonarQube scan results |
staticsecurityscan |
Static security scan results from IBM Application Security on Cloud |
dynamicsecurityscan |
Dynamic security scan results from IBM Application Security on Cloud |
vulnerabilityadvisor |
Vulnerability Advisor results from IBM Vulnerability Advisor on Cloud |
Test records must provide data in one of the following supported formats:
| Test Type | Supported Formats |
|---|---|
| Unit test | Mocha, xUnit, Karma/Mocha |
| Functional verification test | Mocha, xUnit, Karma/Mocha |
| Code coverage | Cobertura, lcov, JaCoCo |
| SonarQube | Scan data that is provided by SonarQube scans |
| Static AppScan | Static App Scans that are provided by IBM Application Security on Cloud |
| Dynamic AppScan | Dynamic App Scans that are provided by IBM Application Security on Cloud |
| Vulnerability Advisor Results | Vulnerability Advisor results from IBM Vulnerability Advisor on Cloud |
Viewing test results
When your pipeline runs, it publishes the test result data to DevOps Insights. You can view the test result data on the Quality Dashboard page.
- From the IBM Cloud console, click the menu icon
, and select Resource List. - Select your toolchain.
- From your toolchain's Overview page, on the IBM Cloud tools card, click DevOps Insights.
- Click Quality Dashboard in the navigation to open the page.
For more information about the Quality Dashboard page, see DevOps data aggregation.