Introduction to IBM Knowledge Catalog as a Service
Note: this documentation page is under construction. Refer to Watson Data API documentation page for full description of APIs provided by IBM Knowledge Catalog.
You can use the IBM Knowledge Catalog as a Service APIs to establish business vocabulary, import and enrich data assets, analyze data quality, define data protection rules, and more.
IBM Knowledge Catalog as a Service builds on top of the common core functionality for Data and AI SaaS products. The common core functionality provides fundamental features, such as managing catalogs, projects, assets and connections. You can view common core functionality APIs here.
If you are looking for the IBM Knowledge Catalog software APIs, see here.
For more information, see these resources:
Authentication
Before you can call a common core API you must first create an IAM bearer token. Each token is valid only for one hour, and after a token expires you must create a new one if you want to continue using the API. The recommended method to retrieve a token programmatically is to create an API key for your IBM Cloud identity and then use the IAM token API to exchange that key for a token.
You can create a token in IBM Cloud or by using the IBM Cloud command line interface (CLI).
To create a token in the IBM Cloud:
- Log in to IBM Cloud and select Manage > Access (IAM) > API keys.
- Create an API key for your own personal identity, copy the key value, and save it in a secure place. After you leave the page, you will no longer be able to access this value.
- With your API key, set up a REST API tool and run the following command to the right
You can read more about managing API keys at Understanding API keys documentation page.
- Use the value of the
access_tokenproperty for your API calls. Set theaccess_tokenvalue as the authorization header parameter for requests to the APIs. The format isAuthorization: Bearer <access_token_value_here>. For example:Authorization: Bearer eyJhbGciOiJIUz......sgrKIi8hdFs
To create a token by using the IBM Cloud CLI:
- Follow the steps to install the CLI, log in to IBM Cloud, and get the token
described in Getting an IAM API token programmatically.
Remove
Bearerfrom the returned IAM token value in your API calls.
Curl command with API key to retrieve token
curl -X POST 'https://iam.cloud.ibm.com/identity/token' -H 'Content-Type: application/x-www-form-urlencoded' -d 'grant_type=urn:ibm:params:oauth:grant-type:apikey&apikey=MY_APIKEY'
Response
{
"access_token": "eyJhbGciOiJIUz......sgrKIi8hdFs",
"refresh_token": "SPrXw5tBE3......KBQ+luWQVY=",
"token_type": "Bearer",
"expires_in": 3600,
"expiration": 1473188353
}
Error handling
Responses with 400-series or 500-series status codes are returned when a request cannot be completed. The body of these responses follows the error model, which contains a code field to identify the problem and a message field to explain how to solve the problem. Each individual endpoint has specific error messages. All responses with 500 or 503 status codes are logged and treated as a critical failure that requires an emergency fix.
Rate Limiting
The following rate limiting headers are supported by some of the common core service APIs:
- X-RateLimit-Limit: If rate limiting is active, this header indicates the number of requests permitted per hour;
- X-RateLimit-Remaining: If rate limiting is active, this header indicates the number of requests remaining in the current rate limit window;
- X-RateLimit-Reset: If rate limiting is active, this header indicates the time at which the current rate limit window resets, as a UNIX timestamp.
Methods
List Hummingbird tasks
Status: Complete
Get the list of Hummingbird tasks for a specified accout or type, provided the caller has the necessary rights to do so.
GET /v1/hb_tasks
Request
Query Parameters
The BSS Account ID to use.
The type of the job.
Fetch complete details of job.
Default:
false
Response
Collection of HB tasks.
The ID of the task
The ID of the account
Indicates if account plan is free plan
Type of the job
The job payload
The id to track complete flow
Priority of the HB job
The ID of the HB job
Timestamp of task creation
Timestamp of job submission
Timestamp of job completion
The duration for which job was run
The number of times the job was retried
Status of the HB job
Possible values: [
REQUESTED,SUBMITTED,RUNNING,COMPLETED,ERROR,STOPPED,STOPPED_CAPACITY_UNIT_EXCEEDED,QUEUED]The ID of the project where the spark job logs need to be saved. If this value is not provided or is invalid then the spark job logs will be stored in default COS bucket.
A RuntimeRegistrationParameters holds details to be sent to runtimes api to report CUH for HB job.
A ComputeUnits holds details to determine if a particular account has surpassed its CUH.
Status Code
Success.
You are not permitted to perform this action.
You are not authorized to list the available Hummingbird tasks.
Not Found.
An error occurred. The HB tasks cannot be listed.
No Sample Response
Request
The Hummingbird task to create.
The ID of the account
Indicates if account plan is free plan
Type of the job
The job payload
The id to track complete flow
Status of the HB job
Allowable values: [
REQUESTED,QUEUED]Default:
REQUESTEDPriority of the HB job
Default:
1The ID of the project where the spark job logs need to be saved. If this value is not provided or is invalid then the spark job logs will be stored in default COS bucket.
A RuntimeRegistrationParameters holds details to be sent to runtimes api to report CUH for HB job.
A ComputeUnits holds details to determine if a particular account has surpassed its CUH.
Response
An HbTaskResponse holds details about Hummingbird job and the status of task in queue.
The ID of the task
The ID of the account
Indicates if account plan is free plan
Type of the job
The job payload
The id to track complete flow
Priority of the HB job
The ID of the HB job
Timestamp of task creation
Timestamp of job submission
Timestamp of job completion
The duration for which job was run
The number of times the job was retried
Status of the HB job
Possible values: [
REQUESTED,SUBMITTED,RUNNING,COMPLETED,ERROR,STOPPED,STOPPED_CAPACITY_UNIT_EXCEEDED,QUEUED]The ID of the project where the spark job logs need to be saved. If this value is not provided or is invalid then the spark job logs will be stored in default COS bucket.
A RuntimeRegistrationParameters holds details to be sent to runtimes api to report CUH for HB job.
A ComputeUnits holds details to determine if a particular account has surpassed its CUH.
Status Code
Success.
You are not authorized to create a Hummingbird task.
You are not permitted to perform this action.
The data profile cannot be found.
An error occurred. The Hummingbird task cannot be created.
No Sample Response
Stop Hummingbird tasks
Status: Complete
Stops given Hummingbird task ids. If the Hummingbird task is already completed or is not found then it will be ignored. If the task is locked by other process then it will be ignored as well.
POST /v1/hb_tasks/stop
Response
A StopHbTasksResponse holds details about stop Hummingbird tasks response.
The array of hb_task_id that were already completed. These Hummingbird tasks will be ignored.
The array of hb_task_id that were not found. These Hummingbird tasks will be ignored.
The array of hb_task_id that were actually stopped
The array of hb_task_id that were locked by other process.
Status Code
Success.
You are not authorized to cancel Hummingbird tasks.
You are not permitted to perform this action.
An error occurred. The Hummingbird tasks cannot be stopped.
No Sample Response
Delete Hummingbird tasks
Status: Complete
Delete given Hummingbird task ids. If a Hummingbird task is not found then it will be ignored. If the Hummingbird task is running and stop_in_progress_job is not true then it will be ignored. If the task is locked by other process then it will be ignored as well.
POST /v1/hb_tasks/delete
Request
Query Parameters
Stop running job.
Default:
false
The Hummingbird task ids to delete.
The array of hb_task_id to use during bulk processing
Response
A StopHbTasksResponse holds details about stop Hummingbird tasks response.
The array of hb_task_id that were deleted.
The array of hb_task_id that were not found. These Hummingbird tasks will be ignored.
The array of hb_task_id that are running
The array of hb_task_id that were locked by other process.
Status Code
Success.
You are not authorized to cancel a Hummingbird task.
You are not permitted to perform this action.
An error occurred. The Hummingbird tasks cannot be deleted.
No Sample Response
Response
An HbTaskResponse holds details about Hummingbird job and the status of task in queue.
The ID of the task
The ID of the account
Indicates if account plan is free plan
Type of the job
The job payload
The id to track complete flow
Priority of the HB job
The ID of the HB job
Timestamp of task creation
Timestamp of job submission
Timestamp of job completion
The duration for which job was run
The number of times the job was retried
Status of the HB job
Possible values: [
REQUESTED,SUBMITTED,RUNNING,COMPLETED,ERROR,STOPPED,STOPPED_CAPACITY_UNIT_EXCEEDED,QUEUED]The ID of the project where the spark job logs need to be saved. If this value is not provided or is invalid then the spark job logs will be stored in default COS bucket.
A RuntimeRegistrationParameters holds details to be sent to runtimes api to report CUH for HB job.
A ComputeUnits holds details to determine if a particular account has surpassed its CUH.
Status Code
Success.
You are not authorized to get the data profile you specified.
You are not permitted to perform this action.
The HB task you specified cannot be found.
An error occurred. The HB task you specified cannot be fetched.
No Sample Response
Delete Hummingbird task
Status: Complete
Deletes a specified Hummingbird task.
This api will throw an exception if stop_in_progress_job parameter is not true and if job is running.
DELETE /v1/hb_tasks/{hb_task_id}Get Hummingbird task logs
Status: Complete
Get the spark job logs for Hummingbird task. In order to retrieve the job logs, the HB spark job must be successfully submitted. This will generate a job_id. When successful, this api would return the Hummingbird spark job log.
GET /v1/hb_tasks/{hb_task_id}/logsList data profiles
Status: Complete
Get the list of data profiles in the specified project or catalog for a given dataAsset, provided the caller has the necessary rights to do so. The returned results can be filtered by using one or more of the listed parameters.
| Field | Match type | Example | | --------------- | ------------ | ------------------------------ | | dataset_id | Equals | ?dataset_id=5210c7d-cf6b |
GET /v2/data_profiles
Request
Query Parameters
The ID of the data set to use.
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
Whether to include the entity component. If set to false, only the metadata component is populated in the response.
Default:
trueInclude minimum payload for profiling results. If set to true, only minimum required details for DPS are available in the response.
Default:
falseReturn the analysis results for specified column. This option will be ignored for data profile run using data flow.
Create data profile
Status: Complete
Creates a data profile for a given data set in the specified project or catalog, provided the caller has the necessary rights to do so. Subsequent calls to use the data profile must specify the relevant project or catalog ID that the data profile was created in.
The request payload must include the 'metadata' section containing the data set id and catalog/project id.
The request payload can have the 'entity' section which is optional specifying the data profile options. If these options are not specified, default options are taken. To create several data profiles at once in the specified project or catalog, specify an array of data set in the attribute dataset_ids instead of a single data set in the attribute dataset_id. In this case, the data profiles are created in a single job that can be found in the execution section of the returned data profile entity.
AMQP 1.0 Messages
When a new data profile is created, a message is fired with a state of the new data profile in the body.
Topic
v2.data_profiles.:guid.POST ,where the ":guid" represents the profile_id of the created Data Profile
Subscribe to it by using "v2.data_profiles.*.POST" Binding Key.
Example Message
Topic: v2.data_profiles.5210c7d-cf6b-4204-95d2-95d84ecbe382.POST
Message:
{
"event": "CREATE_DATA_PROFILE",
"actor": {
"user_name": "john@acme.com"
},
"published": "2015-05-10T15:04:15Z",
"url": { /* The href of the DataProfile created */}
"status_code": { /* The Http Status code , 201 if the DataProfile is created successfully */}
"state": { /* the data profile object equivalent to the one obtained by GET /v2/data_profiles/{profile_id} API */ }
"details": {
"catalog_id": "f3c59258-abdd-4e24-828b-0495ec519339",
"dataset_id": "e522db21-59e8-44ab-81b2-bb40c3030a6f",
"profile_id": "5165d439-96f0-40d4-90b2-93795deab61b",
"is_governed": false { /* set to true if the catalog is_governed else set it to false}
}
}
POST /v2/data_profiles
Request
Query Parameters
Whether to start the profiling service immediately after the data profile is created.
Whether to use Metadata enrichment for profiling. If set to true, Metadata enrichment features are eanbled (will use new Lite services).
Default:
falseWhether to charge the bulk profiling to include term assignment charges.
Default:
false
The data profile to create.
Profiling specific metadata
Profiling specific metadata
Information about the number of records
The underlying asset entity.
- entity
The data profile definition.
The data profile definition.
Column metadata inferred by analysis.
The location url where the data profile is created.
Response
A data profile holds the data profiling controls, options and results of a data set.
Profiling specific metadata
Profiling specific metadata
Information about the number of records
The underlying asset entity.
- entity
The data profile definition.
The data profile definition.
Column metadata inferred by analysis.
The location url where the data profile is created.
Status Code
Success.
Accepted.
You are not authorized to create a data profile.
You are not permitted to perform this action.
The data profile cannot be found.
An error occurred. The data profile cannot be created.
No Sample Response
Get data profile
Status: Complete
Gets a data profile from a project or catalog, provided the call has the required right to do so.
GET /v2/data_profiles/{profile_id}Request
Path Parameters
The profile ID to use.
Query Parameters
The ID of the data set to use.
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
Include minimum payload for profiling results. If set to true, only minimum required details for DPS are available in the response.
Default:
false
Response
A data profile holds the data profiling controls, options and results of a data set.
Profiling specific metadata
Profiling specific metadata
Information about the number of records
The underlying asset entity.
- entity
The data profile definition.
The data profile definition.
Column metadata inferred by analysis.
The location url where the data profile is created.
Status Code
Success.
You are not authorized to get the data profile you specified.
You are not permitted to perform this action.
The data profile you specified cannot be found.
An error occurred. The data profile you specified cannot be fetched.
No Sample Response
Delete data profile
Status: Complete
Deletes a specified data profile in a project or catalog, provided the caller has the necessary rights to do so.
The data profile is not deleted if the profiling process is still running unless the stop_in_progress_runs parameter is set to true.
DELETE /v2/data_profiles/{profile_id}Request
Path Parameters
The profile ID to use.
Query Parameters
The ID of the data set to use.
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
Whether to stop all running profiling processes. Data profiles that are running must be stopped before they can be deleted.
Default:
falseWhether to use Metadata enrichment for profiling. If set to true, Metadata enrichment features are eanbled (will use new Lite services).
Default:
false
Update data profile
Status: Complete
Updates a data profile in the project or catalog specified in the data profile, provided the caller has the necessary rights to do so.
During update, the entire data profile is replaced, apart from any read-only or response-only attributes.
If the profiling processes is running and the start parameter is set to true, then a data profile is only updated if the stop_in_progress_runs parameter is set to true.
The updates must be specified by using the JSON patch format, described in RFC 6902.
AMQP 1.0 Messages
When a data profile is modified, a message is fired with a state of the modified data profile in the body.
Topic
v2.data_profiles.:guid.PATCH
Subscribe to it by using "v2.data_profiles.*.PATCH" Binding Key.
Example Message
Topic: v2.data_profiles.5210c7d-cf6b-4204-95d2-95d84ecbe382.PATCH
Message:
{
"event": "UPDATE_DATA_PROFILE",
"actor": {
"user_name": "john@acme.com",
},
"published": "2015-05-10T15:04:15Z",
"url": { /* The href of the DataProfile created */}
"status_code": { /* The Http Status code , 200 if the DataProfile is updated successfully */}
"state": { /* the data profile object equivalent to the one obtained by GET /v2/data_profiles/{profile_id} API */ }
"details": {
"catalog_id": "f3c59258-abdd-4e24-828b-0495ec519339",
"dataset_id": "e522db21-59e8-44ab-81b2-bb40c3030a6f",
"profile_id": "5165d439-96f0-40d4-90b2-93795deab61b",
"is_governed": false { /* set to true if the catalog is_governed else set it to false}
}
}
PATCH /v2/data_profiles/{profile_id}Request
Path Parameters
The profile ID to use.
Query Parameters
The ID of the data set to use.
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
Whether to stop all running profiling processes. Data profiles that are running must be stopped before they can be deleted.
Default:
falseWhether to use Metadata enrichment for profiling. If set to true, Metadata enrichment features are eanbled (will use new Lite services).
Default:
falseWhether to start the profiling service immediately after the data profile is updated.
The updates to make in the data profile.
The operation to be performed
Allowable values: [
add,remove,replace,move,copy,test]A JSON pointer to the field to update
A string containing a JSON pointer value
- value
Response
A data profile holds the data profiling controls, options and results of a data set.
Profiling specific metadata
Profiling specific metadata
Information about the number of records
The underlying asset entity.
- entity
The data profile definition.
The data profile definition.
Column metadata inferred by analysis.
The location url where the data profile is created.
Status Code
Success.
You are not authorized to update the data profile you specified.
You are not permitted to perform this action.
The data profile you specified cannot be found.
An error occurred. The data profile you specified cannot be updated.
No Sample Response
Update data profiles
Status: Complete
Updates data profiles in the project or catalog for one or more assets, provided the caller has the necessary rights to do so.
During update, the entire data profiles are replaced, apart from any read-only or response-only attributes.
If any profiling process is running and the start parameter is set to true, then a data profile is only updated if the stop_in_progress_runs parameter is set to true.
The updates must be specified by using the JSON patch format, described in RFC 6902.
AMQP 1.0 Messages
When a data profile is modified, a message is fired with a state of the modified data profile in the body.
Topic
v2.data_profiles.:guid.PATCH
Subscribe to it by using "v2.data_profiles.*.PATCH" Binding Key.
Example Message
Topic: v2.data_profiles.5210c7d-cf6b-4204-95d2-95d84ecbe382.PATCH Message:
{
"event": "UPDATE_DATA_PROFILE",
"actor": {
"user_name": "john@acme.com",
},
"published": "2015-05-10T15:04:15Z",
"url": { /* The href of the DataProfile created */}
"status_code": { /* The Http Status code , 200 if the DataProfile is updated successfully */}
"state": { /* the data profile object equivalent to the one obtained by GET /v2/data_profiles/{profile_id} API */ }
"details": {
"catalog_id": "f3c59258-abdd-4e24-828b-0495ec519339",
"dataset_id": "e522db21-59e8-44ab-81b2-bb40c3030a6f",
"profile_id": "5165d439-96f0-40d4-90b2-93795deab61b",
"is_governed": false { /* set to true if the catalog is_governed else set it to false}
}
}
POST /v2/data_profiles/bulk
Request
Query Parameters
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
The updates to make in the one or more data profiles.
The ID of the profile asset
Contains updates to an object via a PATCH request, as specified by the JSON patch format in RFC 6902
Response
Contains updates to more than one asset via a POST request, as specified by the JSON patch format in RFC 6902
Contains respone details for one asset in a bulk assets update response, as specified by the JSON patch format in RFC 6902
Trace key associated with the current request
Status Code
Success.
You are not authorized to update the data profile you specified.
You are not permitted to perform this action.
The catalog / project you specified cannot be found.
An error occurred. The catalog / project you specified cannot be updated.
No Sample Response
Stop profiling process
Status: Complete
Stops an in-progress profiling run in the specified project or catalog, provided the caller has the necessary rights to do so. This API will request the action to be performed and return, not perform the action and wait for it to complete.
PUT /v2/data_profiles/{profile_id}/executionRequest
Path Parameters
The profile ID to use.
Query Parameters
The ID of the data set to use.
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
Whether to use Metadata enrichment for profiling. If set to true, Metadata enrichment features are eanbled (will use new Lite services).
Default:
false
The requested action to perform on the profile run.
The requested action (Response only).
Allowable values: [
stop,stop_all]
Get profiling logs
Status: Complete
Get the logs for the data profile job run on data set in a project or catalog.
GET /v2/data_profiles/{profile_id}/logsRequest
Path Parameters
The profile ID to use.
Query Parameters
The ID of the data set to use.
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
Response
A collection of data profile job run log entries.
A collection of data profile job run log entries.
Status Code
Success.
You are not authorized to get the profiling logs.
You are not permitted to perform this action.
The data profile you specified cannot be found.
An error occurred. Failed to get profiling logs.
No Sample Response
Modify asset level classification
Status: Complete
Modifies asset level classification detail in the data_profile attribute in the specified project or catalog, provided the caller has the necessary rights to do so. This API is used for CRUD operations on asset level classification.
The patch request for classification should contain the classification details that are to be added to the data_profile attribute.
The updates must be specified by using the JSON patch format, described in RFC 6902.
PATCH /v2/data_profiles/classification
Request
Query Parameters
The ID of the data set to use.
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
Whether to use Metadata enrichment for profiling. If set to true, Metadata enrichment features are eanbled (will use new Lite services).
Default:
false
The asset level classification details that are added to the data_profile attribute.
The operation to be performed
Allowable values: [
add,remove,replace,move,copy,test]A JSON pointer to the field to update
A string containing a JSON pointer value
- value
Response
Custom data to be associated with a given object
Status Code
Success.
Invalid classification details provided as part of the patch call.
You are not authorized to modify the data_profile attribute with asset level classification details.
You are not permitted to perform this action.
The asset classification cannot be found.
An error occurred. The data_profile attribute cannot be modified with asset level classification detail.
No Sample Response
Export data profile
Status: Complete
Export data profile of dataset in a project or catalog. (For internal use only)
GET /v2/data_profiles/export
Request
Query Parameters
The ID of the data set to use.
The ID of the export task ID to use.
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
Response
List of file paths that hold the profiling results. These files should be added to the project export archive.
Status Code
Success.
You are not authorized to persist results.
You are not permitted to perform this action.
Asset or profile is missing
An error occurred. The profiling results cannot be saved.
This API is not Implemented.
No Sample Response
Import data profile
Status: Complete
Import data profile of dataset in a project or catalog. (For internal use only)
POST /v2/data_profiles/import
Request
Query Parameters
The ID of the data set to use.
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
The data profile to import.
Source path where the import archive is deflated. If this value is provided, the profiling results are read from source_path/assets/profiling directory by setting the root=true. If this value is not provided, results files are read from profiling directory by setting the root=false.
Map of the old asset ids to new asset ids
- asset_ref_map
IBM cloud IAM user id of the creator of the asset.
Possible values: 1 ≤ length ≤ 100
Example:
IBMid-270002ABCD
This filters rows from the frequency distribution.
This filters rows from the frequency distribution.
POST /v2/data_profiles/{profile_id}/unique_valuesRequest
Path Parameters
The profile ID to use.
filters to get frequency distribution
Example:
dataset_idExample:
col1Example:
project_idExample:
catalog_idDefault:
falseExample:
some_valueExample:
10Example:
AAAExample:
100Possible values: 0 ≤ value ≤ 1000
Example:
100Example:
INT8Example:
10Example:
5Example:
2Example:
AAAExample:
U
Response
list of frequency distribution result of the column
Example:
120frequency distribution result
Status Code
Success.
You are not authorized to get the key analysis you specified.
You are not permitted to perform this action.
The key analysis task you specified cannot be found.
An error occurred. The data profile you specified cannot be fetched.
No Sample Response
Get data profile options
Status: Complete
Get the data profile options for the Account specified in the IAM token. If the account level data profile options is not available check if it is specified at project or catalog account level, provided the caller has the necessary rights to do so.
GET /v2/data_profiles/options
Request
Query Parameters
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
Whether to use Metadata enrichment for profiling. If set to true, Metadata enrichment features are eanbled (will use new Lite services).
Default:
false
Response
Definition of a data profile options.
Metadata about data profile options
The underlying asset entity.
- entity
The options of the profiling execution.
Status Code
Success.
You are not permitted to perform this action.
You are not authorized to list the available data profiles options.
Not Found.
An error occurred. The data profiles options cannot be listed.
This API is not Implemented.
No Sample Response
Create data profile options
Status: Complete
Creates a data profile options for a given Account Id in the specified project or catalog, provided the caller has the necessary rights to do so.
POST /v2/data_profiles/options
Request
Query Parameters
Whether to use Metadata enrichment for profiling. If set to true, Metadata enrichment features are eanbled (will use new Lite services).
Default:
false
The data profile option to create.
Metadata about data profile options
The underlying asset entity.
- entity
The options of the profiling execution.
Response
Definition of a data profile options.
Metadata about data profile options
The underlying asset entity.
- entity
The options of the profiling execution.
Status Code
Success.
You are not authorized to create data profile options.
You are not permitted to perform this action.
The data profile options cannot be found.
An error occurred. The data profile options cannot be created.
This API is not Implemented.
No Sample Response
Delete data profile options
Status: Complete
Delete the list of data profile options in the specified project or catalog, provided the caller has the necessary rights to do so. If the user specifies the catalog_id or the project_id then the data profile options specified for the catalog or project is deleted. If none of the query parameters,catalog_id or project_id is specified, the default is to delete the account level data profile options.
DELETE /v2/data_profiles/options
Request
Query Parameters
Comma separated list of the BSS Account IDs to use (restriced usage).
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
Whether to use Metadata enrichment for profiling. If set to true, Metadata enrichment features are eanbled (will use new Lite services).
Default:
false
Update the data profile options.
Status: Complete
Updates the data profile options for a given Account in the specified project or catalog, provided the caller has the necessary rights to do so. The data profile options that can be modified are max_row_count,row_percentage,max_distribution_size,max_numeric_stats_bin disabled flag in classification options,ibm_class_codes and custom_class_codes.
If the user specifies the catalog_id or the project_id then the data profile options specified for the catalog or project is updated. Update the account level data profile options if catalog_id or project_id is not specified.
PATCH /v2/data_profiles/options
Request
Query Parameters
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
Whether to use Metadata enrichment for profiling. If set to true, Metadata enrichment features are eanbled (will use new Lite services).
Default:
false
The data profile options to add.
The operation to be performed
Allowable values: [
add,remove,replace,move,copy,test]A JSON pointer to the field to update
A string containing a JSON pointer value
- value
Response
Definition of a data profile options.
Metadata about data profile options
The underlying asset entity.
- entity
The options of the profiling execution.
Status Code
Success.
You are not authorized to update the data profile options.
You are not permitted to perform this action.
The data profile options you specified cannot be found.
An error occurred. The data profile options cannot be updated.
This API is not Implemented.
No Sample Response
Persist profiling results
Status: Complete
Persist profiling results of dataset in a project or catalog. (For internal use only)
POST /v2/data_profiles/results
Update data profile results status
Status: Complete
Update data profile result status of dataset in a project or catalog. (For internal use only)
POST /v2/data_profiles/results/status
Request
Query Parameters
The ID of the data set to use.
The tenant ID to use.
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
The ID to be used for tracking end to end flow.
The data profile result status to update.
The data profile status.
Get bivariate results of given asset.
Status: Complete
Get bivariate results of dataset in a project or catalog. (For internal use only)
GET /v2/data_profiles/bivariate_results
Persist bivariate results of given asset.
Status: Complete
Persist bivariate results of dataset in a project or catalog. (For internal use only)
POST /v2/data_profiles/bivariate_results
Migrate data profile quality results to new Data Quality service
Status: Complete
Migrate data profile quality results of dataset in a project or catalog to new Data Quality service.
POST /v2/data_profiles/migrate
Request
Query Parameters
The ID of the catalog to use. catalog_id or project_id is required.
The ID of the project to use. catalog_id or project_id is required.
The ID of the data set to use.(Use comma separated data set Id's for multiple data sets)
Response
Migration status logging to track progress.
Status Code
Successfully initiated migration process.
Bad request. Provided parameters could be invalid.
You are not authorized, either token is expired or invalid.
You are not permitted to perform this action. Only Editor or Admin are allowed.
Catalog or Project not found.
An error occurred. The profiling results cannot be migrated to data quality service.
This API is not Implemented.
No Sample Response
List all data quality rules or a subset of them
Get a list of data quality rules in the project.
GET /data_quality/v3/projects/{project_id}/rulesRequest
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
The start token of the resource from where the page should begin.
Possible values: 1 ≤ length ≤ 512
Example:
g1AAAAA-eJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqzmxiYWJiZGYGkOWDSyBJZAPCBD58The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Default:
200Example:
20Comma-separated list of data quality rule identifiers.
Possible values: 1 ≤ length ≤ 10000
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5,b1ba1d22-71a7-4adf-99b2-3c8ba1949710
Response
A collection of data quality rules to be returned.
The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Example:
20Total number of resources available.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
100The link to a page in paginated collection.
A collection of data quality rules.
Possible values: 1 ≤ number of items ≤ 200
The link to a page in paginated collection.
The link to a page in paginated collection.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the list of data quality rules in the specified project.
An error occurred. The list of data quality rules could not be returned.
{"total_count":100,"limit":50,"first":{"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/rules?limit=50"},"rules":[{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf","bound_expression":["TEST.table1.col1<=TEST.table2.col2"],"is_valid":true,"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/rules/7b3f3a79-6412-480b-a20c-393a3f7addbf","name":"table1.col1LessOrEqualTable2.col2","description":"The column TEST.table1.col1 has fewer or the same number of values as column TEST.table2.col2","sampling":{"size":2500,"interval":13,"sampling_type":"every_nth"},"output":{"columns":[{"variable_name":"col1","name":"out1","type":"rule_variable","disambiguator":1},{"name":"out2","type":"column","source_column":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}},{"expression":"col1-col2","name":"out4","type":"rule_expression","disambiguator":2},{"metric":"system_time","name":"out5","type":"metric"}],"database":{"records_type":"all_records","update_type":"append","location":{"connection":{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf"},"schema_name":"TEST","table_name":"output"}},"maximum_record_count":500},"input":{"definitions":[{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":1,"bindings":[{"variable_name":"col1","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}}]},{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":2,"bindings":[{"variable_name":"col2","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col2","type":"column"}}]}]},"joins":[{"type":"inner_join","left_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"right_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93xyz"},"left_column_name":"col1","right_column_name":"col2"}],"dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"apply_all_present_dimensions":false}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Create data quality rule
Create a data quality rule.
POST /data_quality/v3/projects/{project_id}/rulesRequest
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Data quality rule to create.
Example: {"name":"table1.col1LessOrEqualTable2.col2","description":"The column TEST.table1.col1 has fewer or the same number of values as column TEST.table2.col2","sampling":{"size":2500,"interval":13,"sampling_type":"every_nth"},"output":{"columns":[{"variable_name":"col1","name":"out1","type":"rule_variable","disambiguator":1},{"name":"out2","type":"column","source_column":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}},{"expression":"col1-col2","name":"out4","type":"rule_expression","disambiguator":2},{"metric":"system_time","name":"out5","type":"metric"}],"database":{"records_type":"all_records","update_type":"append","location":{"connection":{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf"},"schema_name":"TEST","table_name":"output"},"inherit_project_level_output_setting":false,"create_table_only_when_issues_are_found":false,"import_table_in_project":false},"maximum_record_count":500},"input":{"definitions":[{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":1,"bindings":[{"variable_name":"col1","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}}]},{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":2,"bindings":[{"variable_name":"col2","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col2","type":"column"}}]}]},"joins":[{"type":"inner_join","left_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"right_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93xyz"},"left_column_name":"col1","right_column_name":"col2"}],"dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"apply_all_present_dimensions":false}
The name of the data quality rule. The rule name must be unique in the given project. If no unique name is provided, the quality rule will not be created.
Possible values: 1 ≤ length ≤ 200
Example:
address_exists_ruleData quality rule input details.
The description of the data quality rule. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Rule to check address exists.Identity of a data quality dimension resource. If this property is omitted, no data quality dimension is associated with the resource.
Flag indicating whether the rule contributes to all dimensions that are defined in the data quality definitions used in this rule. If the value of this field is set to true, then
dimensionwill be ignored.The output details of a data quality rule. If this property is omitted, no output records are saved.
The joins between data assets referenced in bindings and output. This property is not required if the rule is to be run on a single data asset. This property is also not required if a value for
data_stageis provided.Possible values: 1 ≤ number of items ≤ 50
The sampling options to be used during data quality rule run. If no sampling options are set, the rule is run against all the rows of the source.
Representation of the data stage flow resource to create. If this property is omitted, no subflow is created for the data quality rule.
Response
A data quality rule defines an executable applying a boolean expression on bound columns.
The name of the data quality rule. The name of the quality rule will be unique in a given project.
Possible values: 1 ≤ length ≤ 200
Example:
address_exists_ruleData quality rule input details.
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5Flag indicating whether the rule is valid or not.
The location URL of a resource.
Possible values: 1 ≤ length ≤ 512
Example:
https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbfThe description of the data quality rule. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Rule to check address exists.Identity of a data quality dimension resource.
Flag indicating whether the rule contributes to all dimensions that are defined in the data quality definitions used in this rule. If the value of this field is set to true, then
dimensionwill be ignored.The output details of a data quality rule.
The joins between data assets referenced in bindings and output. This property is not required if the rule is to be run on a single data asset. This property is also not required if a
data_stageelement is provided.Possible values: 1 ≤ number of items ≤ 50
The sampling options to be used during data quality rule run. If no sampling options are set, the rule is run against all the rows of the source.
Data stage flow details.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to create data quality rules in the specified project.
An error occurred. The data quality rule could not be created.
{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf","bound_expression":["TEST.table1.col1<=TEST.table2.col2"],"is_valid":true,"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/rules/7b3f3a79-6412-480b-a20c-393a3f7addbf","name":"table1.col1LessOrEqualTable2.col2","description":"The column TEST.table1.col1 has fewer or the same number of values as column TEST.table2.col2","sampling":{"size":2500,"interval":13,"sampling_type":"every_nth"},"output":{"columns":[{"variable_name":"col1","name":"out1","type":"rule_variable","disambiguator":1},{"name":"out2","type":"column","source_column":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}},{"expression":"col1-col2","name":"out4","type":"rule_expression","disambiguator":2},{"metric":"system_time","name":"out5","type":"metric"}],"database":{"records_type":"all_records","update_type":"append","location":{"connection":{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf"},"schema_name":"TEST","table_name":"output"}},"inherit_project_level_output_setting":"false,","create_table_only_when_issues_are_found":"true,","import_table_in_project":true,"maximum_record_count":500},"input":{"definitions":[{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":1,"bindings":[{"variable_name":"col1","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}}]},{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":2,"bindings":[{"variable_name":"col2","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col2","type":"column"}}]}]},"joins":[{"type":"inner_join","left_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"right_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93xyz"},"left_column_name":"col1","right_column_name":"col2"}],"dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"apply_all_present_dimensions":false}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Delete data quality rules
Delete the data quality rules for the given list of rule identifiers.
DELETE /data_quality/v3/projects/{project_id}/rulesRequest
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
Comma-separated list of data quality rule identifiers.
Possible values: 1 ≤ length ≤ 10000
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5,b1ba1d22-71a7-4adf-99b2-3c8ba1949710The option to delete related output tables when deleting data quality rules.
Default:
falseThe option to cancel unfinished jobs before deleting or updating data quality rules.
Default:
false
Response
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to delete the data quality rules in the specified project.
An error occurred. The data quality rules cannot be deleted.
{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Validate data quality rule
Check the validity of the data quality rule.
POST /data_quality/v3/projects/{project_id}/validate_ruleRequest
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Data quality rule to validate.
Example: {"name":"table1.col1LessOrEqualTable2.col2","description":"The column TEST.table1.col1 has fewer or the same number of values as column TEST.table2.col2","sampling":{"size":2500,"interval":13,"sampling_type":"every_nth"},"output":{"columns":[{"variable_name":"col1","name":"out1","type":"rule_variable","disambiguator":1},{"name":"out2","type":"column","source_column":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}},{"expression":"col1-col2","name":"out4","type":"rule_expression","disambiguator":2},{"metric":"system_time","name":"out5","type":"metric"}],"database":{"records_type":"all_records","update_type":"append","location":{"connection":{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf"},"schema_name":"TEST","table_name":"output"},"inherit_project_level_output_setting":false,"create_table_only_when_issues_are_found":false,"import_table_in_project":false},"maximum_record_count":500},"input":{"definitions":[{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":1,"bindings":[{"variable_name":"col1","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}}]},{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":2,"bindings":[{"variable_name":"col2","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col2","type":"column"}}]}]},"joins":[{"type":"inner_join","left_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"right_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93xyz"},"left_column_name":"col1","right_column_name":"col2"}],"dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"apply_all_present_dimensions":false}
The name of the data quality rule. The rule name must be unique in the given project. If no unique name is provided, the quality rule will not be created.
Possible values: 1 ≤ length ≤ 200
Example:
address_exists_ruleData quality rule input details.
The description of the data quality rule. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Rule to check address exists.Identity of a data quality dimension resource. If this property is omitted, no data quality dimension is associated with the resource.
Flag indicating whether the rule contributes to all dimensions that are defined in the data quality definitions used in this rule. If the value of this field is set to true, then
dimensionwill be ignored.The output details of a data quality rule. If this property is omitted, no output records are saved.
The joins between data assets referenced in bindings and output. This property is not required if the rule is to be run on a single data asset. This property is also not required if a value for
data_stageis provided.Possible values: 1 ≤ number of items ≤ 50
The sampling options to be used during data quality rule run. If no sampling options are set, the rule is run against all the rows of the source.
Representation of the data stage flow resource to create. If this property is omitted, no subflow is created for the data quality rule.
Response
A data quality rule defines an executable applying a boolean expression on bound columns.
The name of the data quality rule. The name of the quality rule will be unique in a given project.
Possible values: 1 ≤ length ≤ 200
Example:
address_exists_ruleData quality rule input details.
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5Flag indicating whether the rule is valid or not.
The location URL of a resource.
Possible values: 1 ≤ length ≤ 512
Example:
https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbfThe description of the data quality rule. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Rule to check address exists.Identity of a data quality dimension resource.
Flag indicating whether the rule contributes to all dimensions that are defined in the data quality definitions used in this rule. If the value of this field is set to true, then
dimensionwill be ignored.The output details of a data quality rule.
The joins between data assets referenced in bindings and output. This property is not required if the rule is to be run on a single data asset. This property is also not required if a
data_stageelement is provided.Possible values: 1 ≤ number of items ≤ 50
The sampling options to be used during data quality rule run. If no sampling options are set, the rule is run against all the rows of the source.
Data stage flow details.
Status Code
The data quality rule is valid.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to validate data quality rules in the specified project.
The data quality rule is invalid. See the error message for the cause.
{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf","bound_expression":["TEST.table1.col1<=TEST.table2.col2"],"is_valid":true,"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/rules/7b3f3a79-6412-480b-a20c-393a3f7addbf","name":"table1.col1LessOrEqualTable2.col2","description":"The column TEST.table1.col1 has fewer or the same number of values as column TEST.table2.col2","sampling":{"size":2500,"interval":13,"sampling_type":"every_nth"},"output":{"columns":[{"variable_name":"col1","name":"out1","type":"rule_variable","disambiguator":1},{"name":"out2","type":"column","source_column":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}},{"expression":"col1-col2","name":"out4","type":"rule_expression","disambiguator":2},{"metric":"system_time","name":"out5","type":"metric"}],"database":{"records_type":"all_records","update_type":"append","location":{"connection":{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf"},"schema_name":"TEST","table_name":"output"}},"inherit_project_level_output_setting":"false,","create_table_only_when_issues_are_found":"true,","import_table_in_project":true,"maximum_record_count":500},"input":{"definitions":[{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":1,"bindings":[{"variable_name":"col1","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}}]},{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":2,"bindings":[{"variable_name":"col2","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col2","type":"column"}}]}]},"joins":[{"type":"inner_join","left_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"right_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93xyz"},"left_column_name":"col1","right_column_name":"col2"}],"dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"apply_all_present_dimensions":false}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Get data quality rule
Gets the data quality rule with the given identifier.
GET /data_quality/v3/projects/{project_id}/rules/{id}Request
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The data quality rule identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Response
A data quality rule defines an executable applying a boolean expression on bound columns.
The name of the data quality rule. The name of the quality rule will be unique in a given project.
Possible values: 1 ≤ length ≤ 200
Example:
address_exists_ruleData quality rule input details.
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5Flag indicating whether the rule is valid or not.
The location URL of a resource.
Possible values: 1 ≤ length ≤ 512
Example:
https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbfThe description of the data quality rule. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Rule to check address exists.Identity of a data quality dimension resource.
Flag indicating whether the rule contributes to all dimensions that are defined in the data quality definitions used in this rule. If the value of this field is set to true, then
dimensionwill be ignored.The output details of a data quality rule.
The joins between data assets referenced in bindings and output. This property is not required if the rule is to be run on a single data asset. This property is also not required if a
data_stageelement is provided.Possible values: 1 ≤ number of items ≤ 50
The sampling options to be used during data quality rule run. If no sampling options are set, the rule is run against all the rows of the source.
Data stage flow details.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the data quality rule with the given identifier from the specified project.
The data quality rule cannot be found.
An error occurred. The data quality rule with the given identifier cannot be returned.
{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf","bound_expression":["TEST.table1.col1<=TEST.table2.col2"],"is_valid":true,"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/rules/7b3f3a79-6412-480b-a20c-393a3f7addbf","name":"table1.col1LessOrEqualTable2.col2","description":"The column TEST.table1.col1 has fewer or the same number of values as column TEST.table2.col2","sampling":{"size":2500,"interval":13,"sampling_type":"every_nth"},"output":{"columns":[{"variable_name":"col1","name":"out1","type":"rule_variable","disambiguator":1},{"name":"out2","type":"column","source_column":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}},{"expression":"col1-col2","name":"out4","type":"rule_expression","disambiguator":2},{"metric":"system_time","name":"out5","type":"metric"}],"database":{"records_type":"all_records","update_type":"append","location":{"connection":{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf"},"schema_name":"TEST","table_name":"output"}},"inherit_project_level_output_setting":"false,","create_table_only_when_issues_are_found":"true,","import_table_in_project":true,"maximum_record_count":500},"input":{"definitions":[{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":1,"bindings":[{"variable_name":"col1","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}}]},{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":2,"bindings":[{"variable_name":"col2","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col2","type":"column"}}]}]},"joins":[{"type":"inner_join","left_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"right_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93xyz"},"left_column_name":"col1","right_column_name":"col2"}],"dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"apply_all_present_dimensions":false}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Delete data quality rule
Delete the data quality rule with the given identifier.
DELETE /data_quality/v3/projects/{project_id}/rules/{id}Request
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The data quality rule identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
The option to delete related output tables when deleting data quality rules.
Default:
falseThe option to cancel unfinished jobs before deleting or updating data quality rules.
Default:
false
Response
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to delete the data quality rule from the specified project.
The data quality rule cannot be found.
An error occurred. The data quality rule cannot be deleted.
{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Update data quality rule
Updates a data quality rule as specified in the payload details of the update rule request. The updates must be specified by using the JSON patch format, described in RFC 6902.
The following attributes can be patched:
- name (value can only be replaced)
- definition (value can only be replaced)
- description (value can be added, removed, or replaced)
- dimension (value can be added, removed, or replaced)
- input (value can be added, removed or replaced)
- output (value can be added, removed, or replaced)
- joins (value can be added, removed, or replaced)
- sampling (value can be added, removed, or replaced)
- data_stage/propagate_all_incoming_columns (value can be added, removed, or replaced)
PATCH /data_quality/v3/projects/{project_id}/rules/{id}Request
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The data quality rule identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
The option to cancel unfinished jobs before deleting or updating data quality rules.
Default:
falseThe option to keep the current data quality rule input binding relationships while updating the data quality rule.
Default:
false
The updates to make in the data quality rule.
Example: [{"op":"replace","path":"/description","value":"Column col1 has fewer or the same number of values as column col2"}]
The operation to be performed
Allowable values: [
add,remove,replace,move,copy,test]A JSON pointer to the field to update
A string containing a JSON pointer value
- value
Response
A data quality rule defines an executable applying a boolean expression on bound columns.
The name of the data quality rule. The name of the quality rule will be unique in a given project.
Possible values: 1 ≤ length ≤ 200
Example:
address_exists_ruleData quality rule input details.
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5Flag indicating whether the rule is valid or not.
The location URL of a resource.
Possible values: 1 ≤ length ≤ 512
Example:
https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbfThe description of the data quality rule. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Rule to check address exists.Identity of a data quality dimension resource.
Flag indicating whether the rule contributes to all dimensions that are defined in the data quality definitions used in this rule. If the value of this field is set to true, then
dimensionwill be ignored.The output details of a data quality rule.
The joins between data assets referenced in bindings and output. This property is not required if the rule is to be run on a single data asset. This property is also not required if a
data_stageelement is provided.Possible values: 1 ≤ number of items ≤ 50
The sampling options to be used during data quality rule run. If no sampling options are set, the rule is run against all the rows of the source.
Data stage flow details.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to update the data quality rule in the specified project.
The data quality rule cannot be found.
An error occurred. The data quality rule was not updated.
{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf","bound_expression":["TEST.table1.col1<=TEST.table2.col2"],"is_valid":true,"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/rules/7b3f3a79-6412-480b-a20c-393a3f7addbf","name":"table1.col1LessOrEqualTable2.col2","description":"The column TEST.table1.col1 has fewer or the same number of values as column TEST.table2.col2","sampling":{"size":2500,"interval":13,"sampling_type":"every_nth"},"output":{"columns":[{"variable_name":"col1","name":"out1","type":"rule_variable","disambiguator":1},{"name":"out2","type":"column","source_column":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}},{"expression":"col1-col2","name":"out4","type":"rule_expression","disambiguator":2},{"metric":"system_time","name":"out5","type":"metric"}],"database":{"records_type":"all_records","update_type":"append","location":{"connection":{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf"},"schema_name":"TEST","table_name":"output"}},"inherit_project_level_output_setting":"false,","create_table_only_when_issues_are_found":"true,","import_table_in_project":true,"maximum_record_count":500},"input":{"definitions":[{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":1,"bindings":[{"variable_name":"col1","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}}]},{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":2,"bindings":[{"variable_name":"col2","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col2","type":"column"}}]}]},"joins":[{"type":"inner_join","left_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"right_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93xyz"},"left_column_name":"col1","right_column_name":"col2"}],"dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"apply_all_present_dimensions":false}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Run data quality rule
Run the data quality rule with the given identifier.
POST /data_quality/v3/projects/{project_id}/rules/{id}/executeRequest
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The data quality rule identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Response
The representation of a data quality rule execution.
Identity of a job associated with a rule.
Identity of a run associated with a job, which in turn is associated with a a rule.
The status of the rule run.
The name of the user that ran the rule.
Possible values: 1 ≤ length ≤ 50
Example:
IBMid-270002ABCDResource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The location URL of a resource.
Possible values: 1 ≤ length ≤ 512
Example:
https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbfNumber of passing records.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
900Number of failing records.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
100Data quality rule input details.
The output details of a data quality rule.
The sampling options to be used during data quality rule run. If no sampling options are set, the rule is run against all the rows of the source.
Number of records tested.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
1000The start time of the rule run.
Possible values: 20 ≤ length ≤ 24
The end time of the rule run.
Possible values: 20 ≤ length ≤ 24
Name of the result.
Possible values: 1 ≤ length ≤ 200
Example:
execution1A list of execution results per definition.
Possible values: 1 ≤ number of items ≤ 100
Identity of an output table data asset. This property is omitted if no output table is defined for the data quality rule.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to run the data quality rule in the specified project.
The data quality rule cannot be found.
An error occurred. The data quality rule cannot be run.
{"name":"table1.col1LessOrEqualTable2.col2","job":{"id":"aa398b69-3e91-4830-877a-91a8e21f56de"},"job_run":{"id":"aa398b69-3e91-4830-877a-91a8e21f5123"},"status":{"state":"analyzed"},"started_at":"2022-02-01T18:10:02.000Z","ended_at":"2022-02-01T18:11:02.000Z","run_by":"IBMid-270002ABCD","sampling":{"size":2500,"interval":13,"sampling_type":"every_nth"},"input":{"definitions":[{"bound_expression":"TEST.table1.col1<=TEST.table2.col2"}]},"output":{"columns":[{"variable_name":"col1","name":"out1","type":"rule_variable","disambiguator":1},{"name":"out2","type":"column","source_column":{"data_asset":{"id":"aa398b69-3e91-4830-877a-91a8e21f56de"},"column_name":"col1","type":"column"}},{"expression":"col1-col2","name":"out4","type":"rule_expression","disambiguator":2},{"metric":"system_time","name":"out5","type":"metric"}],"database":{"records_type":"all_records","update_type":"append","location":{"connection":{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf"},"schema_name":"TEST","table_name":"output"}},"inherit_project_level_output_setting":"false,","create_table_only_when_issues_are_found":"true,","import_table_in_project":true,"maximum_record_count":500},"output_table":{"id":"5d4254a6-0c9e-4a02-bc24-4a89eee06f12"},"tested_record_count":1000,"passing_record_count":987,"failing_record_count":13,"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf","href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/rules/7b3f3a79-6412-480b-a20c-393a3f7addbf/executions/7b3f3a79-6412-480b-a20c-393a3f7adabc"}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Get data quality rule from catalog
Gets the data quality rule from catalog with the given identifier.
GET /data_quality/v3/catalogs/{catalog_id}/rules/{id}Request
Path Parameters
The identifier of the catalog to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The data quality rule identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Response
A data quality rule defines an executable applying a boolean expression on bound columns.
The name of the data quality rule. The name of the quality rule will be unique in a given project.
Possible values: 1 ≤ length ≤ 200
Example:
address_exists_ruleData quality rule input details.
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5Flag indicating whether the rule is valid or not.
The location URL of a resource.
Possible values: 1 ≤ length ≤ 512
Example:
https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbfThe description of the data quality rule. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Rule to check address exists.Identity of a data quality dimension resource.
Flag indicating whether the rule contributes to all dimensions that are defined in the data quality definitions used in this rule. If the value of this field is set to true, then
dimensionwill be ignored.The output details of a data quality rule.
The joins between data assets referenced in bindings and output. This property is not required if the rule is to be run on a single data asset. This property is also not required if a
data_stageelement is provided.Possible values: 1 ≤ number of items ≤ 50
The sampling options to be used during data quality rule run. If no sampling options are set, the rule is run against all the rows of the source.
Data stage flow details.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the data quality rule with the given identifier from the specified catalog.
The data quality rule cannot be found.
An error occurred. The data quality rule with the given identifier cannot be returned.
{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf","bound_expression":["TEST.table1.col1<=TEST.table2.col2"],"is_valid":true,"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/rules/7b3f3a79-6412-480b-a20c-393a3f7addbf","name":"table1.col1LessOrEqualTable2.col2","description":"The column TEST.table1.col1 has fewer or the same number of values as column TEST.table2.col2","sampling":{"size":2500,"interval":13,"sampling_type":"every_nth"},"output":{"columns":[{"variable_name":"col1","name":"out1","type":"rule_variable","disambiguator":1},{"name":"out2","type":"column","source_column":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}},{"expression":"col1-col2","name":"out4","type":"rule_expression","disambiguator":2},{"metric":"system_time","name":"out5","type":"metric"}],"database":{"records_type":"all_records","update_type":"append","location":{"connection":{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf"},"schema_name":"TEST","table_name":"output"}},"inherit_project_level_output_setting":"false,","create_table_only_when_issues_are_found":"true,","import_table_in_project":true,"maximum_record_count":500},"input":{"definitions":[{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":1,"bindings":[{"variable_name":"col1","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col1","type":"column"}}]},{"definition":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"disambiguator":2,"bindings":[{"variable_name":"col2","target":{"data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"column_name":"col2","type":"column"}}]}]},"joins":[{"type":"inner_join","left_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"right_data_asset":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93xyz"},"left_column_name":"col1","right_column_name":"col2"}],"dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"apply_all_present_dimensions":false}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
List history of all data quality rule run results or a subset of them
Gets the list of data quality rule runs. By default, the runs are sorted newest to oldest.
GET /data_quality/v3/projects/{project_id}/rules/{rule_id}/executionsRequest
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The data quality rule identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
The start token of the resource from where the page should begin.
Possible values: 1 ≤ length ≤ 512
Example:
g1AAAAA-eJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqzmxiYWJiZGYGkOWDSyBJZAPCBD58The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Default:
200Example:
20The property name by which the results should be sorted. For descending order, use the hyphen (-) prefix with the value. If you omit this query parameter, the order value is
-started_at, which means, results are sorted by thestarted_atvalue in descending order. The allowed fields arestarted_at,passing_record_count,failing_record_count,tested_record_count,status.state.Possible values: 1 ≤ length ≤ 128
Response
Data quality rule run history
The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Example:
20Total number of resources available.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
100The link to a page in paginated collection.
A list of rule run results, sorted based on the value of the
sortquery parameter.Possible values: 1 ≤ number of items ≤ 50
The link to a page in paginated collection.
The link to a page in paginated collection.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the data quality rule history in the specified project.
An error occurred. The data quality rule history could not be returned.
{"limit":20,"total_count":100,"first":{"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/rules/c19cde3a-5940-4c7a-ad0f-ee18f5f29c12/executions?limit=20"},"executions":[{"name":"table1.col1LessOrEqualTable2.col2","description":"The column TEST.table1.col1 has fewer or the same number of values as column TEST.table2.col2","job":{"id":"aa398b69-3e91-4830-877a-91a8e21f56de"},"job_run":{"id":"aa398b69-3e91-4830-877a-91a8e21f5123"},"status":{"state":"analyzed"},"started_at":"2022-02-01T18:10:02.000Z","ended_at":"2022-02-01T18:11:02.000Z","run_by":"IBMid-270002ABCD","sampling":{"size":2500,"interval":13,"sampling_type":"every_nth"},"output":{"columns":[{"variable_name":"col1","name":"out1","type":"rule_variable","disambiguator":1},{"name":"out2","type":"column","source_column":{"data_asset":{"id":"aa398b69-3e91-4830-877a-91a8e21f56de"},"column_name":"col1","type":"column"}},{"expression":"col1-col2","name":"out4","type":"rule_expression","disambiguator":2},{"metric":"system_time","name":"out5","type":"metric"}],"database":{"records_type":"all_records","update_type":"append","location":{"connection":{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf"},"schema_name":"TEST","table_name":"output"}},"maximum_record_count":500},"output_table":{"id":"5d4254a6-0c9e-4a02-bc24-4a89eee06f12"},"tested_record_count":1000,"bound_expression":["TEST.table1.col1<=TEST.table2.col2"],"passing_record_count":987,"failing_record_count":13,"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf","href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/rules/7b3f3a79-6412-480b-a20c-393a3f7addbf/executions/7b3f3a79-6412-480b-a20c-393a3f7adabc"}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Get the data quality rule run
Gets the data quality rule run result with the given identifier.
GET /data_quality/v3/projects/{project_id}/rules/{rule_id}/executions/{id}Request
Path Parameters
The data quality rule identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The run identifier to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Response
The representation of a data quality rule execution.
Identity of a job associated with a rule.
Identity of a run associated with a job, which in turn is associated with a a rule.
The status of the rule run.
The name of the user that ran the rule.
Possible values: 1 ≤ length ≤ 50
Example:
IBMid-270002ABCDResource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The location URL of a resource.
Possible values: 1 ≤ length ≤ 512
Example:
https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbfNumber of passing records.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
900Number of failing records.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
100Data quality rule input details.
The output details of a data quality rule.
The sampling options to be used during data quality rule run. If no sampling options are set, the rule is run against all the rows of the source.
Number of records tested.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
1000The start time of the rule run.
Possible values: 20 ≤ length ≤ 24
The end time of the rule run.
Possible values: 20 ≤ length ≤ 24
Name of the result.
Possible values: 1 ≤ length ≤ 200
Example:
execution1A list of execution results per definition.
Possible values: 1 ≤ number of items ≤ 100
Identity of an output table data asset. This property is omitted if no output table is defined for the data quality rule.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the data quality rule run result in the specified project.
An error occurred. The result of data quality rule run could not be returned.
{"name":"table1.col1LessOrEqualTable2.col2","job":{"id":"aa398b69-3e91-4830-877a-91a8e21f56de"},"job_run":{"id":"aa398b69-3e91-4830-877a-91a8e21f5123"},"status":{"state":"analyzed"},"started_at":"2022-02-01T18:10:02.000Z","ended_at":"2022-02-01T18:11:02.000Z","run_by":"IBMid-270002ABCD","sampling":{"size":2500,"interval":13,"sampling_type":"every_nth"},"input":{"definitions":[{"bound_expression":"TEST.table1.col1<=TEST.table2.col2"}]},"output":{"columns":[{"variable_name":"col1","name":"out1","type":"rule_variable","disambiguator":1},{"name":"out2","type":"column","source_column":{"data_asset":{"id":"aa398b69-3e91-4830-877a-91a8e21f56de"},"column_name":"col1","type":"column"}},{"expression":"col1-col2","name":"out4","type":"rule_expression","disambiguator":2},{"metric":"system_time","name":"out5","type":"metric"}],"database":{"records_type":"all_records","update_type":"append","location":{"connection":{"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf"},"schema_name":"TEST","table_name":"output"}},"inherit_project_level_output_setting":"false,","create_table_only_when_issues_are_found":"true,","import_table_in_project":true,"maximum_record_count":500},"output_table":{"id":"5d4254a6-0c9e-4a02-bc24-4a89eee06f12"},"tested_record_count":1000,"passing_record_count":987,"failing_record_count":13,"id":"7b3f3a79-6412-480b-a20c-393a3f7addbf","href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/rules/7b3f3a79-6412-480b-a20c-393a3f7addbf/executions/7b3f3a79-6412-480b-a20c-393a3f7adabc"}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
List all data quality definitions or a subset of them
Gets the list of all data quality definitions.
GET /data_quality/v3/projects/{project_id}/definitionsRequest
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
The start token of the resource from where the page should begin.
Possible values: 1 ≤ length ≤ 512
Example:
g1AAAAA-eJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqzmxiYWJiZGYGkOWDSyBJZAPCBD58The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Default:
200Example:
20Comma-separated list of data quality definition identifiers.
Possible values: 1 ≤ length ≤ 10000
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5,b1ba1d22-71a7-4adf-99b2-3c8ba1949710
Response
An array of data quality definitions.
The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Example:
20Total number of resources available.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
100The link to a page in paginated collection.
A collection of existing definitions.
Possible values: 1 ≤ number of items ≤ 200
The link to a page in paginated collection.
The link to a page in paginated collection.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the list of data quality definitions in the specified project.
An error occurred. The list of data quality definitions cannot be returned.
{"total_count":100,"limit":200,"first":{"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions?limit=200"},"definitions":[{"description":"Column col1 has fewer or the same number of values as column col2","id":"7b3f3a79-6412-480b-a20c-393a3f7adoi","name":"col1LessOrEqualCol2","expression":"col1 <= col2","dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbf"}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Create data quality definition
Creates a data quality definition in the specified project.
POST /data_quality/v3/projects/{project_id}/definitionsRequest
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Data quality definition to create.
Example: {"name":"col1LessOrEqualCol2","description":"Column col1 has fewer or the same number of values as column col2","expression":"col1 <= col2","dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"}}
The name of the data quality definition. The name of the definition must be unique in the given project. If no unique name is provided, the REST call fails.
Possible values: 1 ≤ length ≤ 200
Example:
Address exists definitionThe expression of the data quality definition.
Possible values: 1 ≤ length ≤ 10000
Example:
field0 existsThe description of the data quality definition. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Definition to check if address existsIdentity of a data quality dimension resource. If this property is omitted, no data quality dimension is associated with the resource.
Response
A data quality definition defines a boolean rule along with the variables.
The name of the data quality definition. The name of the quality definition is unique in a given project.
Possible values: 1 ≤ length ≤ 200
Example:
Address exists definitionThe expression of the data quality definition.
Possible values: 1 ≤ length ≤ 10000
Example:
field0 existsResource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The location URL of a resource.
Possible values: 1 ≤ length ≤ 512
Example:
https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbfThe description of the data quality definition. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Definition to check if address existsIdentity of a data quality dimension resource.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to create a definition in the specified project.
An error occurred. The definition was not created.
{"description":"Column col1 has fewer or the same number of values as column col2","id":"7b3f3a79-6412-480b-a20c-393a3f7adoi","name":"col1LessOrEqualCol2","expression":"col1 <= col2","dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbf"}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Delete data quality definitions
Delete the data quality definitions with the given identifiers.
DELETE /data_quality/v3/projects/{project_id}/definitionsRequest
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
Comma separated list of data quality definition identifiers.
Possible values: 1 ≤ length ≤ 10000
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5,b1ba1d22-71a7-4adf-99b2-3c8ba1949710
Response
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to delete the data quality definitions in the specified project or catalog.
An error occurred. The data quality definitions cannot be deleted.
{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Validate data quality definition expression
Validate the specified definition expression.
POST /data_quality/v3/projects/{project_id}/validate_definitionRequest
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
True if the expression to validate is an output column expression. False if the expression to validate is a data quality definition expression. Default value is false.
Data quality definition to validate.
Example: {"name":"col1LessOrEqualCol2","description":"Column col1 has fewer or the same number of values as column col2","expression":"col1 <= col2","dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"}}
The name of the data quality definition. The name of the definition must be unique in the given project. If no unique name is provided, the REST call fails.
Possible values: 1 ≤ length ≤ 200
Example:
Address exists definitionThe expression of the data quality definition.
Possible values: 1 ≤ length ≤ 10000
Example:
field0 existsThe description of the data quality definition. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Definition to check if address existsIdentity of a data quality dimension resource. If this property is omitted, no data quality dimension is associated with the resource.
Response
This response contains the rule expression variables and their types.
The list of variables in this expression. This field is set if the rule is valid (
is_valid=true).Possible values: 1 ≤ number of items ≤ 100
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to validate the data quality definition in the specified project or catalog.
An error occurred. The data quality definitions could not be validated.
{"variables":[{"name":"col1","type":"numeric","usage":"input_data"},{"name":"col2","type":"numeric","usage":"input_data"}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Get data quality definition
Gets the data quality definition with the given identifier.
GET /data_quality/v3/projects/{project_id}/definitions/{id}Request
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The data quality definition identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Response
A data quality definition defines a boolean rule along with the variables.
The name of the data quality definition. The name of the quality definition is unique in a given project.
Possible values: 1 ≤ length ≤ 200
Example:
Address exists definitionThe expression of the data quality definition.
Possible values: 1 ≤ length ≤ 10000
Example:
field0 existsResource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The location URL of a resource.
Possible values: 1 ≤ length ≤ 512
Example:
https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbfThe description of the data quality definition. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Definition to check if address existsIdentity of a data quality dimension resource.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the data quality definition with the given ID from the specified project or catalog.
The data quality definition cannot be found.
An error occurred. The data quality definition with the given identifier could not be returned.
{"description":"Column col1 has fewer or the same number of values as column col2","id":"7b3f3a79-6412-480b-a20c-393a3f7adoi","name":"col1LessOrEqualCol2","expression":"col1 <= col2","dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbf"}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Replace data quality definition
Replace a data quality definition.
PUT /data_quality/v3/projects/{project_id}/definitions/{id}Request
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The data quality definition identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Data quality definition to replace.
Example: {"name":"col1LessOrEqualCol2","description":"Column col1 has fewer or the same number of values as column col2","expression":"col1 <= col2","dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"}}
The name of the data quality definition. The name of the definition must be unique in the given project. If no unique name is provided, the REST call fails.
Possible values: 1 ≤ length ≤ 200
Example:
Address exists definitionThe expression of the data quality definition.
Possible values: 1 ≤ length ≤ 10000
Example:
field0 existsThe description of the data quality definition. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Definition to check if address existsIdentity of a data quality dimension resource. If this property is omitted, no data quality dimension is associated with the resource.
Response
A data quality definition defines a boolean rule along with the variables.
The name of the data quality definition. The name of the quality definition is unique in a given project.
Possible values: 1 ≤ length ≤ 200
Example:
Address exists definitionThe expression of the data quality definition.
Possible values: 1 ≤ length ≤ 10000
Example:
field0 existsResource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The location URL of a resource.
Possible values: 1 ≤ length ≤ 512
Example:
https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbfThe description of the data quality definition. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Definition to check if address existsIdentity of a data quality dimension resource.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to update data quality definitions in the specified project or catalog.
An error occurred. The data quality definition was not updated.
{"description":"Column col1 has fewer or the same number of values as column col2","id":"7b3f3a79-6412-480b-a20c-393a3f7adoi","name":"col1LessOrEqualCol2","expression":"col1 <= col2","dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbf"}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Delete data quality definition
Delete the data quality definition with the given identifier.
DELETE /data_quality/v3/projects/{project_id}/definitions/{id}Request
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The data quality definition identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Response
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to delete the data quality definition from the specified project.
The data quality definition cannot be found.
An error occurred. The data quality definition cannot be deleted.
{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Update data quality definition
Updates a data quality definition as specified in the payload details of the update definition request. The updates must be specified by using the JSON patch format, described in RFC 6902.
The following attributes can be patched.
- name (value can only be replaced)
- description (value can be added, removed, or replaced)
- expression (value can only be replaced)
- dimension (value can be added, removed, or replaced)
PATCH /data_quality/v3/projects/{project_id}/definitions/{id}Request
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The data quality definition identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
The updates to make in the data quality definition.
Example: [{"op":"replace","path":"/description","value":"Column col1 has fewer or the same number of values as column col2"}]
The operation to be performed
Allowable values: [
add,remove,replace,move,copy,test]A JSON pointer to the field to update
A string containing a JSON pointer value
- value
Response
A data quality definition defines a boolean rule along with the variables.
The name of the data quality definition. The name of the quality definition is unique in a given project.
Possible values: 1 ≤ length ≤ 200
Example:
Address exists definitionThe expression of the data quality definition.
Possible values: 1 ≤ length ≤ 10000
Example:
field0 existsResource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The location URL of a resource.
Possible values: 1 ≤ length ≤ 512
Example:
https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbfThe description of the data quality definition. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Definition to check if address existsIdentity of a data quality dimension resource.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to update the data quality definition in the specified project.
The data quality definition cannot be found.
An error occurred. The data quality definition was not updated.
{"description":"Column col1 has fewer or the same number of values as column col2","id":"7b3f3a79-6412-480b-a20c-393a3f7adoi","name":"col1LessOrEqualCol2","expression":"col1 <= col2","dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbf"}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Get data quality definition from catalog
Gets the data quality definition from catalog with the given identifier.
GET /data_quality/v3/catalogs/{catalog_id}/definitions/{id}Request
Path Parameters
The identifier of the catalog to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The data quality definition identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Response
A data quality definition defines a boolean rule along with the variables.
The name of the data quality definition. The name of the quality definition is unique in a given project.
Possible values: 1 ≤ length ≤ 200
Example:
Address exists definitionThe expression of the data quality definition.
Possible values: 1 ≤ length ≤ 10000
Example:
field0 existsResource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The location URL of a resource.
Possible values: 1 ≤ length ≤ 512
Example:
https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbfThe description of the data quality definition. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Definition to check if address existsIdentity of a data quality dimension resource.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the data quality definition with the given ID from the specified project or catalog.
The data quality definition cannot be found.
An error occurred. The data quality definition with the given identifier could not be returned.
{"description":"Column col1 has fewer or the same number of values as column col2","id":"7b3f3a79-6412-480b-a20c-393a3f7adoi","name":"col1LessOrEqualCol2","expression":"col1 <= col2","dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbf"}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Update data quality definition in catlog
Updates a data quality definition as specified in the payload details of the update definition request. The updates must be specified by using the JSON patch format, described in RFC 6902.
The following attributes can be patched.
- dimension (value can be added, removed, or replaced)
PATCH /data_quality/v3/catalogs/{catalog_id}/definitions/{id}Request
Path Parameters
The identifier of the catalog to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The data quality definition identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
The updates to make in the data quality definition.
Example: [{"op":"replace","path":"/description","value":"Column col1 has fewer or the same number of values as column col2"}]
The operation to be performed
Allowable values: [
add,remove,replace,move,copy,test]A JSON pointer to the field to update
A string containing a JSON pointer value
- value
Response
A data quality definition defines a boolean rule along with the variables.
The name of the data quality definition. The name of the quality definition is unique in a given project.
Possible values: 1 ≤ length ≤ 200
Example:
Address exists definitionThe expression of the data quality definition.
Possible values: 1 ≤ length ≤ 10000
Example:
field0 existsResource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The location URL of a resource.
Possible values: 1 ≤ length ≤ 512
Example:
https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbfThe description of the data quality definition. If this property is omitted, no description is set.
Possible values: 1 ≤ length ≤ 5000
Example:
Definition to check if address existsIdentity of a data quality dimension resource.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to update the data quality definition in the specified catalog.
The data quality definition cannot be found.
An error occurred. The data quality definition was not updated.
{"description":"Column col1 has fewer or the same number of values as column col2","id":"7b3f3a79-6412-480b-a20c-393a3f7adoi","name":"col1LessOrEqualCol2","expression":"col1 <= col2","dimension":{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93abc"},"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/definitions/7b3f3a79-6412-480b-a20c-393a3f7addbf"}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
List all data quality dimensions
Gets the list of all data quality dimensions
GET /data_quality/v3/projects/{project_id}/dimensionsRequest
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
The start token of the resource from where the page should begin.
Possible values: 1 ≤ length ≤ 512
Example:
g1AAAAA-eJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqzmxiYWJiZGYGkOWDSyBJZAPCBD58The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Default:
200Example:
20
Response
A collection of data quality dimensions.
The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Example:
20Total number of resources available.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
100The link to a page in paginated collection.
An array of data quality dimensions.
Possible values: 1 ≤ number of items ≤ 200
The link to a page in paginated collection.
The link to a page in paginated collection.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You are not permitted to get data quality dimensions of the specified project.
An error occurred. The list of data quality dimensions cannot be returned.
{"limit":20,"total_count":100,"first":{"href":"https://cloud.ibm.com/data_quality/v3/projects/c19cde3a-5940-4c7a-ad0f-ee18f5f29c00/dimensions?limit=20"},"dimensions":[{"id":"371114cd-5516-4691-8b2e-1e66edf66486","name":"Completeness","description":"The proportion of data stored against the potential for 100%.","creator_id":"SYSTEM","created_at":"2022-02-01T18:11:02.000Z"},{"id":"57561203-36e6-4bb0-adfb-ece981fba845","name":"Uniqueness","description":"No entity instance (`abc`) will be recorded more than once based on how `abc` is identified.","creator_id":"SYSTEM","created_at":"2022-02-01T18:11:02.000Z"},{"id":"a3134282-ed85-4140-ae61-e5f712a61985","name":"Timeliness","description":"The degree to which data represent reality from the required point in time.","creator_id":"SYSTEM","created_at":"2022-02-01T18:11:02.000Z"},{"id":"ec453723-669c-48bb-82c1-11b69b3b8c93","name":"Validity","description":"Data is valid if it conforms to the syntax (format, type, range) of its definition.","creator_id":"SYSTEM","created_at":"2022-02-01T18:11:02.000Z"},{"id":"5f66813e-adb1-4f3f-a65f-40e3066aac07","name":"Accuracy","description":"The degree to which data correctly describes the real world object or event being described.","creator_id":"SYSTEM","created_at":"2022-02-01T18:11:02.000Z"},{"id":"5f66813e-adb1-4f3f-a65f-40e3066aa123","name":"Consistency","description":"The absence of difference when comparing two or more representations of an object against a definition.","creator_id":"SYSTEM","created_at":"2022-02-01T18:11:02.000Z"}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Get project settings for data quality rules
Gets the project settings for data quality rules.
GET /data_quality/v3/projects/{project_id}/settingsRequest
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Response
Project settings.
In rules containing equality tests between strings, this property indicates whether the values to be tested should consider trailing spaces.
This property indicates whether to trim the value of the data type char/varchar before using that value in comparison.
This property indicates whether values of data quality rule variables have to be casted automatically to the type expected in the data quality rule expression.
The maximum length of String values when executing data quality rules.
Possible values: 1 ≤ value ≤ 1024
Example:
512In rules with multiple definitions, indicates if definitions evaluating as null (undefined) have to be considered failing (false) or passing (true).
This property indicates whether group keys for aggregations in the data quality rule are case sensitive or not.
If set to true, a string surrounded by double quotes in an expression will be treated as a variable instead of string literal value.
The heap size in MB of the Java stages needed to implement data rules.
Possible values: 1 ≤ value ≤ 2048
Example:
512In rules of the form "IF condition THEN test", this property indicates how data records that do not fulfil the condition have to be evaluated. If the property is not set, the data records are not counted. Otherwise, the records are counted as failing (false) or as passing (true) as defined.
Defines the output settings for rules output table in the project.
- project_rule_output
The output table location of the data quality rule.
Indicates whether to delete tables with no records.
Indicates whether tables are imported into the project.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the data quality rule settings in the specified project.
An error occurred. The project settings for data quality rules could not be returned.
{"default_else_value":true,"ignore_trailing_spaces":false,"implicit_casting":true,"max_string_length":1024,"allow_quoted_variables":false,"project_rule_output":{"location":{"connection":{"id":"connection_id"},"catalog_name":"catalogName","schema_name":"schemName","table_name":"tableName"},"create_table_only_when_issues_are_found":false,"import_table_in_project":true}}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Replace project settings for data quality rules
Replace project settings for data quality rules.
PUT /data_quality/v3/projects/{project_id}/settingsRequest
Path Parameters
The identifier of the project to use.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Project settings for data quality rules.
Example: {"default_else_value":true,"ignore_trailing_spaces":false,"implicit_casting":true,"max_string_length":1024,"allow_quoted_variables":false}
In rules of the form "IF condition THEN test", this property indicates how data records that do not fulfil the condition have to be evaluated. If the property is not set, the data records are not counted. Otherwise, the records are counted as failing (false) or as passing (true) as defined.
In rules containing equality tests between strings, this property indicates whether the values to be tested should consider trailing spaces.
Default:
falseThis property indicates whether to trim the value of the data type char/varchar before using that value in comparison.
Default:
trueThis property indicates whether values of data quality rule variables have to be casted automatically to the type expected in the data quality rule expression.
Default:
trueThe maximum length of String values when executing data quality rules.
Possible values: 1 ≤ value ≤ 1024
Default:
1024Example:
512In rules with multiple definitions, indicates if definitions evaluating as null (undefined) have to be considered failing (false) or passing (true).
Default:
trueThis property indicates whether group keys for aggregations in the data quality rule are case sensitive or not.
Default:
trueIf set to true, a string surrounded by double quotes in an expression will be treated as a variable instead of string literal value.
Default:
falseThe heap size in MB of the Java stages needed to implement data rules.
Possible values: 1 ≤ value ≤ 2048
Default:
256Example:
512
Response
Project settings.
In rules containing equality tests between strings, this property indicates whether the values to be tested should consider trailing spaces.
This property indicates whether to trim the value of the data type char/varchar before using that value in comparison.
This property indicates whether values of data quality rule variables have to be casted automatically to the type expected in the data quality rule expression.
The maximum length of String values when executing data quality rules.
Possible values: 1 ≤ value ≤ 1024
Example:
512In rules with multiple definitions, indicates if definitions evaluating as null (undefined) have to be considered failing (false) or passing (true).
This property indicates whether group keys for aggregations in the data quality rule are case sensitive or not.
If set to true, a string surrounded by double quotes in an expression will be treated as a variable instead of string literal value.
The heap size in MB of the Java stages needed to implement data rules.
Possible values: 1 ≤ value ≤ 2048
Example:
512In rules of the form "IF condition THEN test", this property indicates how data records that do not fulfil the condition have to be evaluated. If the property is not set, the data records are not counted. Otherwise, the records are counted as failing (false) or as passing (true) as defined.
Defines the output settings for rules output table in the project.
- project_rule_output
The output table location of the data quality rule.
Indicates whether to delete tables with no records.
Indicates whether tables are imported into the project.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to define project settings for data quality rules in the specified project.
An error occurred. The project settings for data quality rules were not created or updated.
{"default_else_value":true,"ignore_trailing_spaces":false,"implicit_casting":true,"max_string_length":1024,"allow_quoted_variables":false,"project_rule_output":{"location":{"connection":{"id":"connection_id"},"catalog_name":"catalogName","schema_name":"schemName","table_name":"tableName"},"create_table_only_when_issues_are_found":false,"import_table_in_project":true}}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}
Request
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The start token of the resource from where the page should begin. If omitted, begins with first resource.
Possible values: 1 ≤ length ≤ 512
Example:
g1AAAAA-eJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqzmxiYWJiZGYGkOWDSyBJZAPCBD58The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Default:
200Example:
20If true include the children in the returned resource. If false, only return the resource without its eventual children.
Default:
falseThe type of the resource to search. If omitted, filtering on type is not applied
Possible values: 1 ≤ length ≤ 200
Response
A collection of data quality assets.
The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Example:
20Total number of resources available.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
100The link to a page in paginated collection.
An array of data quality assets.
Possible values: 1 ≤ number of items ≤ 200
The link to a page in paginated collection.
The link to a page in paginated collection.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the list of data quality assets.
An error occurred. The list of data quality assets cannot be returned.
{"total_count":1,"limit":200,"first":{"href":"https://cloud.ibm.com/data_quality/v4/assets?limit=200"},"assets":[{"id":"894d01fd-bdfc-4a4f-b68b-62751e06e06a","name":"CUSTOMERS","type":"data_asset","native_id":"8ddc659c-b155-498d-b310-4ac8cf629ccc/9a1f3644-3ad4-47be-8eb5-006d367c8702","weight":1,"creator_id":"SYSTEM","created_at":"2022-02-02T10:51:02.000Z"}]}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Request
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
Data quality asset to create.
Example: {"native_id":"8ddc659c-b155-498d-b310-4ac8cf629ccc/9a1f3644-3ad4-47be-8eb5-006d367c8702","name":"CUSTOMERS","type":"data_asset","weight":1}
Name of a data quality resource.(Name need not be unique)
Possible values: 1 ≤ length ≤ 200
Example:
CUSTOMERSUnique identifier set by the creator of the resource
Possible values: 1 ≤ length ≤ 500
Example:
4cdcd382-4e3a-4537-b7ae-09993acee4cf/6835c729-8d79-4b12-b02d-c82a20ac00a6The type of the data quality resource
Possible values: 1 ≤ length ≤ 200
Example:
data_assetThe weight of this asset in the computation of the data quality scores
The Id of an eventual parent, or null if this asset has no parent
- parent
The type of the IBM Knowledge Catalog asset
Possible values: 1 ≤ length ≤ 200
Example:
DataSetThe identity of the IBM Knowledge Catalog asset this DQ asset is referring to
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The identity of the IBM Knowledge Catalog container (either project/catalog/space) this asset belongs to.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5If specified, indicates that this asset is the virtualization of another asset whose id is specified in this field
- virtualized_asset
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223
Response
A data asset on which data quality is measured.
Name of a data quality resource.(Name need not be unique)
Possible values: 1 ≤ length ≤ 200
Example:
CUSTOMERSUnique identifier set by the creator of the resource
Possible values: 1 ≤ length ≤ 500
Example:
4cdcd382-4e3a-4537-b7ae-09993acee4cf/6835c729-8d79-4b12-b02d-c82a20ac00a6The type of the data quality resource
Possible values: 1 ≤ length ≤ 200
Example:
data_assetThe weight of this asset in the computation of the data quality scores
Possible values: 0 ≤ value ≤ 1
Eventual children of this asset. If omitted, no children for this asset.
Possible values: 0 ≤ number of items ≤ 100000
The Id of an eventual parent, or null if this asset has no parent
- parent
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
The type of the IBM Knowledge Catalog asset
Possible values: 1 ≤ length ≤ 200
Example:
DataSetThe identity of the IBM Knowledge Catalog asset this DQ asset is referring to
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The identity of the IBM Knowledge Catalog container (either project/catalog/space) this asset belongs to.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5If specified, indicates that this asset is the virtualization of another asset whose id is specified in this field
- virtualized_asset
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223
Status Code
Success
The given payload is invalid and the asset is not created.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to create a data quality asset.
The data quality asset exists already and cannot be created.
An error occurred. The data quality asset cannot be created.
{"id":"894d01fd-bdfc-4a4f-b68b-62751e06e06a","name":"CUSTOMERS","type":"data_asset","native_id":"8ddc659c-b155-498d-b310-4ac8cf629ccc/9a1f3644-3ad4-47be-8eb5-006d367c8702","weight":1,"creator_id":"SYSTEM","created_at":"2022-02-02T10:51:02.000Z"}{"trace":"6vbjp9d2t4s9b2ckyspdw4vft","status_code":400,"errors":[{"code":"missing_required_field","message":"Bad Request: {\"trace\":\"8f7b7693-1f08-4b4a-b2bd-5732d06ad19e\", \"errors\":[{\"code\":\"invalid_parameter\", \"message\":\"COMSV3006E: Missing or Invalid asset id\", \"target\":{\"name\":\"asset\",\"type\":\"parameter\"}}]}"}]}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"28izpw5l1a4orh823i6bkqmm","status_code":409,"errors":[{"code":"already_exists","message":"The dimension with name 'Completeness' already exists."}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Get a data quality asset
Get the data quality asset with the given identifier.
GET /data_quality/v4/assets/{id}Request
Path Parameters
The data quality asset identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
If true include the children in the returned resource. If false, only return the resource without its eventual children.
Default:
false
Response
A data asset on which data quality is measured.
Name of a data quality resource.(Name need not be unique)
Possible values: 1 ≤ length ≤ 200
Example:
CUSTOMERSUnique identifier set by the creator of the resource
Possible values: 1 ≤ length ≤ 500
Example:
4cdcd382-4e3a-4537-b7ae-09993acee4cf/6835c729-8d79-4b12-b02d-c82a20ac00a6The type of the data quality resource
Possible values: 1 ≤ length ≤ 200
Example:
data_assetThe weight of this asset in the computation of the data quality scores
Possible values: 0 ≤ value ≤ 1
Eventual children of this asset. If omitted, no children for this asset.
Possible values: 0 ≤ number of items ≤ 100000
The Id of an eventual parent, or null if this asset has no parent
- parent
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
The type of the IBM Knowledge Catalog asset
Possible values: 1 ≤ length ≤ 200
Example:
DataSetThe identity of the IBM Knowledge Catalog asset this DQ asset is referring to
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The identity of the IBM Knowledge Catalog container (either project/catalog/space) this asset belongs to.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5If specified, indicates that this asset is the virtualization of another asset whose id is specified in this field
- virtualized_asset
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the data quality asset.
The data quality asset with the given ID cannot be found.
An error occurred. The data quality asset cannot be returned.
{"id":"894d01fd-bdfc-4a4f-b68b-62751e06e06a","name":"CUSTOMERS","type":"data_asset","native_id":"8ddc659c-b155-498d-b310-4ac8cf629ccc/9a1f3644-3ad4-47be-8eb5-006d367c8702","weight":1,"creator_id":"SYSTEM","created_at":"2022-02-02T10:51:02.000Z"}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Delete data quality asset
Delete the data quality asset with the given identifier and all its children recursively
DELETE /data_quality/v4/assets/{id}Request
Path Parameters
The data quality asset identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
If true, delete all children recursively. Otherwise, only selected one is deleted.
Default:
false
Response
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to delete the data quality asset.
The data quality asset cannot be found.
An error occurred. The data quality asset cannot be deleted.
{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Update or patch a data quality asset
Update a data quality asset as specified in the payload details of the request.
The updates must be specified by using the JSON patch format, described in RFC 6902.
The following attributes can be updated: - /name (value can only be replaced) - /weight (value can only be replaced)
PATCH /data_quality/v4/assets/{id}Request
Path Parameters
The data quality asset identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The updates to make in the data quality rule.
Example: [{"op":"replace","path":"/description","value":"The proportion of data stored against the potential for 100% of the data."}]
The operation to be performed
Allowable values: [
add,remove,replace,move,copy,test]A JSON pointer to the field to update
A string containing a JSON pointer value
- value
Response
A data asset on which data quality is measured.
Name of a data quality resource.(Name need not be unique)
Possible values: 1 ≤ length ≤ 200
Example:
CUSTOMERSUnique identifier set by the creator of the resource
Possible values: 1 ≤ length ≤ 500
Example:
4cdcd382-4e3a-4537-b7ae-09993acee4cf/6835c729-8d79-4b12-b02d-c82a20ac00a6The type of the data quality resource
Possible values: 1 ≤ length ≤ 200
Example:
data_assetThe weight of this asset in the computation of the data quality scores
Possible values: 0 ≤ value ≤ 1
Eventual children of this asset. If omitted, no children for this asset.
Possible values: 0 ≤ number of items ≤ 100000
The Id of an eventual parent, or null if this asset has no parent
- parent
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
The type of the IBM Knowledge Catalog asset
Possible values: 1 ≤ length ≤ 200
Example:
DataSetThe identity of the IBM Knowledge Catalog asset this DQ asset is referring to
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The identity of the IBM Knowledge Catalog container (either project/catalog/space) this asset belongs to.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5If specified, indicates that this asset is the virtualization of another asset whose id is specified in this field
- virtualized_asset
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to update the data quality asset.
The data quality asset with the given ID cannot be found.
An error occurred. The data quality asset cannot be updated.
{"id":"894d01fd-bdfc-4a4f-b68b-62751e06e06a","name":"CUSTOMERS","type":"data_asset","native_id":"8ddc659c-b155-498d-b310-4ac8cf629ccc/9a1f3644-3ad4-47be-8eb5-006d367c8702","weight":1,"creator_id":"SYSTEM","created_at":"2022-02-02T10:51:02.000Z"}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Search data quality asset with given native id and type
Search for a data quality asset by its native id and type
POST /data_quality/v4/search_dq_asset
Request
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The native id, as known by the DQ producer, of the resource to search. If not provided, other identity parameters must be provided.
Possible values: 1 ≤ length ≤ 500
The type of the resource to search. If omitted, filtering on type is not applied
Possible values: 1 ≤ length ≤ 200
The identity of the IBM Knowledge Catalog asset this DQ asset is referring to
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5If true include the children in the returned resource. If false, only return the resource without its eventual children.
Default:
false
Response
A data asset on which data quality is measured.
Name of a data quality resource.(Name need not be unique)
Possible values: 1 ≤ length ≤ 200
Example:
CUSTOMERSUnique identifier set by the creator of the resource
Possible values: 1 ≤ length ≤ 500
Example:
4cdcd382-4e3a-4537-b7ae-09993acee4cf/6835c729-8d79-4b12-b02d-c82a20ac00a6The type of the data quality resource
Possible values: 1 ≤ length ≤ 200
Example:
data_assetThe weight of this asset in the computation of the data quality scores
Possible values: 0 ≤ value ≤ 1
Eventual children of this asset. If omitted, no children for this asset.
Possible values: 0 ≤ number of items ≤ 100000
The Id of an eventual parent, or null if this asset has no parent
- parent
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
The type of the IBM Knowledge Catalog asset
Possible values: 1 ≤ length ≤ 200
Example:
DataSetThe identity of the IBM Knowledge Catalog asset this DQ asset is referring to
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The identity of the IBM Knowledge Catalog container (either project/catalog/space) this asset belongs to.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5If specified, indicates that this asset is the virtualization of another asset whose id is specified in this field
- virtualized_asset
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to search data quality assets.
The data quality asset cannot be found.
An error occurred. The data quality asset cannot be searched.
{"id":"894d01fd-bdfc-4a4f-b68b-62751e06e06a","name":"CUSTOMERS","type":"data_asset","native_id":"8ddc659c-b155-498d-b310-4ac8cf629ccc/9a1f3644-3ad4-47be-8eb5-006d367c8702","weight":1,"creator_id":"SYSTEM","created_at":"2022-02-02T10:51:02.000Z"}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Request
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The asset native id, as known by the DQ producer, of the resource to search.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The data quality asset identifier
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The start token of the resource from where the page should begin. If omitted, begins with first resource.
Possible values: 1 ≤ length ≤ 512
Example:
g1AAAAA-eJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqzmxiYWJiZGYGkOWDSyBJZAPCBD58The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Default:
200Example:
20If true include the children in the returned resource. If false, only return the resource without its eventual children.
Default:
falseThe type of the resource to search. If omitted, filtering on type is not applied
Possible values: 1 ≤ length ≤ 200
Response
A collection of data quality checks.
The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Example:
20Total number of resources available.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
100The link to a page in paginated collection.
An array of data quality checks.
Possible values: 1 ≤ number of items ≤ 200
The link to a page in paginated collection.
The link to a page in paginated collection.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the list of data quality checks.
An error occurred. The list of data quality checks cannot be returned.
{"total_count":1,"limit":200,"first":{"href":"https://cloud.ibm.com/data_quality/v4/checks?limit=200"},"checks":[{"id":"894d01fd-bdfc-4a4f-b68b-62751e06e06a","name":"check_uniqueness_of_id","dimension":{"id":"371114cd-5516-4691-8b2e-1e66edf66486","name":"COMPLETENESS","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"},"type":"data_rule","native_id":"8ddc659c-b155-498d-b310-4ac8cf629ccc/9a1f3644-3ad4-47be-8eb5-006d367c8702","creator_id":"SYSTEM","created_at":"2022-02-02T10:51:02.000Z"}]}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Request
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
Data quality check to create.
Example: {"dimension":{"id":"371114cd-5516-4691-8b2e-1e66edf66486"},"name":"check_uniqueness_of_id","type":"data_rule","native_id":"8ddc659c-b155-498d-b310-4ac8cf629ccc/9a1f3644-3ad4-47be-8eb5-006d367c8702"}
the dimension this check belongs to
- dimension
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Name of a data quality resource.(Name need not be unique)
Possible values: 1 ≤ length ≤ 200
Example:
check_uniqueness_of_idUnique identifier set by the creator of the resource
Possible values: 1 ≤ length ≤ 500
Example:
4cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47The type of the data quality resource
Possible values: 1 ≤ length ≤ 200
Example:
data_ruleThe Id of an eventual parent, or null if this check has no parent
- parent
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223
Response
A data quality check, i.e. an action to measure data quality.
Name of a data quality resource.(Name need not be unique)
Possible values: 1 ≤ length ≤ 200
Example:
check_uniqueness_of_idUnique identifier set by the creator of the resource
Possible values: 1 ≤ length ≤ 500
Example:
4cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47The type of the data quality resource
Possible values: 1 ≤ length ≤ 200
Example:
data_ruleThe optional eventual children of this data quality check.
Possible values: 0 ≤ number of items ≤ 100000
An optional list of issues reported by this data quality check.
Possible values: 0 ≤ number of items ≤ 100000
- issues
the dimension this check belongs to
- dimension
Flag indicating whether the dimension is a built-in dimension or not.
The identity of the IBM Knowledge Catalog container (either project/catalog/space) this check belongs to.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The Id of an eventual parent, or null if this check has no parent
- parent
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223
Status Code
Success
The given payload is invalid and the check is not created.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to create a data quality check.
The data quality check exists already and cannot be created.
An error occurred. The data quality check cannot be created.
{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"check_uniqueness_of_id","dimension":{"id":"371114cd-5516-4691-8b2e-1e66edf66486","name":"COMPLETENESS","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"},"type":"data_rule","native_id":"4cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"}{"trace":"6vbjp9d2t4s9b2ckyspdw4vft","status_code":400,"errors":[{"code":"missing_required_field","message":"Bad Request: {\"trace\":\"8f7b7693-1f08-4b4a-b2bd-5732d06ad19e\", \"errors\":[{\"code\":\"invalid_parameter\", \"message\":\"COMSV3006E: Missing or Invalid asset id\", \"target\":{\"name\":\"asset\",\"type\":\"parameter\"}}]}"}]}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"28izpw5l1a4orh823i6bkqmm","status_code":409,"errors":[{"code":"already_exists","message":"The dimension with name 'Completeness' already exists."}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Get a data quality check
Get the data quality check with the given identifier.
GET /data_quality/v4/checks/{id}Request
Path Parameters
The data quality check identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
If true include the children in the returned resource. If false, only return the resource without its eventual children.
Default:
false
Response
A data quality check, i.e. an action to measure data quality.
Name of a data quality resource.(Name need not be unique)
Possible values: 1 ≤ length ≤ 200
Example:
check_uniqueness_of_idUnique identifier set by the creator of the resource
Possible values: 1 ≤ length ≤ 500
Example:
4cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47The type of the data quality resource
Possible values: 1 ≤ length ≤ 200
Example:
data_ruleThe optional eventual children of this data quality check.
Possible values: 0 ≤ number of items ≤ 100000
An optional list of issues reported by this data quality check.
Possible values: 0 ≤ number of items ≤ 100000
- issues
the dimension this check belongs to
- dimension
Flag indicating whether the dimension is a built-in dimension or not.
The identity of the IBM Knowledge Catalog container (either project/catalog/space) this check belongs to.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The Id of an eventual parent, or null if this check has no parent
- parent
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the data quality check.
The data quality check with the given ID cannot be found.
An error occurred. The data quality check cannot be returned.
{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"check_uniqueness_of_id","dimension":{"id":"371114cd-5516-4691-8b2e-1e66edf66486","name":"COMPLETENESS","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"},"type":"data_rule","native_id":"4cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Delete a data quality check
Delete the data quality check with the given identifier and all its children recursively
DELETE /data_quality/v4/checks/{id}Request
Path Parameters
The data quality check identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
If true, delete all children recursively. Otherwise, only selected one is deleted.
Default:
false
Response
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to delete the data quality check.
The data quality check cannot be found.
An error occurred. The data quality check cannot be deleted.
{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Update or patch a data quality check
Update a data quality check as specified in the payload details of the request.
The updates must be specified by using the JSON patch format, described in RFC 6902.
The following attributes can be updated: - /name (value can only be replace) - /dimension/id (value can be add, remove or replace) - /details (value can be add, remove or replace)
PATCH /data_quality/v4/checks/{id}Request
Path Parameters
The data quality check identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The updates to make in the data quality rule.
Example: [{"op":"replace","path":"/description","value":"The proportion of data stored against the potential for 100% of the data."}]
The operation to be performed
Allowable values: [
add,remove,replace,move,copy,test]A JSON pointer to the field to update
A string containing a JSON pointer value
- value
Response
A data quality check, i.e. an action to measure data quality.
Name of a data quality resource.(Name need not be unique)
Possible values: 1 ≤ length ≤ 200
Example:
check_uniqueness_of_idUnique identifier set by the creator of the resource
Possible values: 1 ≤ length ≤ 500
Example:
4cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47The type of the data quality resource
Possible values: 1 ≤ length ≤ 200
Example:
data_ruleThe optional eventual children of this data quality check.
Possible values: 0 ≤ number of items ≤ 100000
An optional list of issues reported by this data quality check.
Possible values: 0 ≤ number of items ≤ 100000
- issues
the dimension this check belongs to
- dimension
Flag indicating whether the dimension is a built-in dimension or not.
The identity of the IBM Knowledge Catalog container (either project/catalog/space) this check belongs to.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The Id of an eventual parent, or null if this check has no parent
- parent
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to update the data quality check.
The data quality check with the given ID cannot be found.
An error occurred. The data quality check cannot be updated.
{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"check_uniqueness_of_id","dimension":{"id":"371114cd-5516-4691-8b2e-1e66edf66486","name":"COMPLETENESS","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"},"type":"data_rule","native_id":"4cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Search for a data quality check by its native id and type
Search for a data quality check by its native id and type
POST /data_quality/v4/search_dq_check
Request
Query Parameters
The native id, as known by the DQ producer, of the resource to search.
Possible values: 1 ≤ length ≤ 500
The type of the resource to search.
Possible values: 1 ≤ length ≤ 200
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
If true include the children in the returned resource. If false, only return the resource without its eventual children.
Default:
false
Response
A data quality check, i.e. an action to measure data quality.
Name of a data quality resource.(Name need not be unique)
Possible values: 1 ≤ length ≤ 200
Example:
check_uniqueness_of_idUnique identifier set by the creator of the resource
Possible values: 1 ≤ length ≤ 500
Example:
4cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47The type of the data quality resource
Possible values: 1 ≤ length ≤ 200
Example:
data_ruleThe optional eventual children of this data quality check.
Possible values: 0 ≤ number of items ≤ 100000
An optional list of issues reported by this data quality check.
Possible values: 0 ≤ number of items ≤ 100000
- issues
the dimension this check belongs to
- dimension
Flag indicating whether the dimension is a built-in dimension or not.
The identity of the IBM Knowledge Catalog container (either project/catalog/space) this check belongs to.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The Id of an eventual parent, or null if this check has no parent
- parent
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to search data quality checks.
The data quality check cannot be found.
An error occurred. The data quality check cannot be searched.
{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"check_uniqueness_of_id","dimension":{"id":"371114cd-5516-4691-8b2e-1e66edf66486","name":"COMPLETENESS","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"},"type":"data_rule","native_id":"4cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Get a list of data quality dimensions
Get a list of data quality dimensions.
GET /data_quality/v4/dimensions
Request
Query Parameters
The start token of the resource from where the page should begin. If omitted, begins with first resource.
Possible values: 1 ≤ length ≤ 512
Example:
g1AAAAA-eJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqzmxiYWJiZGYGkOWDSyBJZAPCBD58The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Default:
200Example:
20
Response
A collection of data quality dimensions.
The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Example:
20Total number of resources available.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
100The link to a page in paginated collection.
An array of data quality dimensions.
Possible values: 1 ≤ number of items ≤ 200
The link to a page in paginated collection.
The link to a page in paginated collection.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the list of data quality dimensions.
An error occurred. The list of data quality dimensions cannot be returned.
{"total_count":1,"limit":200,"first":{"href":"https://cloud.ibm.com/data_quality/v4/dimensions?limit=200"},"dimensions":[{"id":"371114cd-5516-4691-8b2e-1e66edf66486","name":"Completeness","description":"The proportion of data stored against the potential for 100 percent.","is_default":false,"creator_id":"SYSTEM","created_at":"2022-02-01T18:11:02.000Z"}]}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Request
Data quality dimension to create.
Example: {"name":"Completeness10","description":"The proportion of data stored against the potential for 100 percent represented as integer 10."}
Name of a data quality resource.(Name need not be unique)
Possible values: 1 ≤ length ≤ 200
Example:
CompletenessThe optional description of the dimension. Provide a description to understand the reason for its existence.
Possible values: 0 ≤ length ≤ 255
Example:
The proportion of data stored against the potential for 100 percent. If this property is omitted, no description is set.
Response
Flag indicating whether the dimension is a built-in dimension or not.
Status Code
Success
The given payload is invalid and the dimension is not created.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to create a data quality dimension.
The data quality dimension exists already and cannot be created.
An error occurred. The data quality dimension cannot be created.
{"id":"371114cd-5516-4691-8b2e-1e66edf66486","name":"Completeness","description":"The proportion of data stored against the potential for 100 percent.","is_default":false,"creator_id":"SYSTEM","created_at":"2022-02-01T18:11:02.000Z"}{"trace":"6vbjp9d2t4s9b2ckyspdw4vft","status_code":400,"errors":[{"code":"missing_required_field","message":"Bad Request: {\"trace\":\"8f7b7693-1f08-4b4a-b2bd-5732d06ad19e\", \"errors\":[{\"code\":\"invalid_parameter\", \"message\":\"COMSV3006E: Missing or Invalid asset id\", \"target\":{\"name\":\"asset\",\"type\":\"parameter\"}}]}"}]}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"28izpw5l1a4orh823i6bkqmm","status_code":409,"errors":[{"code":"already_exists","message":"The dimension with name 'Completeness' already exists."}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Get the data quality dimension with the given identifier
Get the data quality dimension with the given identifier.
GET /data_quality/v4/dimensions/{id}Request
Path Parameters
The data quality dimension identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Response
Flag indicating whether the dimension is a built-in dimension or not.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the data quality dimension.
The data quality dimension with the given ID cannot be found.
An error occurred. The data quality dimension cannot be returned.
{"id":"371114cd-5516-4691-8b2e-1e66edf66486","name":"Completeness","description":"The proportion of data stored against the potential for 100 percent.","is_default":false,"creator_id":"SYSTEM","created_at":"2022-02-01T18:11:02.000Z"}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Delete the data quality dimension with the given identifier.
Delete the data quality dimension with the given identifier.
All data quality checks associated to this dimension will point to no dimension. All DQ scores stored for this dimension will be deleted
DELETE /data_quality/v4/dimensions/{id}Request
Path Parameters
The data quality dimension identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Response
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to delete the data quality dimension.
The data quality dimension cannot be found.
An error occurred. The data quality dimension cannot be deleted.
{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Update a data quality dimension
Update a data quality dimension as specified in the payload details of the request.
The updates must be specified by using the JSON patch format, described in RFC 6902.
The following attributes can be updated: - /name (value can only be replaced) - /description (value can be added, removed, or replaced)
PATCH /data_quality/v4/dimensions/{id}Request
Path Parameters
The data quality dimension identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
The updates to make in the data quality rule.
Example: [{"op":"replace","path":"/description","value":"The proportion of data stored against the potential for 100% of the data."}]
The operation to be performed
Allowable values: [
add,remove,replace,move,copy,test]A JSON pointer to the field to update
A string containing a JSON pointer value
- value
Response
Flag indicating whether the dimension is a built-in dimension or not.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to update the data quality dimension.
The data quality dimension with the given ID cannot be found.
An error occurred. The data quality dimension cannot be updated.
{"id":"371114cd-5516-4691-8b2e-1e66edf66486","name":"Completeness","description":"The proportion of data stored against the potential for 100 percent.","is_default":false,"creator_id":"SYSTEM","created_at":"2022-02-01T18:11:02.000Z"}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Search for data quality dimensions by name or part of their names
Search for data quality dimensions by name or part of their names
POST /data_quality/v4/search_dq_dimension
Request
Query Parameters
The name of the resource to search or a part of the name of the resources to search.
Possible values: 1 ≤ length ≤ 200
Response
A collection of data quality dimensions.
The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Example:
20Total number of resources available.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
100The link to a page in paginated collection.
An array of data quality dimensions.
Possible values: 1 ≤ number of items ≤ 200
The link to a page in paginated collection.
The link to a page in paginated collection.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to search data quality dimensions.
An error occurred. The data quality dimension cannot be searched.
{"total_count":1,"limit":200,"first":{"href":"https://cloud.ibm.com/data_quality/v4/dimensions?limit=200"},"dimensions":[{"id":"371114cd-5516-4691-8b2e-1e66edf66486","name":"Completeness","description":"The proportion of data stored against the potential for 100 percent.","is_default":false,"creator_id":"SYSTEM","created_at":"2022-02-01T18:11:02.000Z"}]}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Request
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The start token of the resource from where the page should begin. If omitted, begins with first resource.
Possible values: 1 ≤ length ≤ 512
Example:
g1AAAAA-eJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqzmxiYWJiZGYGkOWDSyBJZAPCBD58The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Default:
200Example:
20Comma-separated list of data quality asset identifier to search issues of. If not provided, filtering on asset is not applied.
Possible values: 1 ≤ length ≤ 10000
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5,b1ba1d22-71a7-4adf-99b2-3c8ba1949710Comma-separated list of data quality check id identifier to search issues of. If not provided, filtering on a check is not applied.
Possible values: 1 ≤ length ≤ 10000
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5,b1ba1d22-71a7-4adf-99b2-3c8ba1949710Comma-separated list of data quality check native id identifier to search issues of. If not provided, filtering on a check is not applied. Only one of checkId or checkNativeId should be provided; if both are provided, an error will be thrown.
Possible values: 1 ≤ length ≤ 10000
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5,b1ba1d22-71a7-4adf-99b2-3c8ba1949710The name or part of the name of data quality check identifier to search issues of. If not provided, filtering on name of check is not applied.
Possible values: 1 ≤ length ≤ 200
Comma-separated list of data quality dimension identifiers.
Possible values: 1 ≤ length ≤ 10000
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5,b1ba1d22-71a7-4adf-99b2-3c8ba1949710If true, returns only the latest issue summary of each DQ check. If false, returns the latest and archived issues.
Default:
trueThe type of the resource to search. If omitted, filtering on type is not applied
Possible values: 1 ≤ length ≤ 200
If true include the children in the returned resource. If false, only return the resource without its eventual children.
Default:
trueThe issue attribute to sort on. If omitted, sorting on last update time is applied
Allowable values: [
check_name,check_type,check_dimension_name,percent_occurrences,ignored,updated_at]Default:
updated_atExample:
check_nameThe direction to sort with. If omitted, sorting attribute descending is applied
Allowable values: [
asc,desc]Default:
descExample:
asc
Response
A collection of data quality issues.
The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Example:
20Total number of resources available.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
100The link to a page in paginated collection.
An array of data quality issues.
Possible values: 1 ≤ number of items ≤ 200
The link to a page in paginated collection.
The link to a page in paginated collection.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the list of data quality issues.
An error occurred. The list of data quality issues cannot be returned.
{"total_count":1,"limit":200,"first":{"href":"https://cloud.ibm.com/data_quality/v4/issues?limit=200"},"issues":[{"id":"046605b5-48d9-489e-b846-8ef96a7a1aba","check":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"MY_RULE","native_id":"cdcd4382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47","type":"ANY","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"},"reported_for":{"id":"894d01fd-bdfc-4a4f-b68b-62751e06e06a","nativeId":"739e06f8-3gk2-4c4u-8ab7-e5682g06f05a"},"number_of_occurrences":123,"number_of_tested_records":456789,"creator_id":"SYSTEM","created_at":"2022-02-02T10:51:02.000Z"}]}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Request
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
Data quality issue to create.
Example: {"check":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b"},"reported_for":{"id":"894d01fd-bdfc-4a4f-b68b-62751e06e06a"},"number_of_occurrences":123,"number_of_tested_records":456789,"status":"actual","ignored":false}
the check that produces this issue
- check
The Id of the asset on which this issue was found
- reported_for
Flag indicating whether the issue is ignored or not.
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223The absolute number of occurrences of the issues which have been identified on the asset. If this is missing, percent_occurrences must be provided.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
123The total number of records which have been tested by the Data Quality Check. If this is missing, percent_occurrences must be provided.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
456789The relative frequency of the issue (percentage of tested records having the issue), as a number between 0.0 and 1.0. If omitted, it is calculated using number_of_occurrences and number_of_tested_records.
Possible values: 0 ≤ value ≤ 1
The status of the issue.
Allowable values: [
actual,aggregation]Default:
actualExample:
actual
Response
A data quality issue, i.e. a data quality problem detected during a check.
the check that produces this issue
- check
Name of a data quality resource.(Name need not be unique)
Possible values: 1 ≤ length ≤ 200
Example:
check_uniqueness_of_idUnique identifier set by the creator of the resource
Possible values: 1 ≤ length ≤ 500
Example:
4cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47The type of the data quality resource
Possible values: 1 ≤ length ≤ 200
Example:
data_ruleThe optional eventual children of this data quality check.
Possible values: 0 ≤ number of items ≤ 100000
An optional list of issues reported by this data quality check.
Possible values: 0 ≤ number of items ≤ 100000
- issues
the dimension this check belongs to
- dimension
Flag indicating whether the dimension is a built-in dimension or not.
The identity of the IBM Knowledge Catalog container (either project/catalog/space) this check belongs to.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The Id of an eventual parent, or null if this check has no parent
- parent
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223
The Id of the asset on which this issue was found
- reported_for
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Flag indicating whether the issue is ignored or not.
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223The absolute number of occurrences of the issues which have been identified on the asset. If this is missing, percent_occurrences must be provided.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
123The total number of records which have been tested by the Data Quality Check. If this is missing, percent_occurrences must be provided.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
456789The relative frequency of the issue (percentage of tested records having the issue), as a number between 0.0 and 1.0. If omitted, it is calculated using number_of_occurrences and number_of_tested_records.
Possible values: 0 ≤ value ≤ 1
The status of the issue.
Possible values: [
actual,aggregation,archive]Example:
actual
Status Code
Success
The given payload is invalid and the issue is not created.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to create a data quality issue.
The data quality issue exists already and cannot be created.
An error occurred. The data quality issue cannot be created.
{"id":"046605b5-48d9-489e-b846-8ef96a7a1aba","check":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"MY_RULE","native_id":"cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47","type":"ANY","created_at":"2022-02-02T14:00:45.000Z","creator_id":"SYSTEM"},"reported_for":{"id":"894d01fd-bdfc-4a4f-b68b-62751e06e06a"},"number_of_occurrences":123,"number_of_tested_records":456789,"status":"actual","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"}{"trace":"6vbjp9d2t4s9b2ckyspdw4vft","status_code":400,"errors":[{"code":"missing_required_field","message":"Bad Request: {\"trace\":\"8f7b7693-1f08-4b4a-b2bd-5732d06ad19e\", \"errors\":[{\"code\":\"invalid_parameter\", \"message\":\"COMSV3006E: Missing or Invalid asset id\", \"target\":{\"name\":\"asset\",\"type\":\"parameter\"}}]}"}]}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"28izpw5l1a4orh823i6bkqmm","status_code":409,"errors":[{"code":"already_exists","message":"The dimension with name 'Completeness' already exists."}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Get the data quality issue with the given identifier
Get the data quality issue with the given identifier.
GET /data_quality/v4/issues/{id}Request
Path Parameters
The data quality issue identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
Response
A data quality issue, i.e. a data quality problem detected during a check.
the check that produces this issue
- check
Name of a data quality resource.(Name need not be unique)
Possible values: 1 ≤ length ≤ 200
Example:
check_uniqueness_of_idUnique identifier set by the creator of the resource
Possible values: 1 ≤ length ≤ 500
Example:
4cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47The type of the data quality resource
Possible values: 1 ≤ length ≤ 200
Example:
data_ruleThe optional eventual children of this data quality check.
Possible values: 0 ≤ number of items ≤ 100000
An optional list of issues reported by this data quality check.
Possible values: 0 ≤ number of items ≤ 100000
- issues
the dimension this check belongs to
- dimension
Flag indicating whether the dimension is a built-in dimension or not.
The identity of the IBM Knowledge Catalog container (either project/catalog/space) this check belongs to.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The Id of an eventual parent, or null if this check has no parent
- parent
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223
The Id of the asset on which this issue was found
- reported_for
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Flag indicating whether the issue is ignored or not.
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223The absolute number of occurrences of the issues which have been identified on the asset. If this is missing, percent_occurrences must be provided.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
123The total number of records which have been tested by the Data Quality Check. If this is missing, percent_occurrences must be provided.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
456789The relative frequency of the issue (percentage of tested records having the issue), as a number between 0.0 and 1.0. If omitted, it is calculated using number_of_occurrences and number_of_tested_records.
Possible values: 0 ≤ value ≤ 1
The status of the issue.
Possible values: [
actual,aggregation,archive]Example:
actual
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the data quality issue.
The data quality issue with the given ID cannot be found.
An error occurred. The data quality issue cannot be returned.
{"id":"046605b5-48d9-489e-b846-8ef96a7a1aba","check":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"MY_RULE","native_id":"cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47","type":"ANY","created_at":"2022-02-02T14:00:45.000Z","creator_id":"SYSTEM"},"reported_for":{"id":"894d01fd-bdfc-4a4f-b68b-62751e06e06a"},"number_of_occurrences":123,"number_of_tested_records":456789,"status":"actual","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Update a data quality issue as specified in the payload details of the request.
Update a data quality issue as specified in the payload details of the request.
The updates must be specified by using the JSON patch format, described in RFC 6902.
The following attributes can be updated: - /ignored (value can only be replaced)
PATCH /data_quality/v4/issues/{id}Request
Path Parameters
The data quality issue identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The updates to make in the data quality rule.
Example: [{"op":"replace","path":"/description","value":"The proportion of data stored against the potential for 100% of the data."}]
The operation to be performed
Allowable values: [
add,remove,replace,move,copy,test]A JSON pointer to the field to update
A string containing a JSON pointer value
- value
Response
A data quality issue, i.e. a data quality problem detected during a check.
the check that produces this issue
- check
Name of a data quality resource.(Name need not be unique)
Possible values: 1 ≤ length ≤ 200
Example:
check_uniqueness_of_idUnique identifier set by the creator of the resource
Possible values: 1 ≤ length ≤ 500
Example:
4cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47The type of the data quality resource
Possible values: 1 ≤ length ≤ 200
Example:
data_ruleThe optional eventual children of this data quality check.
Possible values: 0 ≤ number of items ≤ 100000
An optional list of issues reported by this data quality check.
Possible values: 0 ≤ number of items ≤ 100000
- issues
the dimension this check belongs to
- dimension
Flag indicating whether the dimension is a built-in dimension or not.
The identity of the IBM Knowledge Catalog container (either project/catalog/space) this check belongs to.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The Id of an eventual parent, or null if this check has no parent
- parent
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223
The Id of the asset on which this issue was found
- reported_for
Resource identifier.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5
Flag indicating whether the issue is ignored or not.
Link to corresponding data quality component. No default link is available, it is empty if not provided.
Possible values: 1 ≤ length ≤ 500
Example:
/v2/assets/f0d42bb9-55b5-4ec8-aca5-f77e42978a86?project_id=08716e38-1a1b-4e0e-9533-c2b419063223The absolute number of occurrences of the issues which have been identified on the asset. If this is missing, percent_occurrences must be provided.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
123The total number of records which have been tested by the Data Quality Check. If this is missing, percent_occurrences must be provided.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
456789The relative frequency of the issue (percentage of tested records having the issue), as a number between 0.0 and 1.0. If omitted, it is calculated using number_of_occurrences and number_of_tested_records.
Possible values: 0 ≤ value ≤ 1
The status of the issue.
Possible values: [
actual,aggregation,archive]Example:
actual
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to update the data quality issue.
The data quality issue with the given ID cannot be found.
An error occurred. The data quality issue cannot be updated.
{"id":"046605b5-48d9-489e-b846-8ef96a7a1aba","check":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"MY_RULE","native_id":"cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47","type":"ANY","created_at":"2022-02-02T14:00:45.000Z","creator_id":"SYSTEM"},"reported_for":{"id":"894d01fd-bdfc-4a4f-b68b-62751e06e06a"},"number_of_occurrences":123,"number_of_tested_records":456789,"status":"actual","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Search for a data quality issue by its native id
Search for a data quality issue by its native id
POST /data_quality/v4/search_dq_issue
Request
Query Parameters
The data quality asset identifier
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The data quality check identifier to search issues of.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
Response
A data quality issue, i.e. a data quality problem detected during a check on a given asset.
An array of previous data quality issues detected during the same check on the given asset.
Possible values: 0 ≤ number of items ≤ 100
An array of data quality issues detected during the same check on children of the given asset.
Possible values: 0 ≤ number of items ≤ 10000
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to search data quality issues.
The data quality issue cannot be found.
An error occurred. The data quality issue cannot be searched.
{"id":"046605b5-48d9-489e-b846-8ef96a7a1aba","check":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"MY_RULE","native_id":"cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47","type":"ANY","created_at":"2022-02-02T14:00:45.000Z","creator_id":"SYSTEM"},"reported_for":{"id":"894d01fd-bdfc-4a4f-b68b-62751e06e06a"},"number_of_occurrences":123,"number_of_tested_records":456789,"status":"actual","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z","archived_issues":[{"id":"046605b5-48d9-489e-b846-8ef96a7a1aba","check":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"MY_RULE","native_id":"cdcd382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47","type":"ANY","created_at":"2022-02-02T14:00:45.000Z","creator_id":"SYSTEM"},"reported_for":{"id":"894d01fd-bdfc-4a4f-b68b-62751e06e06a"},"number_of_occurrences":123,"number_of_tested_records":456789,"status":"archive","creator_id":"SYSTEM","created_at":"2022-02-01T14:00:45.000Z"}]}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cfkvi16tv9bimp2rqve666na0","status_code":404,"errors":[{"code":"not_found","message":"Requested resource was not found.","more_info":"https://www.ibm.com/docs/en/cloud-paks/cp-data/4.5.x?topic=rules-creating-data","target":{"type":"field","name":"name"}}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Create several data quality issues
Create several data quality issues. The related DQ assets referenced by their native ID are created if they do not already exist.
POST /data_quality/v4/create_issues
Request
Query Parameters
The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
If true, the assets will be refreshed, and any assets not present in the updated list will be deleted from the database. If false, no action will be taken.
Default:
false
Data quality issues to create and their related DQ assets.
Example: {"issues":[{"check":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b"},"reported_for":{"native_id":"8ddc659c-b155-498d-b310-4ac8cf629ccc/9a1f3644-3ad4-47be-8eb5-006d367c8702","type":"profile"},"number_of_occurrences":123,"number_of_tested_records":456789,"ignored":false}],"assets":[{"name":"CUSTOMERS","type":"data_asset","native_id":"8ddc659c-b155-498d-b310-4ac8cf629ccc/9a1f3644-3ad4-47be-8eb5-006d367c8702","weight":1}],"existing_checks":[{"id":"6be18374-573a-4cf8-8ab7-e428506e428b"}]}
list of issues to create.
Possible values: 0 ≤ number of items ≤ 100000
Optional list of assets to create. For each of these assets the system will check if an asset with the same native_id exists. If yes, it will ignore it, if not it will create it
Possible values: 0 ≤ number of items ≤ 100000
Optional list of existing checks. For each pair of (asset, check) given, system will create, update or remove an issue depending on the current issue stored for this pair and the new issue given in the payload (or not for deletion).
Possible values: 0 ≤ number of items ≤ 100000
- existing_checks
Response
A collection of data quality issues.
The maximum number of resources to return.
Possible values: 1 ≤ value ≤ 200
Example:
20Total number of resources available.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
100The link to a page in paginated collection.
An array of data quality issues.
Possible values: 1 ≤ number of items ≤ 200
The link to a page in paginated collection.
The link to a page in paginated collection.
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to create data quality issues.
An error occurred. The data quality issues cannot be created.
{"total_count":1,"limit":200,"first":{"href":"https://cloud.ibm.com/data_quality/v4/issues?limit=200"},"issues":[{"id":"046605b5-48d9-489e-b846-8ef96a7a1aba","check":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"MY_RULE","native_id":"cdcd4382-4e3a-4537-b7ae-09993acee4cf/3e51167c-6eb2-4069-96dc-5d6df808fd47","type":"ANY","creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"},"reported_for":{"id":"894d01fd-bdfc-4a4f-b68b-62751e06e06a","nativeId":"739e06f8-3gk2-4c4u-8ab7-e5682g06f05a"},"number_of_occurrences":123,"number_of_tested_records":456789,"creator_id":"SYSTEM","created_at":"2022-02-02T10:51:02.000Z"}]}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
Get a list of data quality scores for a given asset
Get a list of data quality scores for a given asset.
GET /data_quality/v4/scores
Request
Query Parameters
The data quality asset identifier
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5The ID of the catalog to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The ID of the project to use. A catalog_id or project_id is required.
Possible values: 1 ≤ length ≤ 100
The data quality dimension identifier. If not provided, filtering on dimension is not applied.
Possible values: 1 ≤ length ≤ 128
Example:
b1ba1d22-71a7-4adf-99b2-3c8ba19497f5If true include the children in the returned resource. If false, only return the resource without its eventual children.
Default:
falseQuery for scores older than the given number of days
Possible values: 0 ≤ value ≤ 10000
Default:
0
Response
A collection of data quality scores.
Total number of scores available.
Possible values: 0 ≤ value ≤ 9007199254740991
Example:
100Information of asset.
An array of data quality scores.
Possible values: 1 ≤ number of items ≤ 200
An array of data quality scores.
Possible values: 1 ≤ number of items ≤ 200
Status Code
Success.
Your authorization to access this method is missing, invalid, or expired.
You do not have permission to get the list of data quality scores.
An error occurred. The list of data quality scores cannot be returned.
{"total_count":100,"asset":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"CUSTOMERS"},"scores":[{"status":"actual","score":0.8,"number_of_checks":1,"timestamp":"2022-02-14T18:11:02Z","dimension_scores":[{"dimension":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"COMPLETENESS","description":"Dimension related to completeness","is_default":false,"creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"},"score":0.4,"timestamp":"2022-02-14T18:11:02Z"},{"dimension":{"id":"7be18374-573a-4cf8-8ab7-e428506e428b","name":"ACCURACY","description":"Dimension related to accuracy","is_default":false,"creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"},"score":0.7,"timestamp":"2022-02-13T18:11:02Z"}]},{"status":"archive","score":0.6,"number_of_checks":2,"timestamp":"2022-02-12T18:11:02Z","dimension_scores":[{"dimension":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"COMPLETENESS","description":"Dimension related to completeness","is_default":false,"creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"},"score":0.23,"timestamp":"2022-02-12T18:11:02Z"}]},{"status":"archive","score":0.7,"number_of_checks":2,"timestamp":"2022-02-11T18:11:02Z","dimension_scores":[{"dimension":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"COMPLETENESS","description":"Dimension related to completeness","is_default":false,"creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"},"score":0.23,"timestamp":"2022-02-11T18:11:02Z"}]}],"children":[{"asset":{"id":"5be18374-573a-4cf8-8ab7-e428506e428b","name":"ID"},"scores":[{"status":"actual","score":0.8,"number_of_checks":1,"timestamp":"2022-02-14T18:11:02Z","dimension_scores":[{"dimension":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"COMPLETENESS","description":"Dimension related to completeness","is_default":false,"creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"},"score":0.4,"timestamp":"2022-02-14T18:11:02Z"}]},{"status":"archive","score":0.6,"number_of_checks":1,"timestamp":"2022-02-12T18:11:02Z","dimension_scores":[{"dimension":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"COMPLETENESS","description":"Dimension related to completeness","is_default":false,"creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"},"score":0.23,"timestamp":"2022-02-12T18:11:02Z"}]},{"status":"archive","score":0.7,"number_of_checks":1,"timestamp":"2022-02-11T18:11:02Z","dimension_scores":[{"dimension":{"id":"6be18374-573a-4cf8-8ab7-e428506e428b","name":"COMPLETENESS","description":"Dimension related to completeness","is_default":false,"creator_id":"SYSTEM","created_at":"2022-02-02T14:00:45.000Z"},"score":0.23,"timestamp":"2022-02-11T18:11:02Z"}]}]}]}{"trace":"3tirwt88tsejqwh801zw88lek","status_code":403,"errors":[{"code":"not_authorized","message":"User IBMid-341983Z157 with role null is not authorized to perform the operation CREATE."}]}{"trace":"cwuhtyfus0m8f6e49e2g6m4hr","status_code":500,"errors":[{"code":"internal_error","message":"An IOException was caught when accessing asset 'cbf21aa7-cb89-4ee8-9817-0f1d86fbe0cd'."}]}
retrieve a governanceType
Retrieves detailed information on a governance type. This includes all of the operations defined for the governance type and the allowed and default outcomes of each operation.
GET /v3/enforcement/governance_types/{governance_type_name}Response
Response for the /v3/enforcement/governance_types/{governanceTypeId} API
metadata object for policy API responses that have no metadata
entity object within the response for the /v3/enforcement/governance_types/{governanceTypeId} API
Status Code
OK
Bad Request
Unauthorized
Forbidden
Internal Server Error
No Sample Response
create user defined left hand side term
Create a left hand side term to be used in rules defined in the policy engine.
POST /v3/enforcement/lhs/udp
Request
LHS RuleTerm JSON
Name of the custom LHS.
Display name of the custom LHS.
Property name of the LHS references. For Asset properties, use the path of the property in asset json. For user property, use the name of the mapped User SCIM attribute. (Custom user predicates are not supported in CPDaaS).
Source for the LHS reference. For custom asset predicate, use 'ASSET'. For custom user predicate use 'CONTEXTUAL' (Custom user predicates are not supported in CPDaaS).
Allowable values: [
ASSET, CONTEXTUAL]Description of the custom LHS.
Response
Response for the /v3/enforcement/udp/{termName} API
metadata object used in responses returned from policy management related APIs
entity object within the response for the /v3/enforcement/udp/{termName} API
Status Code
OK
Bad Request
Unauthorized
Forbidden
Internal Server Error
No Sample Response
get a user defined left hand side term
Get the user defined left hand side term in the term cache of the policy engine.
GET /v3/enforcement/lhs/udp/{lhsTermId}Response
Response for the /v3/enforcement/udp/{termName} API
metadata object used in responses returned from policy management related APIs
entity object within the response for the /v3/enforcement/udp/{termName} API
Status Code
OK
Bad Request
Unauthorized
Forbidden
Internal Server Error
No Sample Response
delete a user defined left hand side term
Delete a left hand side term in the term cache of the policy engine. This cache is maintained for performance and is used for evaluating policies. It is refreshed periodically from technical terms maintained in the glossary.
DELETE /v3/enforcement/lhs/udp/{lhsTermId}retrieve and aggregate policy related metrics
This API retrieves policy enforcement metrics based on query parameters, which include metric type, date range, policy, user, rule, governanceType, outcome, host type, cached, context operation, context location, asset type and asset location. It then sums these metrics according to a specified aggregation type.
GET /v3/enforcement/policy_metrics/analysis/3d
Request
Query Parameters
The type of metrics to return
Allowable values: [
enforcements,denials,operational_policies,operational_rules]The type of aggregation to perform on the selected metrics (days, months, years, policies, rules, users, governanceTypes, outcomes)
Allowable values: [
days,months,years,policies,rules,users,outcomes,governance_type,pep_host_types,is_pep_cache,context_operations,context_locations,asset_types,asset_locations]The type of aggregation to perform on the selected metrics (days, months, years, policies, rules, users, governanceTypes, outcomes)
Allowable values: [
days,months,years,policies,rules,users,outcomes,governance_type,pep_host_types,is_pep_cache,context_operations,context_locations,asset_types,asset_locations]ISO 8601 date/time specifying the starting time to return metrics data
ISO 8601 date/time specifying the ending time to return metrics data
The policy to return metrics of, or all policies if not specified
The rule to return metrics of, or all rules if not specified
The user to return metrics about, or all users if not specified
The governanceType to return metrics about, or all governanceTypes if not specified
Allowable values: [
Access,Classification,Curation,Lifecycle,ResourceControl,DMR,AIGovernance,DLR]The enforcement outcome to return metrics about, or all outcomes if not specified
The PEP host type to return metrics about, or all PEP host types if not specified
Metrics on decisions retrieved from cache versus from server, or both if not specified
The context operation to return metrics about, or all context operations if not specified
The context location to return metrics about, or all context locations if not specified
The asset type to return metrics about, or all asset types if not specified
The asset location to return metrics about, or all asset locations if not specified
The order to sort the metrics. The following values are allowed:
- aggregate, -aggregate -- ascending or descending order by the aggregate
- count, -count -- ascending or descending order by enforcement count
retrieve and aggregate policy related metrics
This API retrieves policy enforcement metrics based on query parameters, which include metric type, date range, policy, user, rule, governanceType, outcome, host type, cached, context operation, context location, asset type and asset location. It then sums these metrics according to a specified aggregation type.
GET /v3/enforcement/policy_metrics
Request
Query Parameters
The type of metrics to return
Allowable values: [
enforcements,denials,operational_policies,operational_rules]The type of aggregation to perform on the selected metrics (days, months, years, policies, rules, users, governanceTypes, outcomes)
Allowable values: [
days,months,years,policies,rules,users,outcomes,governance_type,pep_host_types,is_pep_cache,context_operations,context_locations,asset_types,asset_locations]ISO 8601 date/time specifying the starting time to return metrics data
ISO 8601 date/time specifying the ending time to return metrics data
The policy to return metrics of, or all policies if not specified
The rule to return metrics of, or all rules if not specified
The user to return metrics about, or all users if not specified
The governanceType to return metrics about, or all governanceTypes if not specified
Allowable values: [
Access,Classification,Curation,Lifecycle,ResourceControl,DMR,AIGovernance,DLR]The enforcement outcome to return metrics about, or all outcomes if not specified
The PEP host type to return metrics about, or all PEP host types if not specified
Metrics on decisions retrieved from cache versus from server, or both if not specified
The context operation to return metrics about, or all context operations if not specified
The context location to return metrics about, or all context locations if not specified
The asset type to return metrics about, or all asset types if not specified
The asset location to return metrics about, or all asset locations if not specified
The order to sort the metrics. The following values are allowed:
- aggregate, -aggregate -- ascending or descending order by the aggregate
- count, -count -- ascending or descending order by enforcement count
create a custom role
Creates a cusotm role. This will include a list of existing permissions to be associated with the role. Maximum length allowed for 'name' parameter: 80 characters, maximum length allowed for 'description' parameter: 1000 characters. Allowed characters for the 'name' parameter: letters from any language, numbers in any script, space, dot, underscore, hyphen. Strings with characters other than these are rejected (only for the name parameter).
POST /v3/enforcement/roles
update a custom category role
Updates a custom category role. This API is also used to add and remove permissions from the custom category role. The permissions should be passed as an array of string that has permissions ids.Maximum length allowed for 'name' parameter: 80 characters, maximum length allowed for 'description' parameter: 1000 characters. If the parameter 'name' is modified, allowed characters for the 'name' parameter: letters from any language, numbers in any script, space, dot, underscore, hyphen. Strings with characters other than these are rejected (only for the name parameter).
PUT /v3/enforcement/roles/{role_id}Sync a rule
Syncs a rule in DPS with its global search index. If rule is found in DPS, its global search index is updated. If rule is not found in DPS, its global search index is removed.
PUT /v3/enforcement/rules/sync/{rule_id}Response
Response for the /v3/enforcement/rules/{ruleId} API
metadata object used in responses returned from policy management related APIs
entity within the response for the /v3/enforcement/rules/{ruleId} API
Status Code
OK
Bad Request
Unauthorized
Forbidden
Not Found
Internal Server Error
No Sample Response
retrieve a rule
Retrieves detailed information on a rule given the rule's identifier.
GET /v3/enforcement/rules/{rule_id}Response
Response for the /v3/enforcement/rules/{ruleId} API
metadata object used in responses returned from policy management related APIs
entity within the response for the /v3/enforcement/rules/{ruleId} API
Status Code
OK
Bad Request
Unauthorized
Forbidden
Not Found
Internal Server Error
No Sample Response
update a rule
Updates a rule. Maximum length allowed for 'name' parameter: 80 characters, maximum length allowed for 'description' parameter: 1000 characters. If the parameter 'name' is modified, allowed characters for the 'name' parameter: letters from any language, numbers in any script, space, dot, underscore, hyphen. Strings with characters other than these are rejected (only for the name parameter). The governance_type_id cannot be modified.
PUT /v3/enforcement/rules/{rule_id}Request
Path Parameters
Rule ID
Rule json
the displayed name for the rule
the name of a governance type of this rule
nested arrays representing an expression that is evaluated to determine if the action on the rule should be performed or enforced. See the description on the POST /v3/enforcement/rules for more details.
the action to perform or enforce when the rule triggers
a more detailed description of the rule
the state of the rule. Only the rules of policies in active are enforced.
Allowable values: [
draft,active,archived]
Response
Response for the /v3/enforcement/rules/{ruleId} API
metadata object used in responses returned from policy management related APIs
entity within the response for the /v3/enforcement/rules/{ruleId} API
Status Code
OK
Bad Request
Unauthorized
Forbidden
Not Found
Internal Server Error
No Sample Response
list all rules
Lists all defined rules. This includes all rules associated with policies and rules not associated with any policy. When more than one filter criteria is specified, the resulting collection satisfies all the criteria.
GET /v3/enforcement/rules
Request
Query Parameters
Specify name of the rule to search for or use filter of the form 'contains:xx' to search for rules containing provided phrase as part of name or use filter of the form 'exact:xx' to search for rules with exact name.
Specify description of the rule to search for or use filter of the form 'contains:xx' to search for rules containing provided phrase as part of description.
If specified, only rules with a matching trigger expression will be returned.
If specified, only rules with a matching action will be returned.
If specified, only rules in the matching state will be returned.
Allowable values: [
draft,active,archived]If speficied, only rules with the matching governance type/types will be returned.
The order to sort the rules. The following values are allowed:
- name, -name -- ascending or descending order by the name
- modified_date, -modified_date -- ascending or descending order by modified date
Allowable values: [
name,-name,modified_date,-modified_date]The maximum number of Rules to return. The default value is 50.
Default:
50The index of the first matching Rule to include in the result. The default value is 0.
Default:
0
Response
Response for the /v3/enforcement/rules API
the limit as specified on the query or defaulted
the offset as specified on the query or the default value 0
total number of resources in the collection
link to the previous page of resources, if available
link to the previous page of resources, if available
link to the previous page of resources, if available
link to the previous page of resources, if available
list of rules
Status Code
OK
Bad Request
Unauthorized
Forbidden
Internal Server Error
No Sample Response
create a rule
Creates a rule. A rule has two key properties:
- a trigger defining when the rule should be enforced, and
- an action describing what operations to perform or outcome to enforce.
Trigger
A trigger is a boolean expression represented using nested arrays. The following describes the syntax:
Expression:
[ -conditions- ]
Conditions:
-predicate-
"NOT", -predicate-
-predicate-, "AND"|"OR", -conditions-
"NOT", -predicate-, "AND"|"OR", -conditions-
Predicate:
[ "$-term-", "EXISTS" ]
[ "$-term-", "EQUALS"|"LESS_THAN"|"GREATER_THAN"|"CONTAINS", "#-literal-"|"$-term-" ]
-expression-
where:
- -term- is a technical term defined in the term glossary.
- -literal- is a literal. For numerics a string representation of the number should be specified. For times, milliseconds are used (from Unix epoch). For boolean, #true and #false are used.
The definition of the operators in a predicate:
- EXISTS -- means that the term has a value of some kind.
- EQUALS -- evaluates to true if the left and right sides are equal.
- LESS_THAN -- evaluates to true if the left is less in numeric value than the right.
- GREATER_THAN -- evaluates to true if the left is greater in numeric value than the right.
- CONTAINS -- is meant to test an array term (such as Asset.Tags) with a single value. It evaluates to true if the value on the right side equals one of the values on the array on the left side.
However it will also supports a single value on the left, in which case it behaves just like EQUALS -- regular expressions or wildcards are not supported.
For all of the operators (except EXISTS), if the right hand side evaluates to an array, each value of the array is compared to the left side, according to the operator definition, and if any comparison is true then the result of the evaluation is true.
Examples:
[["$Asset.Type", "EQUALS", "#Project"]]
["NOT", ["$Asset.Tags", "CONTAINS", "#sensitive"], "AND", ["NOT", "$Asset.Tags", "CONTAINS", "#confidential"]]
[["$User.Name", "EQUALS", "$Asset.Owner"]]
Action
The action is an name with optional parameters or subaction describing an operation to perform. For simple actions like Deny, only the action name will be provided. For actions such as Transform, a subaction with parameters describing the type of transform is also provided. The allowed actions depends on the governance type.
Examples:
{"name": "Deny"}
{"name": "Transform", "subaction": {"name": "anonymizeColumns", "parameters": [{"name": "column_name", "value": "CCN"}]}}
The maximum length allowed for 'name' parameter is 80 characters, maximum length allowed for 'description' parameter: 1000 characters. Allowed characters for the 'name' parameter: letters from any language, numbers in any script, space, dot, underscore, hyphen. Strings with characters other than these are rejected (only for the name parameter).
POST /v3/enforcement/rules
Request
Rule json
the displayed name for the rule
the name of a governance type of this rule
nested arrays representing an expression that is evaluated to determine if the action on the rule should be performed or enforced. See the description on the POST /v3/enforcement/rules for more details.
the action to perform or enforce when the rule triggers
a more detailed description of the rule
the state of the rule. Only the rules of policies in active are enforced.
Allowable values: [
draft,active,archived]
Response
Response for the /v3/enforcement/rules/{ruleId} API
metadata object used in responses returned from policy management related APIs
entity within the response for the /v3/enforcement/rules/{ruleId} API
Status Code
Merged with an existing 'ResourceControl' rule.
Created
Bad Request
Unauthorized
Forbidden
Internal Server Error
No Sample Response
export all rules
Export all defined rules as JSON.This includes all rules associated with policies and rules not associated with any policy.
GET /v3/enforcement/rules/export
Request
Query Parameters
The maximum number of Rules to return. The default value is 50.
Default:
50The index of the first matching Rule to include in the result. The default value is 0.
Default:
0For export between two different systems. If specified as true, the ids in rules will be converted to names and added as export_components in the exported file
Default:
false
import rules
Imports rules using content recieved as application/octet-stream media type. Assumes content to be in UTF-8 charset. Expects "rules" JSON array contained in a JSON objects. Ignores other fields. Rule GUID is prerserved. Retruns JSON array containing GUIDs of newly imported rules.
POST /v3/enforcement/rules/import
Request
Query Parameters
For import between two different systems. If specified as true, the names in export_components of the import file will be matched and converted to the target system's ids.
Default:
falseFor external imports, if specified as true, the whole import will fail if there are any rules that fail to match ids in the target system. If specified as false, an analysis report will be returned.
Default:
false
The input stream for reading imported terms
Response
Response for the /v3/enforcement/rules/export API and /v3/enforcement/rules/import
List containing details of rules that are successfully imported.
List containing details of rules that are failed to import.
List containing details of rules that are successfully matched.
Status Code
OK
Bad Request
Unauthorized
Forbidden
Content cannot be more than 1MB
Internal Server Error
No Sample Response
retrieve tenant settings
Retrieves governance information and settings about a particular tenant given the tenant identifier.
GET /v3/enforcement/settings
Response
Response for the /v3/enforcement/settings/{tenantId} API
metadata object used in responses returned from policy management related APIs
entity object within the response for the /v3/enforcement/settings/{tenantId} API
Status Code
OK
Bad Request
Unauthorized
Forbidden
Not Found
Internal Server Error
No Sample Response
update tenant settings
Updates governance information and settings for the given tenant.
PUT /v3/enforcement/settings
Request
governedTenant json
Data protection rules convention. Choose DEAA for 'Locked' convention and AEAD for 'Unlocked' convention.
Allowable values: [
DEAA, AEAD]Oldest - by creation date, Latest - by modified date, columns, Merge - merge annotations (terms and tags) of all aliases
Allowable values: [
Oldest, Latest, Merge]The rule action precedence for data protection rules. The actions, in order of leniency, are: Allow access, Mask and Filter, Deny access.
Allowable values: [
RESTRICTIVE, PERMISSIVE, HIERARCHICAL]The masking method precedence for data protection rules. The masking methods, in privacy order, are: Redact, Substitute, Obfuscate. The masking methods, in utility order, are: Obfuscate, Substitute, Redact.
Allowable values: [
RESTRICTIVE, PERMISSIVE]enable/disable business glossary terms inheritance for DPR/DLR enforcement
enable/disable dataclass terms inheritance for DPR/DLR enforcement
enable/disable classification terms inheritance for DPR/DLR enforcement
enable/disable business glossary term 1 level synonym relationship for DPR/DLR enforcement
Enable evaluation based on asset's lineage
This flag indicates if transform deep enforcement process using 'advanced' or 'classic'. The default value is advanced.
Allowable values: [
advanced, classic]Maximum combined number of DPR and DLR rules supported.
Enable/disable audit log for evaluation
Allowable values: [
false]
Response
Response for the /v3/enforcement/settings/{tenantId} API
metadata object used in responses returned from policy management related APIs
entity object within the response for the /v3/enforcement/settings/{tenantId} API
Status Code
OK
Bad Request
Unauthorized
Forbidden
Not Found
Internal Server Error
No Sample Response
Create a metadata enrichment area asset
Creates a new metadata enrichment asset and the corresponding job definition. If a data scope is provided the decoration of data scope assets will be started in background and the response will contain the execution ID of that background process. After background processing has finished a RabbitMQ event is sent to CAMS topic exchange CatalogServiceMessageHub with topic v2.metadata_enrichment.{mdeAreaId}.create_enrichment_area and message body as described in model RabbitMqNotificationMessage. If no data scope is provided an empty enrichment area is created without starting a background decoration task. In that case the field asyncExecutionInfo will be missing in the response. If query paramter enrichImmediate is true, which is the default setting, and the data scope is not empty a new enrichment job run will be triggered automatically.
POST /v2/metadata_enrichment/metadata_enrichment_area
Request
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
should enrichment be run immediately after area creation
Default:
true
Object containing information for creating a metadata enrichment area asset
name of the metadata enrichment asset
Example:
Sample enrichment- objective
- enrichment_options
- structured
true if profile is enabled for structured data. Required when creating and optional when updating.
true if assign terms is enabled for structured data. Required when creating and optional when updating.
true if analyze quality is enabled for structured data. Required when creating and optional when updating.
- unstructured
always true as profile is always enabled for unstructured data
true if assign terms is enabled for unstructured data
Required when creating and optional when updating.
- sampling
- structured
Set true to apply project level default settings to this metadata enrichment area.
Sampling method for this metadata enrichment area. The value is respected when project_default_settings is false. Possible values: [TOP, top, RANDOM, random]
Example:
topAnalysis method to use when sampling data. Either sample a
fixednumber of rows or a givenpercentageof all table rows. The value is respected when project_default_settings is false. Providing this attribute is optional, if not specified, the analysis method defaults tofixed. Possible values: [FIXED, fixed, PERCENTAGE, percentage]Example:
fixed- sample_size
Allowable values: [
BASIC,COMPREHENSIVE,CUSTOM,MODERATE,BASIC, MODERATE, COMPREHENSIVE, CUSTOM]- options
Enrich top # of rows / table.
Example:
100Classify based on # values / column.
Example:
10
- percentage_options
Percentage of rows to enrich, as decimal
Example:
0.8Minimum number of rows to include in sampling
Example:
100Maximum number of rows to include in sampling
Example:
100Classify based on # values/columns.
Example:
10
- unstructured
sampling method
Allowable values: [
RANDOM]- sample_size
Allowable values: [
BASIC,COMPREHENSIVE,MODERATE,BASIC, MODERATE, COMPREHENSIVE]
Data scope of reruns. Possible values: [DELTA, delta, ALL, all]
Example:
delta
description of the metadata enrichment area asset
Example:
This is a description for my sample metadata enrichment.Id of the catalog to metadata enrichment area assets into
Example:
342b07da-fbe5-4a0a-94ee-ab89696975d5- job
Name of the metadata enrichment job
Example:
My Enrichment JobA cron string defining when the job should be run. If an empty string is provided it means the job is not scheduled to run.
Example:
0 3 21 13 1 ? 2019- schedule_info
Indicate a repeated job
Example:
trueA timestamp in epoch time, the scheduled job will be triggered after this timestamp.
Example:
1547578689512A timestamp in epoch time, the scheduled job will be triggered before this timestamp.
Example:
1547578689512
- publish_job
Id of the metadata publish job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Name of the MDE area publish job
Example:
My MDE area publish JobA cron string defining when the job should be run. This is going to be always empty as publish is not scheduled to run.
- data_scope
IDs of assets to enrich metadata
- container_assets
list of tags
Response
An object containing information about a metadata enrichment area asset
- metadata
The metadata enrichment area id assigned by the system. Read-only
Example:
4412e9b2-3661-11e7-a919-92ebcb67fe33The type of the asset
Example:
metadata_enrichment_areaThe IAM ID of the user that created the asset
Example:
IBMid-550000FRA0Timestamp representing creation datetime of metadata enrichment area asset read-only
Example:
1631089384058The ID of the project which contains the asset.
Example:
487084de-de8d-4981-9205-952732a90b3d- usage
Example:
2019-01-15T18:58:09ZExample:
IBMid-550000FRA0Example:
1547578689512Example:
2019-01-15T18:58:09ZExample:
1547578689512Example:
IBMid-550000FRA0
list of tags
- entity
name of the metadata enrichment area asset
Example:
my_enrichmentdescription of the metadata enrichment area asset
Example:
Enrichment of assetsId of the catalog to metadata enrichment area assets into
Example:
342b07da-fbe5-4a0a-94ee-ab89696975d5- job
Id of the metadata enrichment job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Name of the metadata enrichment job
Example:
My Enrichment JobA cron string defining when the job should be run. If an empty string is provided it means the job is not scheduled to run.
Example:
0 3 21 13 1 ? 2019- schedule_info
Indicate a repeated job
Example:
trueA timestamp in epoch time, the scheduled job will be triggered after this timestamp.
Example:
1547578689512A timestamp in epoch time, the scheduled job will be triggered before this timestamp.
Example:
1547578689512
- data_asset_summary
- structured
Number of structured data assets
Example:
1203
- unstructured
Number of unstructured data assets
Example:
890
- objective
- enrichment_options
- structured
true if profile is enabled for structured data. Required when creating and optional when updating.
true if assign terms is enabled for structured data. Required when creating and optional when updating.
true if analyze quality is enabled for structured data. Required when creating and optional when updating.
- unstructured
always true as profile is always enabled for unstructured data
true if assign terms is enabled for unstructured data
Required when creating and optional when updating.
- sampling
- structured
Set true to apply project level default settings to this metadata enrichment area.
Sampling method for this metadata enrichment area. The value is respected when project_default_settings is false. Possible values: [TOP, top, RANDOM, random]
Example:
topAnalysis method to use when sampling data. Either sample a
fixednumber of rows or a givenpercentageof all table rows. The value is respected when project_default_settings is false. Providing this attribute is optional, if not specified, the analysis method defaults tofixed. Possible values: [FIXED, fixed, PERCENTAGE, percentage]Example:
fixed- sample_size
Possible values: [
BASIC,COMPREHENSIVE,CUSTOM,MODERATE,BASIC, MODERATE, COMPREHENSIVE, CUSTOM]- options
Enrich top # of rows / table.
Example:
100Classify based on # values / column.
Example:
10
- percentage_options
Percentage of rows to enrich, as decimal
Example:
0.8Minimum number of rows to include in sampling
Example:
100Maximum number of rows to include in sampling
Example:
100Classify based on # values/columns.
Example:
10
- unstructured
sampling method
Possible values: [
RANDOM]- sample_size
Possible values: [
BASIC,COMPREHENSIVE,MODERATE,BASIC, MODERATE, COMPREHENSIVE]
Data scope of reruns. Possible values: [DELTA, delta, ALL, all]
Example:
delta
The status of the MDE area. Possible values: [READY, ready, UPDATING, updating, DELETING, deleting, DELETION_PENDING, deletion_pending, FAILED, failed]
Example:
ready- publish_job
Id of the metadata publish job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Name of the MDE area publish job
Example:
My MDE area publish JobA cron string defining when the job should be run. This is going to be always empty as publish is not scheduled to run.
- data_scope_processing
Example:
1631089384058If not null, retries after this interval in millisecond.
Example:
8000Error messages while setting data scope
Status Code
Success
Unauthorized
Forbidden
Error
No Sample Response
Retrieve metadata enrichment settings
Retrieve metadata enrichment settings
GET /v2/metadata_enrichment/metadata_enrichment_area/settings
Create or update metadata enrichment settings
Create or update metadata enrichment settings
PUT /v2/metadata_enrichment/metadata_enrichment_area/settings
Retrieve details of a Metadata Enrichment Area asset
Retrieve details of a metadata enrichment area asset
GET /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}Request
Path Parameters
Id of the metadata enrichment area asset.
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
whether to include data asset count summary in the response
Default:
false
Response
An object containing information about a metadata enrichment area asset
- metadata
The metadata enrichment area id assigned by the system. Read-only
Example:
4412e9b2-3661-11e7-a919-92ebcb67fe33The type of the asset
Example:
metadata_enrichment_areaThe IAM ID of the user that created the asset
Example:
IBMid-550000FRA0Timestamp representing creation datetime of metadata enrichment area asset read-only
Example:
1631089384058The ID of the project which contains the asset.
Example:
487084de-de8d-4981-9205-952732a90b3d- usage
Example:
2019-01-15T18:58:09ZExample:
IBMid-550000FRA0Example:
1547578689512Example:
2019-01-15T18:58:09ZExample:
1547578689512Example:
IBMid-550000FRA0
list of tags
- entity
name of the metadata enrichment area asset
Example:
my_enrichmentdescription of the metadata enrichment area asset
Example:
Enrichment of assetsId of the catalog to metadata enrichment area assets into
Example:
342b07da-fbe5-4a0a-94ee-ab89696975d5- job
Id of the metadata enrichment job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Name of the metadata enrichment job
Example:
My Enrichment JobA cron string defining when the job should be run. If an empty string is provided it means the job is not scheduled to run.
Example:
0 3 21 13 1 ? 2019- schedule_info
Indicate a repeated job
Example:
trueA timestamp in epoch time, the scheduled job will be triggered after this timestamp.
Example:
1547578689512A timestamp in epoch time, the scheduled job will be triggered before this timestamp.
Example:
1547578689512
- data_asset_summary
- structured
Number of structured data assets
Example:
1203
- unstructured
Number of unstructured data assets
Example:
890
- objective
- enrichment_options
- structured
true if profile is enabled for structured data. Required when creating and optional when updating.
true if assign terms is enabled for structured data. Required when creating and optional when updating.
true if analyze quality is enabled for structured data. Required when creating and optional when updating.
- unstructured
always true as profile is always enabled for unstructured data
true if assign terms is enabled for unstructured data
Required when creating and optional when updating.
- sampling
- structured
Set true to apply project level default settings to this metadata enrichment area.
Sampling method for this metadata enrichment area. The value is respected when project_default_settings is false. Possible values: [TOP, top, RANDOM, random]
Example:
topAnalysis method to use when sampling data. Either sample a
fixednumber of rows or a givenpercentageof all table rows. The value is respected when project_default_settings is false. Providing this attribute is optional, if not specified, the analysis method defaults tofixed. Possible values: [FIXED, fixed, PERCENTAGE, percentage]Example:
fixed- sample_size
Possible values: [
BASIC,COMPREHENSIVE,CUSTOM,MODERATE,BASIC, MODERATE, COMPREHENSIVE, CUSTOM]- options
Enrich top # of rows / table.
Example:
100Classify based on # values / column.
Example:
10
- percentage_options
Percentage of rows to enrich, as decimal
Example:
0.8Minimum number of rows to include in sampling
Example:
100Maximum number of rows to include in sampling
Example:
100Classify based on # values/columns.
Example:
10
- unstructured
sampling method
Possible values: [
RANDOM]- sample_size
Possible values: [
BASIC,COMPREHENSIVE,MODERATE,BASIC, MODERATE, COMPREHENSIVE]
Data scope of reruns. Possible values: [DELTA, delta, ALL, all]
Example:
delta
The status of the MDE area. Possible values: [READY, ready, UPDATING, updating, DELETING, deleting, DELETION_PENDING, deletion_pending, FAILED, failed]
Example:
ready- publish_job
Id of the metadata publish job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Name of the MDE area publish job
Example:
My MDE area publish JobA cron string defining when the job should be run. This is going to be always empty as publish is not scheduled to run.
- data_scope_processing
Example:
1631089384058If not null, retries after this interval in millisecond.
Example:
8000Error messages while setting data scope
Status Code
Success
Unauthorized
Forbidden
Not found
Gone
Error
No Sample Response
Delete a metadata enrichment asset
Delete a metadata enrichment area asset
DELETE /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}edit a metadata enrichment area asset
Edit a metadata enrichment area asset
PATCH /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}Request
Path Parameters
Id of the metadata enrichment area asset.
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
Force the operation even when the metadata enrichment area is being updated.
Object carrying the update request.
name of the metadata enrichment area asset
Example:
Sample enrichmentdescription of the metadata enrichment area asset
Example:
This is a description for my sample metadata enrichment.Id of the catalog to metadata enrichment area assets into
Example:
342b07da-fbe5-4a0a-94ee-ab89696975d5- job
Name of the metadata enrichment job
Example:
My Enrichment JobA cron string defining when the job should be run. If an empty string is provided it means the job is not scheduled to run.
Example:
0 3 21 13 1 ? 2019- schedule_info
Indicate a repeated job
Example:
trueA timestamp in epoch time, the scheduled job will be triggered after this timestamp.
Example:
1547578689512A timestamp in epoch time, the scheduled job will be triggered before this timestamp.
Example:
1547578689512
- publish_job
Id of the metadata publish job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Name of the MDE area publish job
Example:
My MDE area publish JobA cron string defining when the job should be run. This is going to be always empty as publish is not scheduled to run.
- objective
- enrichment_options
- structured
true if profile is enabled for structured data. Required when creating and optional when updating.
true if assign terms is enabled for structured data. Required when creating and optional when updating.
true if analyze quality is enabled for structured data. Required when creating and optional when updating.
- unstructured
always true as profile is always enabled for unstructured data
true if assign terms is enabled for unstructured data
Required when creating and optional when updating.
- sampling
- structured
Set true to apply project level default settings to this metadata enrichment area.
Sampling method for this metadata enrichment area. The value is respected when project_default_settings is false. Possible values: [TOP, top, RANDOM, random]
Example:
topAnalysis method to use when sampling data. Either sample a
fixednumber of rows or a givenpercentageof all table rows. The value is respected when project_default_settings is false. Providing this attribute is optional, if not specified, the analysis method defaults tofixed. Possible values: [FIXED, fixed, PERCENTAGE, percentage]Example:
fixed- sample_size
Allowable values: [
BASIC,COMPREHENSIVE,CUSTOM,MODERATE,BASIC, MODERATE, COMPREHENSIVE, CUSTOM]- options
Enrich top # of rows / table.
Example:
100Classify based on # values / column.
Example:
10
- percentage_options
Percentage of rows to enrich, as decimal
Example:
0.8Minimum number of rows to include in sampling
Example:
100Maximum number of rows to include in sampling
Example:
100Classify based on # values/columns.
Example:
10
- unstructured
sampling method
Allowable values: [
RANDOM]- sample_size
Allowable values: [
BASIC,COMPREHENSIVE,MODERATE,BASIC, MODERATE, COMPREHENSIVE]
Data scope of reruns. Possible values: [DELTA, delta, ALL, all]
Example:
delta
list of tags
Response
An object containing information about a metadata enrichment area asset
- metadata
The metadata enrichment area id assigned by the system. Read-only
Example:
4412e9b2-3661-11e7-a919-92ebcb67fe33The type of the asset
Example:
metadata_enrichment_areaThe IAM ID of the user that created the asset
Example:
IBMid-550000FRA0Timestamp representing creation datetime of metadata enrichment area asset read-only
Example:
1631089384058The ID of the project which contains the asset.
Example:
487084de-de8d-4981-9205-952732a90b3d- usage
Example:
2019-01-15T18:58:09ZExample:
IBMid-550000FRA0Example:
1547578689512Example:
2019-01-15T18:58:09ZExample:
1547578689512Example:
IBMid-550000FRA0
list of tags
- entity
name of the metadata enrichment area asset
Example:
my_enrichmentdescription of the metadata enrichment area asset
Example:
Enrichment of assetsId of the catalog to metadata enrichment area assets into
Example:
342b07da-fbe5-4a0a-94ee-ab89696975d5- job
Id of the metadata enrichment job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Name of the metadata enrichment job
Example:
My Enrichment JobA cron string defining when the job should be run. If an empty string is provided it means the job is not scheduled to run.
Example:
0 3 21 13 1 ? 2019- schedule_info
Indicate a repeated job
Example:
trueA timestamp in epoch time, the scheduled job will be triggered after this timestamp.
Example:
1547578689512A timestamp in epoch time, the scheduled job will be triggered before this timestamp.
Example:
1547578689512
- data_asset_summary
- structured
Number of structured data assets
Example:
1203
- unstructured
Number of unstructured data assets
Example:
890
- objective
- enrichment_options
- structured
true if profile is enabled for structured data. Required when creating and optional when updating.
true if assign terms is enabled for structured data. Required when creating and optional when updating.
true if analyze quality is enabled for structured data. Required when creating and optional when updating.
- unstructured
always true as profile is always enabled for unstructured data
true if assign terms is enabled for unstructured data
Required when creating and optional when updating.
- sampling
- structured
Set true to apply project level default settings to this metadata enrichment area.
Sampling method for this metadata enrichment area. The value is respected when project_default_settings is false. Possible values: [TOP, top, RANDOM, random]
Example:
topAnalysis method to use when sampling data. Either sample a
fixednumber of rows or a givenpercentageof all table rows. The value is respected when project_default_settings is false. Providing this attribute is optional, if not specified, the analysis method defaults tofixed. Possible values: [FIXED, fixed, PERCENTAGE, percentage]Example:
fixed- sample_size
Possible values: [
BASIC,COMPREHENSIVE,CUSTOM,MODERATE,BASIC, MODERATE, COMPREHENSIVE, CUSTOM]- options
Enrich top # of rows / table.
Example:
100Classify based on # values / column.
Example:
10
- percentage_options
Percentage of rows to enrich, as decimal
Example:
0.8Minimum number of rows to include in sampling
Example:
100Maximum number of rows to include in sampling
Example:
100Classify based on # values/columns.
Example:
10
- unstructured
sampling method
Possible values: [
RANDOM]- sample_size
Possible values: [
BASIC,COMPREHENSIVE,MODERATE,BASIC, MODERATE, COMPREHENSIVE]
Data scope of reruns. Possible values: [DELTA, delta, ALL, all]
Example:
delta
The status of the MDE area. Possible values: [READY, ready, UPDATING, updating, DELETING, deleting, DELETION_PENDING, deletion_pending, FAILED, failed]
Example:
ready- publish_job
Id of the metadata publish job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Name of the MDE area publish job
Example:
My MDE area publish JobA cron string defining when the job should be run. This is going to be always empty as publish is not scheduled to run.
- data_scope_processing
Example:
1631089384058If not null, retries after this interval in millisecond.
Example:
8000Error messages while setting data scope
Status Code
Success
Unauthorized
Forbidden
Not found
Gone
Error
No Sample Response
Add or remove assets from data scope
Add or remove assets from data scope
PATCH /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}/data_scopeRequest
Path Parameters
Id of the metadata enrichment area asset.
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
Force the operation even when the metadata enrichment area is being updated.
Object carrying the update request.
operation to perform
Allowable values: [
ADD,REMOVE,add, remove]Asset type. e.g. data_asset, metadata_import
Example:
data_assetID of the asset to add to or remove from the metadata enrichment area
Example:
30bc8c7a-ef65-4383-b9c6-d304bc191e26
Assign business terms to data assets
Add terms to given data assets. This api is executed synchronously. It verifies that all assets are part of the given MDE area. If validation fails method ends with an error.
POST /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}/data_scope/enrichment_assets/assign_termsRequest
Path Parameters
Id of the metadata enrichment area asset.
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
whether to process the request asynchronously
Default:
false
List of data assets and terms to assign
IDs of assets to enrich metadata
Assign business terms to columns of data assets
Add terms to given asset columns. This api is executed synchronously. It verifies that all assets are part of the given MDE area. If validation fails method ends with an error.
POST /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}/data_scope/enrichment_assets/columns/assign_termsRequest
Path Parameters
Id of the metadata enrichment area asset.
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
whether to process the request asynchronously
Default:
false
List of columns of data assets and terms to assign
Assign or remove a data class to columns of data assets
Assign or remove data class to columns of data assets
POST /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}/data_scope/enrichment_assets/columns/dataclassRequest
Path Parameters
Id of the metadata enrichment area asset.
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
whether to process the request asynchronously
Default:
false
List of columns of data assets and id of data class and operation to be performed
Example:
b5568a7e-7fca-4824-8466-f18119096e81
Remove business terms from columns of data assets.
Removes either all terms or selected terms from given asset columns. When terms are provided with the payload only the given terms are removed from the given columns. If no terms are provided all terms are removed.
POST /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}/data_scope/enrichment_assets/columns/remove_termsRequest
Path Parameters
Id of the metadata enrichment area asset.
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
whether to process the request asynchronously
Default:
false
List of columns of data assets and terms to assign
Adds or updates the review timestamp for given columns.
Adds/updates the 'reviewed' time stamp to/of the given asset columns.This api is executed synchronously. It fetches the assets that belong to the given columns via CAMS bulk get and verifies that all given assets are part of the given MDE area and are ready for review which means that either profiling or term assignment must be finished. It also validates that the columns exist on the asset. If validation fails the method returns an error. Otherwise for all given columns the review timestamp gets updated via CAMS bulk patch. Returns the new review timestamp.
POST /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}/data_scope/enrichment_assets/columns/reviewRequest
Path Parameters
Id of the metadata enrichment area asset.
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
Array containing list of asset Ids with columns.
asset_id
Example:
75985844-aa3f-410f-82be-9788ffd6ec75columns of an asset
Removes review timestamp for given columns.
Removes the 'reviewed' time stamp from the given asset columns. This api is executed synchronously. It fetches alle assets that belong to the given columns via CAMS bulk get and verifies that all given assets are part of the given MDE area. Assets that have no review data are skipped. It also validates that the columns exist on the asset. If validation fails the method returns an error. Otherwise for all given columns the review timestamp will be removed via CAMS bulk patch.
POST /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}/data_scope/enrichment_assets/columns/unreviewRequest
Path Parameters
Id of the metadata enrichment area asset.
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
Array containing list of asset Ids with columns.
asset_id
Example:
75985844-aa3f-410f-82be-9788ffd6ec75columns of an asset
Enrich selected data assets
Enrich selected data assets
POST /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}/data_scope/enrichment_assets/enrichRemove business terms from data assets
Removes either all terms or selected terms from given list of data assets. When terms are provided with the payload only the given terms are removed from the given assets. If no terms are provided all terms are removed.
POST /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}/data_scope/enrichment_assets/remove_termsRequest
Path Parameters
Id of the metadata enrichment area asset.
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
whether to process the request asynchronously
Default:
false
List of data assets and terms to remove
IDs of assets to enrich metadata
Review a given list of data assets.
Adds/updates the 'reviewed' time stamp to/of given data assets. This api is executed synchronously. It fetches the given assets via CAMS bulk get and verifies that all given assets are part of the given MDE area and are ready for review which means that either profiling or term assignment must be finished. If validation fails the method returns an error. Otherwise the review timestamp of all assets gets updated via CAMS bulk patch. Returns the new review timestamp.
POST /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}/data_scope/enrichment_assets/reviewRemoves review timestamp from the given assets
Removes the 'reviewed' time stamp from the given data assets. This api is executed synchronously. It fetches the given assets via CAMS bulk get and verifies that all given assets are part of the given MDE area. Assets that have no review data are skipped. If validation fails the method returns an error. Otherwise the review timestamp is removed from of all assets via CAMS bulk patch.
POST /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}/data_scope/enrichment_assets/unreviewPublish all assets or a subset of assets of a Metadata Enrichment Area to a catalog
Publish assets asynchronously. If publishScope is 'selected_assets' either /asset_ids or /filter/search_criteria has to be specified. If publisScope is 'all_assets' none of both must be specified.
POST /v2/metadata_enrichment/metadata_enrichment_area/{metadata_enrichment_area_id}/publish_assetsRequest
Path Parameters
Id of the metadata enrichment area asset.
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
tells if all assets in mde area should be published or only selected assets
Allowable values: [
all_assets,selected_assets]
Publish assets to a catalog
catalog Id
Example:
ae568a7e-7fca-4824-8466-f18119096e82action if asset already exists
Allowable values: [
IGNORE,REPLACE,UPDATE,replace, update, ignore]Example:
updateIDs of assets to enrich metadata
- filter
type of search
Allowable values: [
CATALOG_SEARCH,catalog_search]Asset type for catalog search.
Example:
metadata_enrichment_area_infoJSON object for search payload
Examples:{ "query": "asset.name:Asset* AND metadata_enrichment_area_info.area_id:62df1c42-53cd-4503-a6f5-f020aaabca72" }- search_criteria
Request
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
If this value is set to true, a Job is created together. The relation is also done between mdetadata import asset. The response message includes entity.jobId. The job name is generated from the name of metadata import with a unique suffix.
Default:
falseA candidate of Job name. The job is created with a specified name. If the name already exists, an error is returned. The create_job parameter should be set to true.
Object containing information for creating a metadata import asset
name of the metadata import asset
Example:
db2_tables_importlabel to identify new architecture
- datasource_definition_info
- metadata_ingestion_info
- extraction_info
Example:
-Direct -Agent
List of the assets to be update with SAP Business Metdata
description of the metadata import asset
Example:
Import of a db2 tableThe type of the import. The possible values are 'metadata' and 'lineage'.
Allowable values: [
LINEAGE,LINEAGE_AND_DISCOVERY,METADATA,TERMGENERATION,TERMGENERATION_AND_METADATA,metadata, lineage, lineage_and_discovery, termgeneration_and_metadata, termgeneration]Id of the connection which is being discovered
Example:
96e2149d-6916-427b-bf98-a0cb989306d7List of connection_id which is being discovered
Examples:[ "69196df7-714f-416b-a49b-ea025cd5d5e7", "224a78fe-7946-4e10-b685-af7b5f047d0a" ]List of configuration/datasource files for which metadata/lineage has to be extracted.
Examples:[ { "asset_id": "dde62b91-ae75-4d7d-aa08-e93065520aa4", "technology": "erwin" } ]- file_configuration
List of workspace rootPath.
Examples:[ { "id": "dde62b91-ae75-4d7d-aa08-e93065520aa4", "path": "/Workspace/export_uc", "name": "Databricks_connection", "type": "Connection" } ]asset ref map when trying to export and import a project .
specify if name, description, column descriptions to be udpated during re-import.
Examples:[ { "update_name": false, "update_description": false, "update_column_descriptions": false, "delete_when_deleted_at_source": false, "delete_when_removed_from_scope": false } ]- business_extraction
- primary_category
- dictionary_connection
List of the data assets to be updated with Business Metadata
support advanced options of metadata import.
Examples:[ { "exclude_tables": false, "exclude_views": false, "import_incremental_changes_only": false, "include_foreign_key": false, "include_primary_key": false } ]Advanced settings for lineage
Examples:[ { "extract_extended_attributes": false, "performance_profile": "Complete" } ]Id of the catalog to import assets into
Example:
342b07da-fbe5-4a0a-94ee-ab89696975d5Id of the project to import assets into
Example:
c5b3daa2-c880-4f83-9bfa-c50ff70d9f42Id of the metadata import job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Id of the metadata import job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33- scope
list of paths
Examples:[ "/schema1/table1" ]include list in path
Examples:[ "/table2", "/table3" ]exclude list in path
Examples:[ "table4", "table5" ]include jobs list in path
Examples:[ "job1", "job2" ]exclude jobs list in path
Examples:[ "job3", "job4" ]
- scope_lineage
list of paths
Examples:[ "/schema1/table1" ]include list in path
Examples:[ "/table2", "/table3" ]exclude list in path
Examples:[ "table4", "table5" ]include jobs list in path
Examples:[ "job1", "job2" ]exclude jobs list in path
Examples:[ "job3", "job4" ]
- scope_term_generation
list of assets.
Examples:[ "asset1", "asset2" ]
list of tags
Examples:[ "tag1", "tag2" ]check to disable or enable migrating MDI tags to data asset.
Example:
trueName of the metaBroker importflow
Example:
Run open Export- metadata_enrichment_area_info
Example:
2019-01-15T18:58:09ZExample:
96e2149d-6916-427b-bf98-a0cb989306d7
The goal the import. The possible values are 'metadata','lineage',
data_model,data_integration,data_integration_discover,business_intelligence,business_intelligence_discoverandopen_mantaAllowable values: [
BUSINESS_INTELLIGENCE,BUSINESS_INTELLIGENCE_DISCOVER,DATA_INTEGRATION,DATA_INTEGRATION_DISCOVER,DATA_MODEL,LINEAGE,METADATA,OPEN_MANTA,metadata, lineage, data_model, data_integration, data_integration_discover, business_intelligence, business_intelligence_discover, open_manta]Example:
metadata
Response
An object containing information about a metadata import asset
- metadata
The metadata import id assigned by the system. Read-only
The type of the asset
The IAM ID of the user that created the asset
Timestamp representing creation datetime of metadata import asset read-only
Example:
2019-01-15T18:58:09ZThe ID of the project which contains the asset.
- usage
Example:
2019-01-15T18:58:09ZExample:
IBMid-550000FRA0Example:
1547578689512Example:
2019-01-15T18:58:09ZExample:
1547578689512Example:
IBMid-550000FRA0
list of tags
Examples:[ "tag1", "tag2" ]
- entity
name of the metadata import asset
Example:
db2_tables_importlabel to identify new architecture
- datasource_definition_info
- metadata_ingestion_info
- extraction_info
Example:
-Direct -Agent
description of the metadata import asset
Example:
Import of a db2 tableThe type of the import. The possible values are 'metadata' and 'lineage'.
Example:
metadataId of the connection which is being imported
Example:
96e2149d-6916-427b-bf98-a0cb989306d7List of connection_id which is being discovered
List of configuration/datasource files for which metadata/lineage has to be extracted.
Examples:[ { "asset_id": "dde62b91-ae75-4d7d-aa08-e93065520aa4", "technology": "erwin" } ]- file_configuration
List of workspace rootPath.
Examples:[ { "id": "dde62b91-ae75-4d7d-aa08-e93065520aa4", "path": "/Workspace/export_uc", "name": "Databricks_connection", "type": "Connection" } ]Id of the catalog to import assets into
Example:
342b07da-fbe5-4a0a-94ee-ab89696975d5Id of the project to import assets into
Example:
c5b3daa2-c880-4f83-9bfa-c50ff70d9f42Id of the metadata import job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Id of the metadata import job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Id of DataSourceDefinitionInfo
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Scanner name of DataSourceDefinitionInfo
Example:
Oracle- scope
list of paths
Examples:[ "/schema1/table1" ]include list in path
Examples:[ "/table2", "/table3" ]exclude list in path
Examples:[ "table4", "table5" ]include jobs list in path
Examples:[ "job1", "job2" ]exclude jobs list in path
Examples:[ "job3", "job4" ]
- scope_lineage
list of paths
Examples:[ "/schema1/table1" ]include list in path
Examples:[ "/table2", "/table3" ]exclude list in path
Examples:[ "table4", "table5" ]include jobs list in path
Examples:[ "job1", "job2" ]exclude jobs list in path
Examples:[ "job3", "job4" ]
- scope_term_generation
list of assets.
Examples:[ "asset1", "asset2" ]
The type of the data source. The possible values are 'database', 'file', and 'generic'.
Possible values: [
DATABASE,FILE,GENERIC,database, file, generic]Started time of the last run.
Example:
1547578689512Completed time of the last run.
Example:
1547578689512Number of runs.
Example:
10Version of metadata import.
Example:
1Name of the metabroker importflow
Example:
Run Open ExportMetaBroker importflow executionid
The number of assets imported newly since last importing.
Example:
10The number of assets updated since last importing.
Example:
5The number of assets removed since last importing.
Example:
3the flag if the metadata enrichment can work or not.
Example:
truecheck to disable or enable migrating MDI tags to data asset.
Example:
trueoptions while reimporting metadata.
Examples:[ { "update_name": false, "update_description": false, "update_column_descriptions": false, "delete_when_deleted_at_source": false, "delete_when_removed_from_scope": false } ]support advanced options of metadata import.
Examples:[ { "exclude_tables": false, "exclude_views": false, "import_incremental_changes_only": false, "include_foreign_key": false, "include_primary_key": false } ]Advanced settings for lineage
Examples:[ { "extract_extended_attributes": false, "performance_profile": "Complete" } ]- business_extraction
- primary_category
- dictionary_connection
List of the data assets to be updated with Business Metadata
- metadata_enrichment_area_info
Example:
2019-01-15T18:58:09ZExample:
96e2149d-6916-427b-bf98-a0cb989306d7
Contains warning message, if validation of metabroker scanner passes partially.
Example:
trueThe goal the import. The possible values are 'metadata','lineage',
data_model,data_integration,data_integration_discover,business_intelligence,business_intelligence_discoverandopen_mantaExample:
metadata
Status Code
Success
Unauthorized
Forbidden
Error
No Sample Response
Bulk delete data assets in Metadata discovery
Delete assets in bulk
POST /v2/metadata_imports/bulk_delete_assets
Request
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
The catalog_id and / or project_id must be provided. If catalog_id is provided, the catalog must exist and the caller must be an admin or editor member of the catalog. Only data discovery runs associated with the catalog will be returned.
Array containing list of data asset Ids.
Bulk delete metadata import assets in Metadata discovery
Delete metadata import assets in bulk
DELETE /v2/metadata_imports/bulk_delete_metadataImport_assets
Create a job to publish the data assets and configure the list of data assets to be included in the job.
Publish assets in bulk
POST /v2/metadata_imports/bulk_publish_assets
Request
Query Parameters
The connection_id may be optionally provided to retrieve data discovery runs associated with the connection.
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
The catalog_id and / or project_id must be provided. If catalog_id is provided, the catalog must exist and the caller must be an admin or editor member of the catalog. Only data discovery runs associated with the catalog will be returned.
mdi id to retrieve the MDI object
Array containing list of data asset Ids.
Get a list of supported datasource types
INTERNAL: Get list of supported connection types.
GET /v2/metadata_imports/connections
Response
datasource id
Example:
96e2149d-6916-427b-bf98-a0cb989306d7Whether datasource supports scoping.
Example:
trueWhether datasource supports manta.
Example:
trueWhether datasource is of type relational database.
Indicates whether data be can fetched from this connection type, It will be true for SCAPI connections and false for manta only connections.
Example:
true
Status Code
Success
Unauthorized
Forbidden
Error
No Sample Response
Restart the pending a job run
Restart the pending a job run. It should be called after restore from the backup data. This API is not unavailable on IBM Cloud. The Admin role is necessary to call this API.
POST /v2/metadata_imports/recover_task
Request
Object containing information for recovering import
The timestamp when the data was restored (in format YYYY-MM-DDTHH:mm:ssZ or YYYY-MM-DDTHH:mm:ss.sssZ, matching the date-time format as specified by RFC 3339)
Example:
2022-04-13T14:59:04ZThe status that will be restarted. The possible values are 'starting' and 'running'. The 'running' includes 'starting' status, too.
Allowable values: [
RUNNING,STARTING,starting, running]
Retrieve details of a Metadata Import
Retrieve details of a metadata import
GET /v2/metadata_imports/{metadata_import_id}Request
Path Parameters
Id of the metadata import asset.
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
Response
An object containing information about a metadata import asset
- metadata
The metadata import id assigned by the system. Read-only
The type of the asset
The IAM ID of the user that created the asset
Timestamp representing creation datetime of metadata import asset read-only
Example:
2019-01-15T18:58:09ZThe ID of the project which contains the asset.
- usage
Example:
2019-01-15T18:58:09ZExample:
IBMid-550000FRA0Example:
1547578689512Example:
2019-01-15T18:58:09ZExample:
1547578689512Example:
IBMid-550000FRA0
list of tags
Examples:[ "tag1", "tag2" ]
- entity
name of the metadata import asset
Example:
db2_tables_importlabel to identify new architecture
- datasource_definition_info
- metadata_ingestion_info
- extraction_info
Example:
-Direct -Agent
description of the metadata import asset
Example:
Import of a db2 tableThe type of the import. The possible values are 'metadata' and 'lineage'.
Example:
metadataId of the connection which is being imported
Example:
96e2149d-6916-427b-bf98-a0cb989306d7List of connection_id which is being discovered
List of configuration/datasource files for which metadata/lineage has to be extracted.
Examples:[ { "asset_id": "dde62b91-ae75-4d7d-aa08-e93065520aa4", "technology": "erwin" } ]- file_configuration
List of workspace rootPath.
Examples:[ { "id": "dde62b91-ae75-4d7d-aa08-e93065520aa4", "path": "/Workspace/export_uc", "name": "Databricks_connection", "type": "Connection" } ]Id of the catalog to import assets into
Example:
342b07da-fbe5-4a0a-94ee-ab89696975d5Id of the project to import assets into
Example:
c5b3daa2-c880-4f83-9bfa-c50ff70d9f42Id of the metadata import job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Id of the metadata import job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Id of DataSourceDefinitionInfo
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Scanner name of DataSourceDefinitionInfo
Example:
Oracle- scope
list of paths
Examples:[ "/schema1/table1" ]include list in path
Examples:[ "/table2", "/table3" ]exclude list in path
Examples:[ "table4", "table5" ]include jobs list in path
Examples:[ "job1", "job2" ]exclude jobs list in path
Examples:[ "job3", "job4" ]
- scope_lineage
list of paths
Examples:[ "/schema1/table1" ]include list in path
Examples:[ "/table2", "/table3" ]exclude list in path
Examples:[ "table4", "table5" ]include jobs list in path
Examples:[ "job1", "job2" ]exclude jobs list in path
Examples:[ "job3", "job4" ]
- scope_term_generation
list of assets.
Examples:[ "asset1", "asset2" ]
The type of the data source. The possible values are 'database', 'file', and 'generic'.
Possible values: [
DATABASE,FILE,GENERIC,database, file, generic]Started time of the last run.
Example:
1547578689512Completed time of the last run.
Example:
1547578689512Number of runs.
Example:
10Version of metadata import.
Example:
1Name of the metabroker importflow
Example:
Run Open ExportMetaBroker importflow executionid
The number of assets imported newly since last importing.
Example:
10The number of assets updated since last importing.
Example:
5The number of assets removed since last importing.
Example:
3the flag if the metadata enrichment can work or not.
Example:
truecheck to disable or enable migrating MDI tags to data asset.
Example:
trueoptions while reimporting metadata.
Examples:[ { "update_name": false, "update_description": false, "update_column_descriptions": false, "delete_when_deleted_at_source": false, "delete_when_removed_from_scope": false } ]support advanced options of metadata import.
Examples:[ { "exclude_tables": false, "exclude_views": false, "import_incremental_changes_only": false, "include_foreign_key": false, "include_primary_key": false } ]Advanced settings for lineage
Examples:[ { "extract_extended_attributes": false, "performance_profile": "Complete" } ]- business_extraction
- primary_category
- dictionary_connection
List of the data assets to be updated with Business Metadata
- metadata_enrichment_area_info
Example:
2019-01-15T18:58:09ZExample:
96e2149d-6916-427b-bf98-a0cb989306d7
Contains warning message, if validation of metabroker scanner passes partially.
Example:
trueThe goal the import. The possible values are 'metadata','lineage',
data_model,data_integration,data_integration_discover,business_intelligence,business_intelligence_discoverandopen_mantaExample:
metadata
Status Code
Success
Unauthorized
Forbidden
Not found
Gone
Error
No Sample Response
delete a metadata import asset
delete a metadata import asset
DELETE /v2/metadata_imports/{metadata_import_id}edit a metadata import asset
edit a metadata import asset
PATCH /v2/metadata_imports/{metadata_import_id}Request
Path Parameters
Id of the metadata import asset.
Query Parameters
The project_id must be provided. The project must exist and the caller must be an admin or editor member of the project.
Object carrying the update request.
name of the metadata import asset
Example:
db2_tables_importdescription of the metadata import asset
Example:
Import of a db2 tableId of the connection which is being discovered
Example:
96e2149d-6916-427b-bf98-a0cb989306d7Id of the catalog to import assets into
Example:
342b07da-fbe5-4a0a-94ee-ab89696975d5Id of the project to import assets into
Example:
c5b3daa2-c880-4f83-9bfa-c50ff70d9f42Id of the metadata import job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Id of the metadata import job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Name of the metabroker importflow
Example:
Run Open Export- scope
list of paths
Examples:[ "/schema1/table1" ]include list in path
Examples:[ "/table2", "/table3" ]exclude list in path
Examples:[ "table4", "table5" ]include jobs list in path
Examples:[ "job1", "job2" ]exclude jobs list in path
Examples:[ "job3", "job4" ]
list of tags
Examples:[ "tag1", "tag2" ]the flag if the metadata enrichment can work or not.
Example:
truecheck to disable or enable migrating MDI tags to data asset.
Example:
trueasset ref map when trying to export and import a project .
specify if name, description, column descriptions to be udpated during re-import.
Examples:[ { "update_name": false, "update_description": false, "update_column_descriptions": false, "delete_when_deleted_at_source": false, "delete_when_removed_from_scope": false } ]support advanced options of metadata import.
Examples:[ { "exclude_tables": false, "exclude_views": false, "import_incremental_changes_only": false, "include_foreign_key": false, "include_primary_key": false } ]Advanced settings for lineage
Examples:[ { "extract_extended_attributes": false, "performance_profile": "Complete" } ]- metadata_ingestion_info
- extraction_info
Example:
-Direct -Agent
- datasource_definition_info
- file_configuration
- scope_lineage
list of paths
Examples:[ "/schema1/table1" ]include list in path
Examples:[ "/table2", "/table3" ]exclude list in path
Examples:[ "table4", "table5" ]include jobs list in path
Examples:[ "job1", "job2" ]exclude jobs list in path
Examples:[ "job3", "job4" ]
label to identify new architecture
Response
An object containing information about a metadata import asset
- metadata
The metadata import id assigned by the system. Read-only
The type of the asset
The IAM ID of the user that created the asset
Timestamp representing creation datetime of metadata import asset read-only
Example:
2019-01-15T18:58:09ZThe ID of the project which contains the asset.
- usage
Example:
2019-01-15T18:58:09ZExample:
IBMid-550000FRA0Example:
1547578689512Example:
2019-01-15T18:58:09ZExample:
1547578689512Example:
IBMid-550000FRA0
list of tags
Examples:[ "tag1", "tag2" ]
- entity
name of the metadata import asset
Example:
db2_tables_importlabel to identify new architecture
- datasource_definition_info
- metadata_ingestion_info
- extraction_info
Example:
-Direct -Agent
description of the metadata import asset
Example:
Import of a db2 tableThe type of the import. The possible values are 'metadata' and 'lineage'.
Example:
metadataId of the connection which is being imported
Example:
96e2149d-6916-427b-bf98-a0cb989306d7List of connection_id which is being discovered
List of configuration/datasource files for which metadata/lineage has to be extracted.
Examples:[ { "asset_id": "dde62b91-ae75-4d7d-aa08-e93065520aa4", "technology": "erwin" } ]- file_configuration
List of workspace rootPath.
Examples:[ { "id": "dde62b91-ae75-4d7d-aa08-e93065520aa4", "path": "/Workspace/export_uc", "name": "Databricks_connection", "type": "Connection" } ]Id of the catalog to import assets into
Example:
342b07da-fbe5-4a0a-94ee-ab89696975d5Id of the project to import assets into
Example:
c5b3daa2-c880-4f83-9bfa-c50ff70d9f42Id of the metadata import job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Id of the metadata import job
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Id of DataSourceDefinitionInfo
Example:
330c6256-3661-11e7-a919-92ebcb67fe33Scanner name of DataSourceDefinitionInfo
Example:
Oracle- scope
list of paths
Examples:[ "/schema1/table1" ]include list in path
Examples:[ "/table2", "/table3" ]exclude list in path
Examples:[ "table4", "table5" ]include jobs list in path
Examples:[ "job1", "job2" ]exclude jobs list in path
Examples:[ "job3", "job4" ]
- scope_lineage
list of paths
Examples:[ "/schema1/table1" ]include list in path
Examples:[ "/table2", "/table3" ]exclude list in path
Examples:[ "table4", "table5" ]include jobs list in path
Examples:[ "job1", "job2" ]exclude jobs list in path
Examples:[ "job3", "job4" ]
- scope_term_generation
list of assets.
Examples:[ "asset1", "asset2" ]
The type of the data source. The possible values are 'database', 'file', and 'generic'.
Possible values: [
DATABASE,FILE,GENERIC,database, file, generic]Started time of the last run.
Example:
1547578689512Completed time of the last run.
Example:
1547578689512Number of runs.
Example:
10Version of metadata import.
Example:
1Name of the metabroker importflow
Example:
Run Open ExportMetaBroker importflow executionid
The number of assets imported newly since last importing.
Example:
10The number of assets updated since last importing.
Example:
5The number of assets removed since last importing.
Example:
3the flag if the metadata enrichment can work or not.
Example:
truecheck to disable or enable migrating MDI tags to data asset.
Example:
trueoptions while reimporting metadata.
Examples:[ { "update_name": false, "update_description": false, "update_column_descriptions": false, "delete_when_deleted_at_source": false, "delete_when_removed_from_scope": false } ]support advanced options of metadata import.
Examples:[ { "exclude_tables": false, "exclude_views": false, "import_incremental_changes_only": false, "include_foreign_key": false, "include_primary_key": false } ]Advanced settings for lineage
Examples:[ { "extract_extended_attributes": false, "performance_profile": "Complete" } ]- business_extraction
- primary_category
- dictionary_connection
List of the data assets to be updated with Business Metadata
- metadata_enrichment_area_info
Example:
2019-01-15T18:58:09ZExample:
96e2149d-6916-427b-bf98-a0cb989306d7
Contains warning message, if validation of metabroker scanner passes partially.
Example:
trueThe goal the import. The possible values are 'metadata','lineage',
data_model,data_integration,data_integration_discover,business_intelligence,business_intelligence_discoverandopen_mantaExample:
metadata
Status Code
Success
Unauthorized
Forbidden
Not found
Gone
Error
No Sample Response
Creates a category in the glossary
This method can be used to create a new category in the glossary.
POST /v3/categories
Request
Represents a category to be created.
The name of the artifact.
Possible values: 1 ≤ length ≤ 255, Value must match regular expression
.*Example:
Name 1Relationships to classifications.
List of custom attributes with their values.
Custom relationships to asset
The long description of an artifact.
Possible values: 1 ≤ length ≤ 15000, Value must match regular expression
.*Artifact ID of a parent category
Possible values: length = 36, Value must match regular expression
([A-Za-z0-9]{8}-[A-Za-z0-9]{4}-[A-Za-z0-9]{4}-[A-Za-z0-9]{4}-[A-Za-z0-9]{12})Indicates that it is a reference copy of an artifact managed in an external metadata server.
Used by reporting service for authorization. Can be set only for root level categories.
The short description of an artifact.
Possible values: 1 ≤ length ≤ 255, Value must match regular expression
.*The steward groups assigned to an artifact.
Possible values: 1 ≤ length ≤ 255, Value must match regular expression
.*Examples:[ "steward_group1", "steward_group2" ]The stewards assigned to an artifact.
Possible values: 1 ≤ length ≤ 255, Value must match regular expression
.*Examples:[ "steward1", "steward2" ]
Response
- Response returned on creating a category.
It differs from its base class ({@link CreateResponse}) because it does not include {@code /version/{version_id}} in paths of returned URLs (the categories API does not have such endpoints, only {@code GET /categories/{category_id}}).
IDs of the created items.
Status Code
The category has been created successfully.
Bad Request
Unauthorized
Forbidden
UniqueConstraintViolation - category with given name and parent already exists.
Internal Server Error
{ "resources": [ { "href": "/v3/categories/285b7b78-3d64-48e6-a9b6-e02f4c32295f", "artifact_id": "285b7b78-3d64-48e6-a9b6-e02f4c32295f", "version_id": "12e37d76-f28b-4dc9-a35e-0c3e30f9f8ea_0", "global_id": "18091466-981e-4113-8943-2ddf162bff6d_285b7b78-3d64-48e6-a9b6-e02f4c32295f", "entity_type": "category" } ] }
Start category boostrap process
Assigns default owners and view permissions for root categories without any role assignments set
POST /v3/categories/collaborators/bootstrap
Response
Bootstrap Status
Current bootstrap status
Possible values: [
NOT_STARTED,NEW,IN_PROGRESS,SUCCEEDED,FAILED,STALLED]Number of records already bootstrapped successfully
Overall outcome of the bootstrap process. Set once bootstrapping is finished.
Current bootstrap activity
Bootstrap process errors
Total number of records to be processed by the bootstrap process
Status Code
The boostrap process started successfully
The boostrap is already in progress
Unauthorized
Internal Server Error
IN_PROGRESS
{ "status": "IN_PROGRESS", "current_step": "Bootstrapping category e39ada11-8338-3704-90e3-681a71e7c839", "completed_records": 0, "total_records": 3 }
Get status of category bootstrap process
Get status of category bootstrap process
GET /v3/categories/collaborators/bootstrap/status
Response
Bootstrap Status
Current bootstrap status
Possible values: [
NOT_STARTED,NEW,IN_PROGRESS,SUCCEEDED,FAILED,STALLED]Number of records already bootstrapped successfully
Overall outcome of the bootstrap process. Set once bootstrapping is finished.
Current bootstrap activity
Bootstrap process errors
Total number of records to be processed by the bootstrap process
Status Code
Bootstrap status fetched successfully
Unauthorized
Internal Server Error
IN_PROGRESS
{ "status": "IN_PROGRESS", "current_step": "Bootstrapping category e39ada11-8338-3704-90e3-681a71e7c839", "completed_records": 0, "total_records": 3 }NEW
{ "status": "NEW", "current_step": "Initializing role assignment bootstrap process", "completed_records": 0 }NOT_STARTED
{ "status": "NOT_STARTED" }SUCCEEDED
{ "status": "SUCCEEDED", "completion_message": "Bootstrap process completed", "completed_records": 3, "total_records": 3 }
Get history status of category bootstrap process
Get history status of category bootstrap process
GET /v3/categories/collaborators/bootstrap/status/history
Response
Response returned on retrieving bootstrap status history.
Bootstrap status history list
Status Code
Bootstrap status history fetched successfully
Unauthorized
Internal Server Error
SUCCEEDED
{ "bootstrap_status_history": [ { "status": "SUCCEEDED", "completion_message": "No categories requiring bootstrap process", "completed_records": 0, "total_records": 0 }, { "status": "SUCCEEDED", "completion_message": "Bootstrap process completed", "completed_records": 3, "total_records": 3 } ] }
Retrieves category hierarchy paths for given artifact ids of categories
This method can be used for retrieving hierarchy paths of categories
GET /v3/categories/hierarchy
Request
Query Parameters
The id of the category whose path is fetched
Possible values: 1 ≤ length ≤ 100, Value must match regular expression
([A-Za-z0-9]{8}-[A-Za-z0-9]{4}-[A-Za-z0-9]{4}-[A-Za-z0-9]{4}-[A-Za-z0-9]{12})Example:
aa0e33f5-3108-4d45-a530-0307458362d4
Response
Response returned on retrieving hierarchy path for category.
IDs and names of categories in category hierarchy path for each category id
Status Code
The hierarchy paths of categories have been retrieved successfully
Unauthorized
The category with given {guid} does not exist in the glossary.
Internal Server Error
{ "category_hierarchy_paths": [ { "category_id": "b50f0822-eff3-4912-932e-aa61e5e7ac9a", "category_hierarchy_path": [ { "artifact_id": "4c4b18ae-010c-4986-a308-93c6f22095d6", "name": "category name 1", "user_access": true }, { "artifact_id": "fedc7149-5504-4ea9-aca7-3ad025be6f7d", "name": "category name 2", "user_access": true } ] } ] }
Retrieves category hierarchy paths for given artifact ids of categories
This method can be used for retrieving hierarchy paths of categories
POST /v3/categories/hierarchy
Request
Query Parameters
The maximum number of categories to return - must be at least 1 and cannot exceed 1000. The default value is 10.
Possible values: 1 ≤ value ≤ 1000
Default:
10The index of the first matching category to include in the result.
List of glossary resources
{
"resources": [
{
"artifact_id": "2b9bb7cf-502e-43a2-9ecc-645e909cba33"
}
]
}List of resources.
Response
Response returned on retrieving hierarchy path for category.
IDs and names of categories in category hierarchy path for each category id
Status Code
The hierarchy paths of categories have been retrieved successfully
Unauthorized
Internal Server Error
{ "resources": [ { "metadata": { "artifact_type": "category", "artifact_id": "2b9bb7cf-502e-43a2-9ecc-645e909cba33", "version_id": "5b6a662e-10d5-47fd-b485-655521ee491a_0", "source_repository_id": "2513bb40-5ce0-440e-9aa7-f473d1c22f9f", "source_repository_name": "WKC_BG_2513bb40-5ce0-440e-9aa7-f473d1c22f9f", "global_id": "2513bb40-5ce0-440e-9aa7-f473d1c22f9f_2b9bb7cf-502e-43a2-9ecc-645e909cba33", "created_by": "System", "created_at": "2023-11-13T06:50:09.240Z", "modified_by": "System", "modified_at": "2023-11-13T06:50:09.240Z", "revision": "0", "name": "child of child2 category", "short_description": "child of child2-sd", "state": "PUBLISHED", "tags": [], "read_only": false }, "entity": { "parent_category_id": "b02920a4-09ae-4a62-a25f-03a2d7ba0507", "reporting_authorized": false, "long_description": "child of child2-ld", "state": "PUBLISHED", "default_locale_id": "en-US", "reference_copy": false } } ], "offset": 0, "last": { "href": "/v3/categories/hierarchy?limit=10&offset=0", "offset": "0" }, "set_uri": false, "limit": 10, "count": 1, "first": { "href": "/v3/categories/hierarchy?limit=10&offset=0", "offset": "0" } }
Deletes the category with a given guid
This method can be used to delete a category.
Note! The category must be empty. It can contain neither child categories nor artifacts.
DELETE /v3/categories/{category_id}Request
Path Parameters
Artifact ID or global ID of the artifact
Possible values: length = 36, Value must match regular expression
([A-Za-z0-9]{8}-[A-Za-z0-9]{4}-[A-Za-z0-9]{4}-[A-Za-z0-9]{4}-[A-Za-z0-9]{12})Example:
aa0e33f5-3108-4d45-a530-0307458362d4
Response
Glossary archive response object
Guids of the archived objects.
Guids of the deleted objects.
Guids of the updated objects.
Status Code
The category has been deleted successfully.
Bad Request
Unauthorized
Forbidden
The category with given {guid} does not exist in the glossary.
Internal Server Error
No Sample Response
Retrieves category with given guid
This method can be used for retrieving details of a category.
GET /v3/categories/{category_id}Request
Path Parameters
Artifact ID or global ID of the artifact
Possible values: length = 36, Value must match regular expression
([A-Za-z0-9]{8}-[A-Za-z0-9]{4}-[A-Za-z0-9]{4}-[A-Za-z0-9]{4}-[A-Za-z0-9]{12})Example:
aa0e33f5-3108-4d45-a530-0307458362d4
Query Parameters
When set to true fetch category relationships user has access to. It returns at most 1000 relationships. If parameter is not set its default value is 'true'. This is deprecated and default will be change to 'false' in first major 2022 release.
Deprecated, functionality will be replaced by new one introduced in the future releases.Default:
trueIf this parameter is set to true, then all artifact custom attributes are returned. If parameter is not set its default value is 'true'. This is deprecated and default will be change to 'false' in first major 2022 release.
Default:
true
Response
Represents a category object - which contains metadata and category entity.
Represents a category.
Metadata of the entity.
Status Code
The category has been retrieved successfully.
Unauthorized
Forbidden
The category with given {guid} does not exist in the glossary.
Internal Server Error
No Sample Response
Updates category with given id
This method can be used to update a category with given id.
It may be an update of its name, description, etc..
PATCH /v3/categories/{category_id}Request
Path Parameters
Artifact ID or global ID of the artifact
Possible values: length = 36, Value must match regular expression
([A-Za-z0-9]{8}-[A-Za-z0-9]{4}-[A-Za-z0-9]{4}-[A-Za-z0-9]{4}-[A-Za-z0-9]{12})
Category to be updated.
Fields omitted will be unchanged, and fields set to null explicitly will be nulled out.
For multi-valued attributes & relationships, the complete list will be replaced by the given list of values.
Additional Example:
{"description" : "short desc updated"}
Relationships to classifications.