Capturing DDL changes in watsonx.data through event listener

When tables or schemas are created, altered, or deleted in watsonx.data, an event can be captured and notified by enabling an event listener.

Before you begin

Deploy a Kafka server to subscribe and publish the captured event. The server can either be an on-prem server or a service in the public cloud. For example, Event Stream in IBM Cloud. Events are captured and stored in Kafka and can be subscribed by consumers.

Procedure

Enable the listener through watsonx.data REST API to capture the changes.

POST https://{hostname}/lakehouse/api/v2/{instance id}/configure_hms_listener.

Request Payload (JSON):

{
   "kafka_properties":[
      {
        "key":"hive.metastore.kafka.bootstrap.server",
        "value":"kafka-server-1:port,kafka-server-2:port",
        "encrypt":false
      },
      {
         "key":"hive.metastore.kafka.topic.name",
         "value":"TOPIC",
         "encrypt":false},
      {
         "key":"hive.metastore.kafka.topic.message.type",
         "value":"THRIFT_HIVE_JSON",
         "encrypt":false}
      ]
}

The listener is enabled a few minutes after the API returns a success. The object change event is sent to the topic in Kafka if DDL is performed on table or schema. You can get the event from the topic.

If required, change the listener settings and call the API again with a different value.

Retrieve the current listener settings through this REST API:

POST https://{hostname}/lakehouse/api/v2/{instance id}/metastore_custom_properties.

Supported properties:

Table 1. Supported properties
Properties	Default value	Description
hive.metastore.kafka.bootstrap.server		(Required) URL of Kafka brokerlist, separated by comma.
hive.metastore.kafka.topic.name	HMS_MSG	(Optional) The topic name used in Kafka. An event is sent to the specified topic. If a topic does not exist, watsonx.data creates it in Kafka.
hive.metastore.kafka.topic.message.type	HIVE_JSON	(Optional) Specifies the format of the event to be generated.
		HIVE_JSON: JSON
		THRIFT_HIVE_JSON: JSON string serialized by org.apache.thrift.TSerializer
hive.metastore.kafka.username		(Required if SASL is enabled) Username of Kafka server if authentication is enabled in Kafka server.
hive.metastore.kafka.password		(Required if SASL is enabled) Password of Kafka server if authentication is enabled in Kafka server.
hive.metastore.kafka.topic.replication.factor	0	(Required if specified topic does not exist in Kafka.) The replication factors for creating topics. If the setting is different from the replication factors in Kafka, topic creation can fail.
hive.metastore.kafka.topic.partition.number	1	(Optional) The number of partitions per topic, which is used for creating topic.
hive.metastore.kafka.security.protocol		(Required if SASL is enabled and a non-default value is used in Kafka.) Protocol setting in Kafka to communicate with Kafka brokers.
hive.metastore.kafka.sasl.mechanism		(Required if SASL is enabled and a non-default value is used in Kafka.) SASL mechanism used for client connections.
hive.metastore.kafka.truststore.certificates		(Required if certificate is used in Kafka and hive.metastore.kafka.disable.cert.verification is false) Base64 encoded PEM cert chain string. The string can be generated by the command echo `cat <your_cert_file>`
hive.metastore.kafka.disable.cert.verification		(Optional) Disable cert verification when value is ‘true’ (case-insensitive). No effect for other values.
hive.metastore.kafka.listener.partition.event.enabled	FALSE	(Optional) Capture partition event

Data is captured for the following events:

Create schema
Drop schema
Create table
Alter table
Drop table
Add partition
Alter partition

When an event is triggered, details about the event are pushed to the Kafka server. You must subscribe to the Kafka topic through an application or Kafka client, and then the event is pushed by the Kafka server. The changed object (table/schema) details are included in the event. For alter event, the event includes the object details before and after the alter.

Example: Create table event

The DDL performed is:

CREATE TABLE "iceberg_data"."example"."table1"
(
"ID" INT,
"Name" VARCHAR
)

THRIFT_HIVE_JSON

Details about an event in THRIFT_HIVE_JSON format contain event type, schema name, table name, and table details. Table details are serialized as a string. An event size is smaller than HIVE_JSON.

{
  "eventType": "CREATE_TABLE",
  "dbName": "example",
  "tableName": "table1",
  "tableType": "EXTERNAL_TABLE",
  "table": "{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"example\"},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1699947381},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",2,{\"1\":{\"str\":\"id\"},\"2\":{\"str\":\"int\"}},{\"1\":{\"str\":\"name\"},\"2\":{\"str\":\"string\"}}]},\"2\":{\"str\":\"s3a://iceberg-bucket/example/table1\"},\"3\":{\"str\":\"org.apache.hadoop.mapred.FileInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.mapred.FileOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"totalSize\":\"1226\",\"EXTERNAL\":\"TRUE\",\"numFiles\":\"1\",\"metadata_location\":\"s3a://iceberg-bucket/example/table1/metadata/00000-12f298e9-747c-4730-9062-3784b7777280.metadata.json\",\"transient_lastDdlTime\":\"1699947381\",\"numFilesErasureCoded\":\"0\",\"table_type\":\"iceberg\"}]},\"12\":{\"str\":\"EXTERNAL_TABLE\"},\"13\":{\"rec\":{\"1\":{\"map\":[\"str\",\"lst\",1,{\"ibmlhadmin\":[\"rec\",4,{\"1\":{\"str\":\"select\"},\"2\":{\"i32\":0},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"insert\"},\"2\":{\"i32\":0},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"update\"},\"2\":{\"i32\":0},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"delete\"},\"2\":{\"i32\":0},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}}]}]},\"2\":{\"map\":[\"str\",\"lst\",0,{}]},\"3\":{\"map\":[\"str\",\"lst\",0,{}]}}},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1},\"25\":{\"i64\":1}}"
}

You can check the event types and the table information from JSON without de-serializing the table details. If you need the details, import the library org.apache.thrift and get Table Object from the JSON string in event. You can retrieve the details from the Java Object.

import org.apache.thrift.TDeserializer;
import org.apache.thrift.protocol.TJSONProtocol;
import org.apache.hadoop.hive.metastore.api.Table;
...
//Get string from "table" field first
TDeserializer deSerializer = new TDeserializer(new TJSONProtocol.Factory());
org.apache.hadoop.hive.metastore.api.Table table = new Table();
deSerializer.deserialize(table, tableStr, "UTF-8");
List<FieldSchema> columns = table.getSd().getCols()

HIVE_JSON

Details about an event in HIVE_JSON format contain event type and table details in JSON format. You can parse the JSON by any JSON library and get the details

{
    "eventType": "CREATE_TABLE",
    "table": {
        "tableName": "table1",
        "dbName": "example",
        "owner": "ibmlhadmin",
        "createTime": 1699948569,
        "lastAccessTime": 0,
        "retention": 0,
        "sd": {
        "cols": [
            {
            "name": "id",
            "type": "int",
            "comment": null,
            "setName": true,
            "setType": true,
            "setComment": false
            },
            {
            "name": "name",
            "type": "string",
            "comment": null,
            "setName": true,
            "setType": true,
            "setComment": false
            }
        ],
        "location": "s3a://iceberg-bucket/example/table1",
        "inputFormat": "org.apache.hadoop.mapred.FileInputFormat",
        "outputFormat": "org.apache.hadoop.mapred.FileOutputFormat",
        "compressed": false,
        "numBuckets": 0,
        "serdeInfo": {
            "name": "table1",
            "serializationLib": "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe",
            "parameters": {},
            "description": null,
            "serializerClass": null,
            "deserializerClass": null,
            "serdeType": null,
            "setName": true,
            "setSerializationLib": true,
            "parametersSize": 0,
            "setParameters": true,
            "setDescription": false,
            "setSerializerClass": false,
            "setDeserializerClass": false,
            "setSerdeType": false
        },
        "bucketCols": null,
        "sortCols": null,
        "parameters": {},
        "skewedInfo": null,
        "storedAsSubDirectories": false,
        "colsSize": 2,
        "colsIterator": [
            {
            "name": "id",
            "type": "int",
            "comment": null,
            "setName": true,
            "setType": true,
            "setComment": false
            },
            {
            "name": "name",
            "type": "string",
            "comment": null,
            "setName": true,
            "setType": true,
            "setComment": false
            }
        ],
        "setCols": true,
        "setLocation": true,
        "setInputFormat": true,
        "setOutputFormat": true,
        "setCompressed": true,
        "setNumBuckets": true,
        "setSerdeInfo": true,
        "bucketColsSize": 0,
        "bucketColsIterator": null,
        "setBucketCols": false,
        "sortColsSize": 0,
        "sortColsIterator": null,
        "setSortCols": false,
        "parametersSize": 0,
        "setParameters": true,
        "setSkewedInfo": false,
        "setStoredAsSubDirectories": false
        },
        "partitionKeys": [],
        "parameters": {
        "totalSize": "4294",
        "EXTERNAL": "TRUE",
        "numFiles": "3",
        "metadata_location": "s3a://iceberg-bucket/example/table1/metadata/00000-5814ea67-633c-4b4c-abe6-d85b40d68ea8.metadata.json",
        "transient_lastDdlTime": "1699948569",
        "numFilesErasureCoded": "0",
        "table_type": "iceberg"
        },
        "viewOriginalText": null,
        "viewExpandedText": null,
        "tableType": "EXTERNAL_TABLE",
        "privileges": {
        "userPrivileges": {
            "ibmlhadmin": [
            {
                "privilege": "select",
                "createTime": 0,
                "grantor": "ibmlhadmin",
                "grantorType": "USER",
                "grantOption": true,
                "setPrivilege": true,
                "setCreateTime": true,
                "setGrantor": true,
                "setGrantorType": true,
                "setGrantOption": true
            },
            {
                "privilege": "insert",
                "createTime": 0,
                "grantor": "ibmlhadmin",
                "grantorType": "USER",
                "grantOption": true,
                "setPrivilege": true,
                "setCreateTime": true,
                "setGrantor": true,
                "setGrantorType": true,
                "setGrantOption": true
            },
            {
                "privilege": "update",
                "createTime": 0,
                "grantor": "ibmlhadmin",
                "grantorType": "USER",
                "grantOption": true,
                "setPrivilege": true,
                "setCreateTime": true,
                "setGrantor": true,
                "setGrantorType": true,
                "setGrantOption": true
            },
            {
                "privilege": "delete",
                "createTime": 0,
                "grantor": "ibmlhadmin",
                "grantorType": "USER",
                "grantOption": true,
                "setPrivilege": true,
                "setCreateTime": true,
                "setGrantor": true,
                "setGrantorType": true,
                "setGrantOption": true
            }
            ]
        },
        "groupPrivileges": {},
        "rolePrivileges": {},
        "userPrivilegesSize": 1,
        "setUserPrivileges": true,
        "groupPrivilegesSize": 0,
        "setGroupPrivileges": true,
        "rolePrivilegesSize": 0,
        "setRolePrivileges": true
        },
        "temporary": false,
        "rewriteEnabled": false,
        "creationMetadata": null,
        "catName": "hive",
        "ownerType": "USER",
        "writeId": -1,
        "isStatsCompliant": false,
        "colStats": null,
        "accessType": 0,
        "requiredReadCapabilities": null,
        "requiredWriteCapabilities": null,
        "id": 11,
        "fileMetadata": null,
        "dictionary": null,
        "txnId": 0,
        "setTableName": true,
        "setDbName": true,
        "setOwner": true,
        "setCreateTime": true,
        "setLastAccessTime": true,
        "setRetention": true,
        "setSd": true,
        "partitionKeysSize": 0,
        "partitionKeysIterator": [],
        "setPartitionKeys": true,
        "parametersSize": 7,
        "setParameters": true,
        "setViewOriginalText": false,
        "setViewExpandedText": false,
        "setTableType": true,
        "setPrivileges": true,
        "setTemporary": false,
        "setRewriteEnabled": false,
        "setCreationMetadata": false,
        "setCatName": true,
        "setOwnerType": true,
        "setWriteId": false,
        "setIsStatsCompliant": false,
        "setColStats": false,
        "setAccessType": false,
        "requiredReadCapabilitiesSize": 0,
        "requiredReadCapabilitiesIterator": null,
        "setRequiredReadCapabilities": false,
        "requiredWriteCapabilitiesSize": 0,
        "requiredWriteCapabilitiesIterator": null,
        "setRequiredWriteCapabilities": false,
        "setId": true,
        "setFileMetadata": false,
        "setDictionary": false,
        "setTxnId": false
    }
}

Example

Event content example

THRIFT_HIVE_JSON

Create schema

{
    "eventType": "CREATE_DATABASE",
    "dbName": "example",
    "db": "{\"1\":{\"str\":\"example\"},\"3\":{\"str\":\"s3a://iceberg-bucket/example\"},\"4\":{\"map\":[\"str\",\"str\",0,{}]},\"6\":{\"str\":\"ibmlhadmin\"},\"7\":{\"i32\":1},\"8\":{\"str\":\"hive\"},\"9\":{\"i32\":1699947172}}"
}

Drop schema

{
    "eventType": "DROP_DATABASE",
    "dbName": "example",
    "db": "{\"1\":{\"str\":\"example\"},\"3\":{\"str\":\"s3a://iceberg-bucket/example\"},\"4\":{\"map\":[\"str\",\"str\",0,{}]},\"6\":{\"str\":\"ibmlhadmin\"},\"7\":{\"i32\":1},\"8\":{\"str\":\"hive\"},\"9\":{\"i32\":1699947172},\"11\":{\"i32\":1}}"
}

Create table

{
    "eventType": "CREATE_TABLE",
    "dbName": "example",
    "tableName": "table1",
    "tableType": "EXTERNAL_TABLE",
    "table": "{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"example\"},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1699947381},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",2,{\"1\":{\"str\":\"id\"},\"2\":{\"str\":\"int\"}},{\"1\":{\"str\":\"name\"},\"2\":{\"str\":\"string\"}}]},\"2\":{\"str\":\"s3a://iceberg-bucket/example/table1\"},\"3\":{\"str\":\"org.apache.hadoop.mapred.FileInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.mapred.FileOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"totalSize\":\"1226\",\"EXTERNAL\":\"TRUE\",\"numFiles\":\"1\",\"metadata_location\":\"s3a://iceberg-bucket/example/table1/metadata/00000-12f298e9-747c-4730-9062-3784b7777280.metadata.json\",\"transient_lastDdlTime\":\"1699947381\",\"numFilesErasureCoded\":\"0\",\"table_type\":\"iceberg\"}]},\"12\":{\"str\":\"EXTERNAL_TABLE\"},\"13\":{\"rec\":{\"1\":{\"map\":[\"str\",\"lst\",1,{\"ibmlhadmin\":[\"rec\",4,{\"1\":{\"str\":\"select\"},\"2\":{\"i32\":0},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"insert\"},\"2\":{\"i32\":0},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"update\"},\"2\":{\"i32\":0},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"delete\"},\"2\":{\"i32\":0},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}}]}]},\"2\":{\"map\":[\"str\",\"lst\",0,{}]},\"3\":{\"map\":[\"str\",\"lst\",0,{}]}}},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1},\"25\":{\"i64\":1}}"
}

Alter table

{
    "eventType": "ALTER_TABLE",
    "dbName": "example",
    "tableName": "table1",
    "tableType": "EXTERNAL_TABLE",
    "oldTable": "{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"example\"},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1699947381},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",2,{\"1\":{\"str\":\"id\"},\"2\":{\"str\":\"int\"}},{\"1\":{\"str\":\"name\"},\"2\":{\"str\":\"string\"}}]},\"2\":{\"str\":\"s3a://iceberg-bucket/example/table1\"},\"3\":{\"str\":\"org.apache.hadoop.mapred.FileInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.mapred.FileOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"totalSize\":\"1226\",\"EXTERNAL\":\"TRUE\",\"numFiles\":\"1\",\"transient_lastDdlTime\":\"1699947381\",\"metadata_location\":\"s3a://iceberg-bucket/example/table1/metadata/00000-12f298e9-747c-4730-9062-3784b7777280.metadata.json\",\"numFilesErasureCoded\":\"0\",\"table_type\":\"iceberg\"}]},\"12\":{\"str\":\"EXTERNAL_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1},\"19\":{\"i64\":0},\"25\":{\"i64\":1}}",
    "newTable": "{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"example\"},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":0},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",3,{\"1\":{\"str\":\"id\"},\"2\":{\"str\":\"int\"}},{\"1\":{\"str\":\"name\"},\"2\":{\"str\":\"string\"}},{\"1\":{\"str\":\"age\"},\"2\":{\"str\":\"int\"}}]},\"2\":{\"str\":\"s3a://iceberg-bucket/example/table1\"},\"3\":{\"str\":\"org.apache.hadoop.mapred.FileInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.mapred.FileOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",8,{\"previous_metadata_location\":\"s3a://iceberg-bucket/example/table1/metadata/00000-12f298e9-747c-4730-9062-3784b7777280.metadata.json\",\"totalSize\":\"3068\",\"EXTERNAL\":\"TRUE\",\"numFiles\":\"2\",\"transient_lastDdlTime\":\"1699947381\",\"metadata_location\":\"s3a://iceberg-bucket/example/table1/metadata/00001-e358db94-34f8-4a59-b5ca-a17022edf957.metadata.json\",\"numFilesErasureCoded\":\"0\",\"table_type\":\"iceberg\"}]},\"12\":{\"str\":\"EXTERNAL_TABLE\"},\"13\":{\"rec\":{\"1\":{\"map\":[\"str\",\"lst\",1,{\"ibmlhadmin\":[\"rec\",4,{\"1\":{\"str\":\"select\"},\"2\":{\"i32\":0},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"insert\"},\"2\":{\"i32\":0},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"update\"},\"2\":{\"i32\":0},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"delete\"},\"2\":{\"i32\":0},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}}]}]},\"2\":{\"map\":[\"str\",\"lst\",0,{}]},\"3\":{\"map\":[\"str\",\"lst\",0,{}]}}},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}"
}

Drop table

{
    "eventType": "DROP_TABLE",
    "dbName": "example",
    "tableName": "table1",
    "tableType": "EXTERNAL_TABLE",
    "table": "{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"example\"},\"3\":{\"str\":\"ibmlhadmin\"},\"4\":{\"i32\":1699947381},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",3,{\"1\":{\"str\":\"id\"},\"2\":{\"str\":\"int\"}},{\"1\":{\"str\":\"name\"},\"2\":{\"str\":\"string\"}},{\"1\":{\"str\":\"age\"},\"2\":{\"str\":\"int\"}}]},\"2\":{\"str\":\"s3a://iceberg-bucket/example/table1\"},\"3\":{\"str\":\"org.apache.hadoop.mapred.FileInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.mapred.FileOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",8,{\"previous_metadata_location\":\"s3a://iceberg-bucket/example/table1/metadata/00000-12f298e9-747c-4730-9062-3784b7777280.metadata.json\",\"totalSize\":\"3068\",\"EXTERNAL\":\"TRUE\",\"numFiles\":\"2\",\"transient_lastDdlTime\":\"1699947381\",\"metadata_location\":\"s3a://iceberg-bucket/example/table1/metadata/00001-e358db94-34f8-4a59-b5ca-a17022edf957.metadata.json\",\"numFilesErasureCoded\":\"0\",\"table_type\":\"iceberg\"}]},\"12\":{\"str\":\"EXTERNAL_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1},\"19\":{\"i64\":0},\"25\":{\"i64\":1}}"
}

Add partition

{
    "eventType": "ADD_PARTITION",
    "dbName": "partition_example",
    "tableName": "table1",
    "partitions": [
        {
        "country": "US"
        }
    ],
    "partitionListJson": [
        "{\"1\":{\"lst\":[\"str\",1,\"US\"]},\"2\":{\"str\":\"partition_example\"},\"3\":{\"str\":\"table1\"},\"4\":{\"i32\":1700735087},\"5\":{\"i32\":0},\"6\":{\"rec\":{\"1\":{\"lst\":[\"rec\",2,{\"1\":{\"str\":\"id\"},\"2\":{\"str\":\"int\"}},{\"1\":{\"str\":\"name\"},\"2\":{\"str\":\"string\"}}]},\"2\":{\"str\":\"s3a://hive-bucket/partition_example/table1/country=US\"},\"3\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"10\":{\"map\":[\"str\",\"str\",1,{\"preferred_ordering_columns\":\"\"}]}}},\"7\":{\"map\":[\"str\",\"str\",8,{\"presto_query_id\":\"20231123_102444_00028_9tgi3\",\"totalSize\":\"619\",\"numRows\":\"1\",\"rawDataSize\":\"14\",\"numFiles\":\"1\",\"transient_lastDdlTime\":\"1700735087\",\"numFilesErasureCoded\":\"0\",\"presto_version\":\"0.282\"}]},\"9\":{\"str\":\"hive\"}}"
    ],
    "partitionList": [
        "{\"1\":{\"lst\":[\"str\",1,\"US\"]},\"2\":{\"str\":\"partition_example\"},\"3\":{\"str\":\"table1\"},\"4\":{\"i32\":1700735087},\"5\":{\"i32\":0},\"6\":{\"rec\":{\"1\":{\"lst\":[\"rec\",2,{\"1\":{\"str\":\"id\"},\"2\":{\"str\":\"int\"}},{\"1\":{\"str\":\"name\"},\"2\":{\"str\":\"string\"}}]},\"2\":{\"str\":\"s3a://hive-bucket/partition_example/table1/country=US\"},\"3\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"10\":{\"map\":[\"str\",\"str\",1,{\"preferred_ordering_columns\":\"\"}]}}},\"7\":{\"map\":[\"str\",\"str\",8,{\"presto_query_id\":\"20231123_102444_00028_9tgi3\",\"totalSize\":\"619\",\"numRows\":\"1\",\"rawDataSize\":\"14\",\"numFiles\":\"1\",\"transient_lastDdlTime\":\"1700735087\",\"numFilesErasureCoded\":\"0\",\"presto_version\":\"0.282\"}]},\"9\":{\"str\":\"hive\"}}"
    ]
}

Alter partition

{
    "eventType": "ALTER_PARTITION",
    "dbName": "partition_example",
    "tableName": "table1",
    "oldPartition": "{\"1\":{\"lst\":[\"str\",1,\"US\"]},\"2\":{\"str\":\"partition_example\"},\"3\":{\"str\":\"table1\"},\"4\":{\"i32\":1700735087},\"5\":{\"i32\":0},\"6\":{\"rec\":{\"1\":{\"lst\":[\"rec\",2,{\"1\":{\"str\":\"id\"},\"2\":{\"str\":\"int\"}},{\"1\":{\"str\":\"name\"},\"2\":{\"str\":\"string\"}}]},\"2\":{\"str\":\"s3a://hive-bucket/partition_example/table1/country=US\"},\"3\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",1,{\"preferred_ordering_columns\":\"\"}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"7\":{\"map\":[\"str\",\"str\",8,{\"presto_query_id\":\"20231123_102444_00028_9tgi3\",\"totalSize\":\"619\",\"numRows\":\"1\",\"rawDataSize\":\"14\",\"numFiles\":\"1\",\"transient_lastDdlTime\":\"1700735087\",\"numFilesErasureCoded\":\"0\",\"presto_version\":\"0.282\"}]},\"9\":{\"str\":\"hive\"},\"10\":{\"i64\":-1}}",
    "newPartition": "{\"1\":{\"lst\":[\"str\",1,\"US\"]},\"2\":{\"str\":\"partition_example\"},\"3\":{\"str\":\"table1\"},\"4\":{\"i32\":1700735087},\"5\":{\"i32\":0},\"6\":{\"rec\":{\"1\":{\"lst\":[\"rec\",2,{\"1\":{\"str\":\"id\"},\"2\":{\"str\":\"int\"}},{\"1\":{\"str\":\"name\"},\"2\":{\"str\":\"string\"}}]},\"2\":{\"str\":\"s3a://hive-bucket/partition_example/table1/country=US\"},\"3\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",1,{\"preferred_ordering_columns\":\"\"}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"7\":{\"map\":[\"str\",\"str\",8,{\"presto_query_id\":\"20231123_102444_00028_9tgi3\",\"totalSize\":\"1231\",\"numRows\":\"2\",\"rawDataSize\":\"27\",\"numFiles\":\"2\",\"transient_lastDdlTime\":\"1700735168\",\"numFilesErasureCoded\":\"0\",\"presto_version\":\"0.282\"}]},\"9\":{\"str\":\"hive\"},\"10\":{\"i64\":-1}}",
    "keyValues": {
        "country": "US"
    }
}

Drop partition

{
    "eventType": "DROP_PARTITION",
    "dbName": "partition_example",
    "tableName": "table1",
    "partitions": [
        {
        "country": "US"
        }
    ],
    "partitionListJson": [
        "{\"1\":{\"lst\":[\"str\",1,\"US\"]},\"2\":{\"str\":\"partition_example\"},\"3\":{\"str\":\"table1\"},\"4\":{\"i32\":1700734787},\"5\":{\"i32\":0},\"6\":{\"rec\":{\"1\":{\"lst\":[\"rec\",2,{\"1\":{\"str\":\"id\"},\"2\":{\"str\":\"int\"}},{\"1\":{\"str\":\"name\"},\"2\":{\"str\":\"string\"}}]},\"2\":{\"str\":\"s3a://hive-bucket/partition_example/table1/country=US\"},\"3\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",1,{\"preferred_ordering_columns\":\"\"}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"7\":{\"map\":[\"str\",\"str\",8,{\"presto_query_id\":\"20231123_101944_00025_9tgi3\",\"totalSize\":\"619\",\"numRows\":\"1\",\"rawDataSize\":\"14\",\"numFiles\":\"1\",\"transient_lastDdlTime\":\"1700734787\",\"numFilesErasureCoded\":\"0\",\"presto_version\":\"0.282\"}]},\"9\":{\"str\":\"hive\"},\"10\":{\"i64\":-1}}"
    ],
    "deleteData": true,
    "partitionList": [
        "{\"1\":{\"lst\":[\"str\",1,\"US\"]},\"2\":{\"str\":\"partition_example\"},\"3\":{\"str\":\"table1\"},\"4\":{\"i32\":1700734787},\"5\":{\"i32\":0},\"6\":{\"rec\":{\"1\":{\"lst\":[\"rec\",2,{\"1\":{\"str\":\"id\"},\"2\":{\"str\":\"int\"}},{\"1\":{\"str\":\"name\"},\"2\":{\"str\":\"string\"}}]},\"2\":{\"str\":\"s3a://hive-bucket/partition_example/table1/country=US\"},\"3\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"table1\"},\"2\":{\"str\":\"org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",1,{\"preferred_ordering_columns\":\"\"}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"7\":{\"map\":[\"str\",\"str\",8,{\"presto_query_id\":\"20231123_101944_00025_9tgi3\",\"totalSize\":\"619\",\"numRows\":\"1\",\"rawDataSize\":\"14\",\"numFiles\":\"1\",\"transient_lastDdlTime\":\"1700734787\",\"numFilesErasureCoded\":\"0\",\"presto_version\":\"0.282\"}]},\"9\":{\"str\":\"hive\"},\"10\":{\"i64\":-1}}"
    ]
}