dbt Configuration (setting up your dbt profile)

To connect dbt core to your Spark engine, configure the profiles.yml file that is located in .dbt of your home directory.

You can either copy or export the Spark connection details to create the profiles.yml. To do that, see Getting connection information.

The following is an example configuration:

profile_name:
  target: "dev"
  outputs:
    dev:
      type: "watsonx_spark"
      method: "http"
      schema: "<wxd-schema>"
      host: "https://us-south.lakehouse.cloud.ibm.com"
      uri: "/lakehouse/api/v2/spark_engines/spark216/query_servers/02bda638-1399-4914-8ae7-ab4223764d26/connect/cliservice"
      catalog: "<wxd-catalog>"
      auth:
        instance: "<watsonx.data_instance_crn>"
        user: "<username>"
        apikey: "<apikey>"

The following list covers the parameter details:

• profile_name: The profile name as the dbt project name.

• schema: The table schema name associated with the Spark engine's catalog.

  If the specified schema does not exist in the chosen catalog, a new schema is created automatically.

• host: Hostname of your watsonx.data console. For more information, see Getting connection information.

• uri: URI of your query server that is running on watsonx.data. For more information, see Getting connection information.

• catalog: The catalog that is associated with the Spark engine.

• instance: The instance CRN of your watsonx.data instance. To retrieve the CRN, see Getting connection information.

• user: Your email-id if you are using your API key or it should be in the format <Service-id>-<GUID>. For more information on generating service id and GUID, see Creating service IDs.

• apikey: Your API key. To generate the API key, see Managing user API keys.