IBM Cloud Docs
Managing Spark query server

Managing Spark query server

Applies to : Spark engine

Spark query server allows you to start Spark Hive based thrift server that provides the interface to collect, store, query and analyse all your lakehouse data from Spark engine.

Spark query server functionality can be leveraged in various ways, see Integration.

Before you begin

  • Install watsonx.data.
  • Provision native Spark engine in watsonx.data.

Create a Spark query server

For the Spark engine to integrate with dbt tool and work as a query engine, you must create a Spark query server.

  1. Log in to the watsonx.data instance.

  2. Navigate to Infrastructure manager. Click the Spark engine.

  3. Click Query servers tab.

  4. Click Create query servers. The Create query servers page opens.

  5. Provide the following details:

    • Name: Enter a name for the query server that you create.
    • Driver and Executor cores and memory.
    • Username: The watsonx.data login username.
    • API key: Your API key. To generate an API key, see Generating the API key.
  6. Click Create. The query server is created and will be in ACCEPTED status. You can start using the server when the status becomes ACTIVE.

You can view the query server details like, the name, ID, status, server start time and stop time, and the connection URL.

Retrieve the query server connection details

To configure the profile file in dbt tool, you must save the query server connection details.

  1. From the Query servers tab, select the query server that is in ACTIVE state.

  2. Click the overflow menu against the query server that you select.

  3. Click View connection details. The Connection details page opens with the profile configuration.

  4. Copy the connection details.

    Alternatively, you can retrieve the connection details from the Connection information page. From the navigation menu, go to ConfigurationsConnection informationData Build Tool (DBT).

Stopping the Spark query server

  1. From the Query servers tab, you can view the lat of query servers.
  2. Click the overflow menu against the query server that you select.
  3. Click Stop.

Integrations supported