Viewing Native Spark engine details

You can view the Native Spark engine details in list and topology views.

1. Log in to the watsonx.data cluster and go to the Infrastructure manager page.

2. Click on the name of the Spark engine (either from list or topology view). The Engine information window opens.

3. In the Details tab, you can view the following details:

   Viewing Native Spark engine details

   Field	Description
   Display name	The Spark engine name.
   Engine ID	The unique identifier of the Spark instance.
   Description	The description of the engine.
   Tags	The tag that is specified at the time of registering an engine.
   Default Spark version	The Spark runtime version that is used by default for any application that is submitted to the Spark engine.
   Engine Home bucket	Specify the bucket name that store the events and logs related to Spark.
   Type	The engine type (here, Spark and Gluten accelerated Spark).
   watsonx.data application endpoint	The endpoint is used at the time of application submission. To submit an application by using API, see API Docs.
   Spark engine endpoint	The Native Spark endpoint.
   Default Spark Configuration	The Spark configuration properties that are applied to any application that is submitted to the Spark engine.

You can also use private endpoints, see VPE Gateway Configuration to configure Virtual Private Endpoint Gateway in the your Virtual Private Cloud (VPC).

Editing Spark details from the console UI

You can edit the Spark details in list and topology views.

1. Log in to the watsonx.data cluster and go to the Infrastructure manager page.
2. Click the name of the Spark engine (either from list or topology view). The Engine information window opens.
3. In the Details tab, click Edit.
4. In the Display name field, enter the display name for the Spark engine.
5. In the Description field, enter the description of the engine or edit the existing description.
6. In the Tags field, select the tags from the list or start typing to define a new tag.
7. In the Default Spark version field, select the Spark runtime version that must be considered for processing the applications.
8. In the Default Spark configuration field, click Edit configuration link to update the default Spark configuration. For more information about different properties, see Available Properties.
   1. Enter the key-value pair for the Spark configuration that applies to all applications.
   2. Click Apply.
9. Click Save and click the name of the Spark engine (either from list or topology view). The Engine information window opens.

Editing Spark details by using API

Use the following curl to update the Spark engine details like tags, description, default configuration, and Spark version.

   curl --request PATCH \
     --url https://<region>.lakehouse.cloud.ibm.com/lakehouse/api/v2/spark_engines/<spark_engine_id> \
     --header 'Authorization: Bearer <token>' \
     --header 'Content-Type: application/merge-patch+json' \
     --header 'AuthInstanceID: <crn_instance>' \
     --data '{
     "engine_details": {
       "default_config": <map_of_spark_properties>,
       "default_version": "<spark_version>"
     }
   }'

Following are the details of the parameter values to be used in the curl command:

Example:

   curl --request PATCH \
     --url https://<region>.lakehouse.cloud.ibm.com/lakehouse/api/v2/spark_engines/<spark_engine_id> \
     --header 'Authorization: Bearer <token>' \
     --header 'Content-Type: application/merge-patch+json' \
     --header 'AuthInstanceID: <crn_instance>' \
     --data '{
     "engine_details": {
       "default_config": {
         "spark.driver.cores": "1",
         "spark.driver.memory": "4g"
       },
       "default_version": "3.4"
     }
   }'

To add new properties to the default_config parameter or to update existing properties, specify the property name and value. To delete a property, specify the property name and the set the value to NULL.

Related API

For information on related API, see

• List all spark engines
• Get spark engine
• Update spark engine