Accessing the Spark history server

The Spark history server is a web UI where you can view the status of running and completed Spark applications on a watsonx.datainstance. If you want to analyze how different stages of your Spark application performed, you can view the details in the Spark history server UI.

1. Log in to watsonx.data console.

2. From the navigation menu, select Infrastructure manager.

3. Click the name of the Spark engine (from list view or topology view). Engine information window opens.

4. In the Spark history tab, click Start history server.

5. By default, the Spark history server consumes 1 CPU core and 4 GiB of memory while it is running. If you want your Spark history server to use more resources, select the Cores and Memory required for the server and click Start. The history server starts and the status is displayed as STARTED.

6. Click View Spark history. The History Server page opens.

7. The History Server page includes the following functionalities:

   • View the list of completed Spark application and details such as the application ID, duration and event log for each application.
   • Click Download link from the Event Log field to download the events log information for each application.
   • To view the details of individual application, click the individual application ID link. The Spark Jobs page opens. This page displays details such as the different stages of execution, the storage used, the Spark environment and executor (memory and driver) details.

8. You can stop the history server when you no longer need it to release unnecessary resources. To do that, go to Spark history tab and click Stop history server. Click Stop on the confirmation message.