--- name: AnalyticsEngine-spark-history-serverless title: Spark history server description: '' last-updated: 2023-01-05 --- > ## Documentation Index > The table of contents for this documentation set is at https://cloud.ibm.com/docs/AnalyticsEngine?format=markdown > The index for all IBM Cloud docs is at: https://cloud.ibm.com/docs/llms.txt > Use these files to discover more information as needed. {:new_window: target="_blank"} {:shortdesc: .shortdesc} {:note: .note} {:codeblock: .codeblock} {:screen: .screen} {:pre: .pre} {:external: target="_blank" .external} # Spark history server {: #spark-history-serverless} The Spark applications that are submitted on an IBM Analytics Engine instance forward their Spark events to the Object Storage bucket that was defined as the *instance home*. The Spark history server provides a Web UI to view these Spark events. The Web UI helps you to analyze how your Spark applications ran by displaying useful information like: - A list of the stages that the application goes through when it is run - The number of tasks in each stage - The configuration details such as the running executors and memory usage See the [Spark History server documentation](https://spark.apache.org/docs/latest/monitoring.html#viewing-after-the-fact){: external} for more details. You can disable forwarding Spark events from a Spark application by setting the property `spark.eventLog.enabled` to `false` in the Spark application configuration. {: note} ## Starting and stopping the Spark history server {: #spark-history-serverless-1} Before accessing the Spark history server, you need to start the server. When you no longer need it, you should stop the server. You will be charged for the CPU cores and memory consumed by the Spark history server while it is running. The Spark history server can be started and stopped by using: - The [Analytics Engine REST API](#rest-api) - The [Analytics Engine instance UI](#iae-ui) ### Analytics Engine REST API {: #rest-api} You can use the Analytics Engine REST API: 1. To view the status of the Spark history server ```sh curl "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines//spark_history_server" --header "Authorization: bearer " ``` {: codeblock} 1. To start the Spark history server ```sh curl --location --request POST "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines//spark_history_server" --header "Authorization: bearer " ``` {: codeblock} 1. To stop the Spark history server ```sh curl --location --request DELETE "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines//spark_history_server" --header "Authorization: bearer " ``` {: codeblock} ### Analytics Engine instance UI {: #iae-ui} You can use the Analytics Engine instance UI: ​ 1. To view the Spark history server status: 1. Open your resource list on [IBM Cloud](https://cloud.ibm.com/resources). 2. Click **Services and software** and select your instance to open the details page. 3. Select the **Spark history** tab. The current status of the server is shown on this page. If the status of the Spark history server is set to `Started`, you can also click **View Spark history** to launch the Web UI of the Spark history server in a new browser tab. 1. To start the Spark history server: 1. On the `Spark history` page, click **Start history server**. 2. Choose the server configuration and click **Start** to start the Spark history server. 1. To stop the Spark history server: 1. On the `Spark history` page, click **Stop history server**. ## Opening the Spark history server Web UI {: #spark-history-serverless-6} You can open the Spark history Web UI by opening the instance details page of your Analytics Engine service instance, switching to the **Spark history** tab and clicking **View Spark history**. Alternatively, the Spark history server Web UI URL can be obtained through a service endpoint that is made available to you as a service key (also known as a service credential). See [Retrieving service endpoints](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-endpoints-serverless&format=markdown). Ensure that the Spark history server is running before you open the Web UI. Log links under the Stages and Executors tabs of the Spark history server UI will not work as logs are not preserved with the Spark events. To review the task and executor logs, enable platform logging. For details, see [Configuring and viewing logs](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-viewing-logs&format=markdown). {: note} To view older applications on Spark history server UI, copy the spark-events from old path to new path by using the command: ibmcloud cos object-copy --bucket --key /spark-events/ --copy-source /spark-events/ {: important} ## Accessing the Spark history server REST API {: #spark-history-serverless-7} In addition to the web UI, the Spark history server also provides a REST API which can be queried to view the Spark events generated by your Spark applications. The Spark history server REST API is available as a service endpoint in a service key (also known as service credential). See [Retrieving service endpoints](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-endpoints-serverless&format=markdown). When you invoke the Spark history server REST API, you must specify your IAM token as a bearer token in the Authorization header. E.g. ```sh curl --location --request GET 'https://spark-console.us-south.ae.cloud.ibm.com/v3/analytics_engines//spark_history_api/v1/applications?status=completed' \ --header 'Authorization: Bearer ' ``` {: codeblock} See the [Spark history server REST API documentation](https://spark.apache.org/docs/latest/monitoring.html#rest-api){: external} for more details. ## Customizing the Spark history server {: #spark-history-serverless-2} By default, the Spark history server consumes 1 CPU core and 4 GiB of memory while it is running. If you want your Spark history server to use more resources, you can set the following properties in your Analytics Engine instance default configurations: - `ae.spark.history-server.cores` for the number of CPU cores - `ae.spark.history-server.memory` for the amount of memory ### Updating the CPU cores and memory settings using the REST API {: #spark-history-serverless-3} You can update the CPU cores and memory settings using the Analytics Engine REST API as follows: ```sh curl --location --request PATCH "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines//default_configs" \ --header "Authorization: bearer " \ --header 'Content-Type: application/json' \ --data-raw '{ "ae.spark.history-server.cores": "2", "ae.spark.history-server.memory": "8G" }' ``` {: codeblock} Only a pre-defined list of Spark driver and executor vCPU, and memory size combinations are supported. See [Supported Spark driver and executor vCPU and memory combinations](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-limits&format=markdown#cpu-mem-combination). ### Additional customisations {: #spark-history-serverless-4} You can customize the Spark history server further by adding properties to the default Spark configuration of your Analytics Engine instance. See standard [Spark history configuration options](https://spark.apache.org/docs/latest/monitoring.html#spark-history-server-configuration-options){: external}. As this is a managed offering, you can't customize all of the standard Spark configuration options. {: note} For a list of the supported configurations and default values for settings, see [Default Spark configuration](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-serverless-architecture-concepts&format=markdown#default-spark-config). ## Best practices {: #spark-history-serverless-5} Always stop the Spark history server when you no longer need to use it. Bear in mind that the Spark history server consumes CPU amd memory resources continuously while its state is `Started`.