Accessing Spark logs for ingestion jobs

This topic provides the steps required to locate and view Spark logs associated with submitted ingestion jobs within IBM® watsonx.data. By accessing these logs, you can gain valuable insights into the execution details, potential error messages related to the ingestion process, and troubleshooting ingestion jobs.

Before you begin

For a Failed or Finished status, you must start the Spark history server to get the ingestion log details. For more information, see Accessing the Spark history server.

Procedure

Log in to watsonx.data console.
From the navigation menu, select Data manager and click Ingestion history tab.
Click a specific Job log to get the details and logs of that ingestion job.
Click Spark application history to get log details based on the status of the ingestion job as follows:

Starting: If the ingestion job is in Starting status, Spark application history is disabled.

Running: If the job status is Running, clicking Spark application history will generally take you to the Spark UI. You can monitor the ingestion job progress, resource usage, and other details in real-time through this web interface. This Spark UI information will be temporarily available only when the job is in the Running status.

Failed or Finished: If the job status is Failed or Finished, clicking Spark application history will navigate you to the Spark history UI. This interface provides a summary of the ingestion job execution, including its timeline, resource usage, and any error messages.

Cancelled: If the job is Cancelled, Spark application history is disabled.