Overview
When you submit applications or Spark workloads in IBM Analytics Engine, you can monitor the application execution. IBM Analytics Engine allows to debug the application errors or trace the application execution by monitoring the application-generated logs and the Spark-generated logs. Log analysis together with monitoring feature of the Spark History page provides better troubleshooting of the applications.
IBM Analytics Engine allows you to forward application logs to the instance home or (and) to IBM Log Analysis service.
Default Behavior
When you run applications in IBM Analytics Engine, the application logs (only Spark Driver WARN logs) are forwarded to IBM Cloud Object Storage by default. You can access the log information from your instance home.
For more information, see instance home.
Forwarding logs to instance home
Application Logs are forwarded to the Analytics Engine instance home by default. You can access the log information from the IBM Cloud Object Storage bucket. You can download the log file for any specific application from the Cloud Object Storage bucket for recording, sharing, and debugging purposes. If you want to change the default behavior to include executor logs or INFO logs for driver or executor, you need to change the configurations. For more information, see Forwarding logs to instance home.
Forwarding logs to IBM Log Analysis
IBM Log Analysis service allows you to view indexed logs, enable full-text search through all generated messages and query based on specific fields. Enabling the log forwarding feature forward the logs to IBM Log Analysis service
(in addition to instance home). Only the Spark Driver WARN logs are forwarded by default. To change the behavior to include executor logs, enable it by modifying the payload in the log forwarding API.For more
information, see Forwarding logs to IBM Log Analysis
Changing the Spark Log Levels
To change the default behavior to forward logs at different log levels (INFO, DEBUG, and so on), see Configuring Spark log level information.
Disabling Logging feature
To disable sending log information to both instance home and IBM Log Analysis, set the driver and executor log level configuration to OFF. For more information see, Configuring Spark log level information.