Forwarding logs to instance home

Downloading the logs

To download the logs from the instance home, run the following command by using the AWS CLI. For more information about using CLI, see Configure the CLI to connect to Object Storage.

aws --endpoint-url <bucket_endpoint> s3 cp s3://<bucket_name>/<instance_id>/logs/<app_id>/ ./ --recursive

Example :

aws --endpoint-url https://s3.us-south.cloud-object-storage.appdomain.cloud/ s3 cp s3://do-not-delete-ae-bucket-a0048029-c78a-439a-a864-14fdd3b2d95b/a0048029-c78a-439a-a864-14fdd3b2d95b/logs/2cd9036a-9e4a-4b79-81ec-2d8ae96d674c/ ./ --recursive

Changing Spark Log Level

You can edit the spark log level configuration to set different log levels like INFO, DEBUG for driver and executor logs that are forwarded to instance home.

Disable the logging feature completely by specifying the following driver and (or) executor log level.

ae.spark.driver.log.level = OFF
ae.spark.executor.log.level = OFF

For more information about configuring the log level information, see Configuring Spark log level information.

By default the driver log level is WARN and the executor log level is OFF. You can view only the driver logs in your instance home, if the Spark log level configuration is not specified. To include the executor logs, change the log level to a different value than OFF. For more information, see Configuring Spark log level information.

Also, you can also choose to forward logs to IBM Log Analysis service. For more information, see Forwarding logs to IBM Log Analysis. If the driver or executor log level information is configured as 'OFF', then logs are not forwarded to IBM Log Analysis service even if it is enabled through the Logging APIs.