---
name: AnalyticsEngine-log_frwd_log_anlys
title: Forwarding logs to IBM Cloud Logs
description: ''
last-updated: 2024-01-31
---

> ## Documentation Index
> The table of contents for this documentation set is at https://cloud.ibm.com/docs/AnalyticsEngine?format=markdown
> The index for all IBM Cloud docs is at: https://cloud.ibm.com/docs/llms.txt
> Use these files to discover more information as needed.

{:new_window: target="_blank"}
{:shortdesc: .shortdesc}
{:codeblock: .codeblock}
{:screen: .screen}
{:pre: .pre}
{:external: target="_blank" .external}

# Forwarding logs to IBM Cloud Logs
{: #platform-logs}

*Platform logs* are logs that are exposed to users by various services on the IBM Cloud platform. IBM Analytics Engine supports forwarding logs from the IBM Analytics Engine service to an IBM Cloud Logs instance that you have enabled to receive platform logs.


- Platform logs are regional.

    You can monitor logs from IBM Analytics Engine on IBM Cloud in the region where the service is available.

- You can configure only one instance of IBM Cloud Logs per region and per IBM Cloud account to collect platform logs in that location.

    You can have multiple IBM Cloud Logs instances in a location. However, only one instance in a location (region) can be configured to receive logs from IBM Cloud services such as IBM Analytics Engine in that IBM Cloud location.

    Also, the logs that are generated by an IBM Analytics Engine instance belonging to an account are forwarded to an IBM Cloud Logs instance under the same IBM Cloud account.

To check for the IBM Cloud Logs service that is configured to receive platform logs, see the [Observability dashboard](https://cloud.ibm.com/observe/logging).


## Prerequisite to enable platform logging
{: #enable-logging-from-dashboard}

To view IBM Analytics Engine platform logs, you must use the Observability dashboard in IBM Cloud to configure platform logging. See [Configuring platform logs through the Observability dashboard](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-config_svc_logs&format=markdown#config_svc_logs_ui) for the steps that you need to follow to enable logging through the Observability dashboard.



## Logging APIs
{: #logging-apis}


After you have enabled platform logging through the Observability dashboard, proceed with the steps in the following sections to learn how to use the API endpoints to enable logging, disable logging, and view the logging configuration.

The logging API calls require the GUID of the service instance. If you didn't make a note of the GUID, see [Retrieving the GUID of a serverless instance](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-instance-details&format=markdown).

Now, two logging API endpoints are exposed:
- `log_forwarding_config`
- `logging`. This API endpoint is deprecated and will be discontinued soon. Although you may use this endpoint, you should start using the `log_forwarding_config` endpoint.

## Enabling log forwarding from IBM Analytics Engine
{: #enable-logging-from-iae}

To enable forwarding IBM Analytics Engine platform logs to IBM Cloud Logs, invoke the following API against an existing Analytics Engine instance.

There are two methods that you can use to enable forwarding logs:
- *Default*: In this case you only put a request for enabling log forwarding, and the logs related to the default log source or sources (`spark-driver`), will start getting forwarded. You can do this in one of the following ways:

    - By using the `log_forwarding_config` API endpoint:
        ```sh
        curl -X PUT https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/log_forwarding_config -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d '{"enabled":true}'
        ```
        {: codeblock}

    - By using the `logging` API endpoint (deprecated):
        ```sh
        curl -X PUT https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/logging -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d '{"enable":true}
        ```
        {: codeblock}

- *Customized*: With this option, you get more control over the log forwarding configuration. For example, you can specify the sources of the logs you want to get forwarded. You can also choose to associate some tags with the logs, which can later be used in IBM Cloud Logs as search keywords to narrow down the search on the applicable logs. You can do this in one of the following ways:

    - By using the `log_forwarding_config` API endpoint:
        ```sh
        curl -X PUT https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/log_forwarding_config -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d ' {"enabled":true,"sources": ["spark-driver", "spark-executor"],"tags": ["<tag_1>", "<tag_2>",...]} '
        ```
        {: codeblock}

    - By using the `logging` API endpoint (deprecated):
        ```sh
        curl -X PUT https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/logging -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d ' {"enable":true,"components": ["spark-driver", "spark-executor"],"tags": ["<tag_1>", "<tag_2>",...]} '
        ```
        {: codeblock}

## Disabling platform logging from IBM Analytics Engine
{: #disable-loggingfrom-iae}

Disable forwarding IBM Analytics Engine platform logs to IBM Cloud Logs, in one of the following ways.

- By using the `log_forwarding_config` API endpoint:

    ```sh
    curl -X PATCH https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/log_forwarding_config -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d '{"enabled":false}'
    ```
    {: codeblock}

- By using the `logging` API endpoint (deprecated):

    ```sh
    curl -X PATCH https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/logging -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d '{"enable":false}'
    ```
    {: codeblock}

## Viewing the log forwarding configuration of an IBM Analytics Engine instance
{: #log-config}

View the current logging configuration of an IBM Analytics Engine instance, in one of the following ways:

- By using the `log_forwarding_config` API endpoint:

    ```sh
    curl -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/log_forwarding_config -H "Authorization: Bearer $TOKEN"
    ```
    {: codeblock}

- By using the `logging` API endpoint (deprecated):

    ```sh
    curl -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/logging -H "Authorization: Bearer $TOKEN"
    ```
    {: codeblock}


## Examples of IBM Analytics Engine log searches in the IBM Cloud Logs dashboard
{: #analyzing-logs}

You can narrow down the search for the platform logs related to IBM Analytics Engine as follows:

1. Click `Sources` on IBM Cloud Logs dashboard and choose `ibmanalyticsengine` to get the logs from IBM Analytics Engine.
1. To further narrow down your search, you can enter and search for any keywords related to your Spark application in the search window.

### Examples of searches
{: #analyzing-logs-1}

This section lists examples of how you can search for logs. The examples all use the following sample values:

- IBM Analytics Engine instance ID: `4308f7a6-2cfd-4d7e-b7a7-678e71a27957`
- Spark application ID: `5438585f-6413-4bb2-9251-8162d34f3dc3`
- Sources for which log forwarding is enabled: `spark-driver` and `spark-executor`
- Associated tags: `prod` and `us-south`

The following image shows you an example of the returned search results in the IBM Cloud Logs dashboard. At the bottom of the image you can see the search entry field where you enter your search queries. In the example, the application ID `5438585f-6413-4bb2-9251-8162d34f3dc3` and the source `spark-driver` were entered.




Examples of searches:

- To retrieve the logs for all Spark applications submitted in a specific IBM Analytics Engine instance, pass the instance ID in the search window:
    ```bash
    4308f7a6-2cfd-4d7e-b7a7-678e71a27957
    ```
- To fetch the logs for a specific Spark application, you can pass:

    - A combination of the IBM Analytics Engine instance ID and the Spark application ID:
        ```bash
        4308f7a6-2cfd-4d7e-b7a7-678e71a27957 5438585f-6413-4bb2-9251-8162d34f3dc3
        ```
    - Only the application ID:
        ```bash
        5438585f-6413-4bb2-9251-8162d34f3dc3
        ```
- To search the Spark master logs of an application, you can pass:

    - A combination of the IBM Analytics Engine instance ID, the Spark application ID and the keyword `spark-driver`:
        ```bash
        4308f7a6-2cfd-4d7e-b7a7-678e71a27957 5438585f-6413-4bb2-9251-8162d34f3dc3 spark-driver
        ```
    - A combination of the application ID and the keyword `spark-driver`:
        ```bash
        5438585f-6413-4bb2-9251-8162d34f3dc3 spark-driver
        ```
- To search by multiple log source names, you can pass:

    - A combination of the IBM Analytics Engine instance ID, the Spark application ID and the source names:
        ```bash
        4308f7a6-2cfd-4d7e-b7a7-678e71a27957 5438585f-6413-4bb2-9251-8162d34f3dc3 (spark-driver OR spark-executor-2)
        ```
    - A combination of the application ID and the log source names:
        ```bash
        5438585f-6413-4bb2-9251-8162d34f3dc3 (spark-driver OR spark-executor-2)
        ```
- To get the logs related to all the applications running in a particular instance within the `us-south` region and the `prod` environment, you can enter the following in the search field:
    ```bash
    4308f7a6-2cfd-4d7e-b7a7-678e71a27957 "prod us-south"
    ```