Creating an IBM Cloud Logs tenant in IBM® Cloud Logs Routing

Before you begin

Be sure that you have completed the following steps:

1. Review About IBM® Cloud Logs Routing to understand concepts.

2. Review the getting started to understand configuration steps.

3. Install all prerequisite tools as described in the getting started.

4. Set up permissions to manage targets in the account. For more information, see Setting up IAM permissions for managing tenants.

5. To create a target by using the API, check that you can connect to IBM Cloud Logs Routing by using the management API. For more information, see Enabling connectivity to manage tenants in IBM® Cloud Logs Routing.

Retrieving the IAM bearer token

You must get an IBM Cloud® Identity and Access Management (IAM) access token to authenticate your requests to the IBM® Cloud Logs Routing service. For more information, see Retrieving an access token.

For example, you can retrieve your IAM bearer token and export it as an environment variable by running the following CLI command:

export IAM_TOKEN=`ibmcloud iam oauth-tokens --output json | jq -r '.iam_token'`

Creating a service to service authorization

You must define a service to service (S2S) authorization between IBM Cloud Logs and IBM® Cloud Logs Routing so the IBM® Cloud Logs Routing service can send logs to your tenant.

Run the following command:

ibmcloud iam authorization-policy-create logs-router logs Sender

Retrieving your IBM Cloud Logs information

To onboard as an IBM Cloud Logs tenant, you must supply information about the destination where you want logs delivered. You need to supply the following information for your IBM Log Analysis instance:

• The instance CRN
• The ingress endpoint and port

To obtain the instance CRN for the IBM Cloud Logs instance, run this command:

ibmcloud resource service-instances --service-name logs --long --output JSON

This command lists all of your IBM Cloud Logs instances. You need the following fields:

• ID field: Contains the instance's CRN.
• extensions.external_ingress: Contains the ingress endpoint
• extensions.virtual_private_endpoints.endpoints.ports.port_max: Contains the port value.

The following sample shows some of the information that you can get:

{
        "guid": "xxxxxxx",
        "id": "crn:v1:bluemix:public:logs:eu-gb:a/xxxxxxx:b951ceb1-06a4-4a1d-93d5-eaf47cedb86d::",
        "url": "/v2/resource_instances/b951ceb1-06a4-4a1d-93d5-eaf47cedb86d",
        "created_at": "2024-03-08T04:58:17.137066137Z",
        "updated_at": "2024-03-08T04:58:39.870280882Z",
        "deleted_at": null,
        "name": "My instance",
        "region_id": "eu-gb",
        "account_id": "xxxxxxx",
        "resource_plan_id": "0c1a62c7-a433-44df-a63e-b3fe8e629a94",
        "resource_group_id": "8f0e17c350d091415246eca5cd5239b1",
        "crn": "crn:v1:bluemix:public:logs:eu-gb:a/xxxxxxx:b951ceb1-06a4-4a1d-93d5-eaf47cedb86d::",
        "extensions": {
            "external_api": "xxxxxxx.api.eu-gb.logs.cloud.ibm.com",
            "external_dashboard": "https://dashboard.eu-gb.logs.appdomain.cloud/xxxxxxx/#/dashboard",
            "external_ingress": "xxxxxxx.ingress.eu-gb.logs.cloud.ibm.com",
            "virtual_private_endpoints": {
                "dns_domain": "private.eu-gb.logs.cloud.ibm.com",
                "dns_hosts": [
                    "xxxxxxx.ingress",
                    "xxxxxxx.api",
                    "dashboard"
                ],
                "endpoints": [
                    {
                        "ip_address": "10.12.93.39",
                        "zone": "eu-gb-1"
                    },
                    {
                        "ip_address": "10.12.94.38",
                        "zone": "eu-gb-2"
                    },
                    {
                        "ip_address": "10.12.95.38",
                        "zone": "eu-gb-3"
                    }
                ],
                "origin_type": "vpc",
                "ports": [
                    {
                        "port_max": 443,
                        "port_min": 443
                    }
                ]
            }
        },
        "create_time": 0,
        "created_by": "IBMid-xxxxxxx",
        "state": "active",
        "type": "service_instance",
        "resource_id": "cd515180-d78a-11ec-b396-db7d306c4f73",
        "dashboard_url": "https://cloud.ibm.com/observe/logging/xxxxxxx/overview",
        "allow_cleanup": false,
        "locked": false,
        "last_operation": {
            "type": "create",
            "state": "succeeded",
            "description": "service instance Provision succeeded",
            "updated_at": null,
            "cancelable": false
        },
        "account_url": "",
        "resource_plan_url": "",
        "resource_bindings_url": "/v2/resource_instances/xxxxxxx/resource_bindings",
        "resource_aliases_url": "/v2/resource_instances/xxxxxxx/resource_aliases",
        "siblings_url": "",
        "target_crn": "crn:v1:staging:public:globalcatalog::::deployment:0c1a62c7-a433-44df-a63e-b3fe8e629a94%3Aeu-xxx"
    }

Creating a tenant by using the API

Submit the create (onboard) request to IBM Cloud Logs Routing by using the appropriate management endpoint URL for the correct region.

curl -X POST https://<MANAGEMENT-API-ENDPOINT>:443/v1/tenants \
-H "Content-Type: application/json" \
-H "Authorization: ${IAM_TOKEN}" \
-H 'IBM-API-Version: DATE' \
--data "{
    "name": "TENANT_NAME",
    "targets": [
        {
            "log_sink_crn": "CLOUD_LOGS_INSTANCE_CRN",
            "name": "TARGET_NAME",
            "parameters": {
                "host": "CLOUD_LOGS_INSTANCE_INGRESS_ENDPOINT",
                "port": CLOUD_LOGS_INSTANCE_TARGET_PORT
            }
        }
    ]
}'

Where

• TENANT_NAME: Name of the tenant. The name must be unique across tenants for this account and can be up to 35 characters long. The value can only contain these characters: a-z,0-9,-./
• TARGET_NAME: Name of the target destination. The name must be unique across all targets for this tenant and can be up to 35 characters long. The value can only contain these characters: a-z,0-9,-./
• MANAGEMENT-API-ENDPOINT is the IBM® Cloud Logs Routing endpoint in the region where you plan to collect logs. For more information, see Endpoints.
• CLOUD_LOGS_INSTANCE_CRN is the CRN of the IBM Cloud Logs instance.
• CLOUD_LOGS_INSTANCE_INGRESS_ENDPOINT is the full qualified ingress endpoint for the destination of logs.
• DATE: Specify the current date to request the latest version of the API. The valid format is YYYY-MM-DD. Any date up to the current date can be provided.

The following example shows the creation of a tenant to IBM Cloud Logs Routing in the eu-es region by using a VPE, and specifying target information for a IBM Cloud Logs instance that is also in eu-es:

curl -X POST "https://management.private.eu-es.logs-router.cloud.ibm.com/v1/tenants" --header 'Authorization: Bearer TOKEN' --header 'Content-Type: application/json' --header 'IBM-API-Version: 2024-03-01' --data '{
    "name": "tenant",
    "targets": [
        {
            "log_sink_crn": "crn:v1:bluemix:public:logs:eu-es:a/91d1d1e42f9b684df920bbd06461c222:743e3b4f-72bc-4669-9fa6-0e6621d0b232::",
            "name": "my-log-sink",
            "parameters": {
                "host": "743e3b4f-72bc-4669-9fa6-0e6621d0b232.ingress.eu-es.logs.cloud.ibm.com",
                "port": 443
            }
        }
    ]
}'

If the creation (onboarding) request was successful, a response that contains your tenant metadata is returned. For example,

{
    "id": "3517d2ed-9429-af34-ad52-34278391cbc8:",
    "created_at": "2023-10-20T18:30:00.143156Z",
    "updated_at": "2023-10-20T18:30:00.143156Z",
    "crn": "crn:v1:bluemix:public:logs-router:eu-de:a/3516b8fa0a174a71899f5affa4f18d78:3517d2ed-9429-af34-ad52-34278391cbc8::",
    "name": "tenant",
    "etag": "\"822b4b5423e225206c1d75666595714a11925cd0f82b229839864443d6c3c049\"",
    "targets": [
    {
        "id": "86432b-66a6-df12-7003-888a21a2b3",
        "log_sink_crn": "crn:v1:bluemix:public:logs:eu-gb:a/91d1d1e42f9b684df920bbd06461c222:743e3b4f-72bc-4669-9fa6-0e6621d0b232::",
        "name": "my-log-sink",
        "etag": "\"c3a43545a7f2675970671ac3a57b8db067a1866b2222e1b950ee8da612e347c6\"",
        "type": "logs",
        "created_at": "2023-10-20T18:30:00.143156Z",
        "updated_at": "2023-10-20T18:30:00.143156Z",
        "parameters": {
            "host": "743e3b4f-72bc-4669-9fa6-0e6621d0b232.ingress.eu-gb.logs.cloud.ibm.com",
            "port": 443
        }
    }
    ]
}

Setting application and subsystem name

If the subsystemName value is not specified, it is set to the container name if the container name is included in the metadata (for example, using the kubernetes filter). The applicationName will be set to namespace_name if it is not set and included in the metadata.

It is possible to add static values for the application and subsystem name. This can be done when installing the agent. See Managing the IBM® Cloud Logs Routing agent for IBM Cloud Kubernetes Service clusters or Managing the IBM® Cloud Logs Routing agent for Red Hat OpenShift on IBM Cloud clusters for more information.

In addition, a dynamic value for the application and subsystem name can be set depending on the log line. The config map of the agent has to be modified to create the custom application name. The same steps can be used to create a custom subsystem name.

For a JSON formatted message, do the following:

1. Add the applicationName to the log line, for example {"level":"info", "msg":"test message", "applicationName":"my-application"}

2. Download the YAML configuration file for your environment.

3. Open the downloaded YAML configuration file in a text editor.

4. Add a json parser to extract those fields:

   [PARSER]
    Name json
    Format json
   

5. Add the parser using a filter:

   [FILTER]
      Name parser
      Match *
      Key_Name message
      Parser json
   

6. Extract the application name using the nest filter:

   [FILTER]
      Name nest
      Match *
      Operation lift
      Nested_under message
      Wildcard applicationName
   

   __applicationName__ must be at the top level since nested fields will not be extracted.

In this example, the applicationName for the log line would be my-application.

Add the following filters after the the configuration changes made in the previous steps. This filter will add default values for all log lines not containing an applicationName or subsystemName.

  filter-add-ICL-meta-data.conf: |
    [FILTER]
        Name modify
        Match *
        Condition Key_Does_Not_Exist subsystemName
        Add subsystemName REPLACE_SUBSYSTEM_NAME

    [FILTER]
        Name modify
        Match *
        Condition Key_Does_Not_Exist applicationName
        Add applicationName REPLACE_APPLICATION_NAME