Resetting the ingestion key that is used by a Kubernetes cluster

If the ingestion key that you use to forward logs from a cluster to an IBM Log Analysis instance in the IBM Cloud is compromised, you must reset the key and update the Kubernetes cluster configuration to use the new ingestion key.

As of 28 March 2024 the IBM Log Analysis and IBM Cloud Activity Tracker services are deprecated and will no longer be supported as of 30 March 2025. Customers will need to migrate to IBM Cloud Logs, which replaces these two services, prior to 30 March 2025.

Before you begin

Work in a supported region. Note: You can work with a Kubernetes cluster that is located in the same region or in a different region.

Read about IBM Log Analysis. For more information, see About.

Use a user ID that is a member or an owner of an IBM Cloud account. To get an IBM Cloud user ID, go to: Registration.

The IBM Log Analysis instance is provisioned in the default resource group.

To complete the steps in this tutorial, your IBMID must have assigned IAM policies for each of the following resources:

Table 1. List of IAM policies required to complete the tutorial
Resource	Scope of the access policy	Roles	Region	Information
Resource group Default	Resource group	Viewer	us-south	This policy is required to allow the user to see service instances in the Default resource group.
IBM Log Analysis service	Resource group	Editor Manager	us-south	This policy is required to allow the user to reset the ingestion key.
Kubernetes cluster instance	Resource	Editor	us-south	This policy is required to delete and configure the secret and the logging agent in the Kubernetes cluster.

For more information about the IBM Cloud® Kubernetes Service IAM roles, see User access permissions.

Install the IBM Cloud CLI and the Kubernetes CLI plug-in. For more information, see Installing the IBM Cloud CLI.

Reset the ingestion key

To renew the ingestion key for an IBM Log Analysis instance by using the IBM Log Analysis Web UI, complete the following steps:

Launch the IBM Log Analysis web UI.
Click the Settings icon .
Select Organization.
Select API keys.

You can see the ingestion keys that have been created.
Select Generate Ingestion Key.

A new key is added to the list.
Delete the old ingestion key. Click the X next to the ingestion key to be deleted.

Remove any configuration in the cluster that uses the old ingestion key

Complete the following steps:

Open a terminal. Then, log in to the IBM Cloud. Run the following command and follow the prompts:
```
ibmcloud login -a cloud.ibm.com
```
Select the account where you have provisioned the IBM Log Analysis instance.
Set up the cluster environment. Run the following commands:

First, get the command to set the environment variable and download the Kubernetes configuration files.
```
ibmcloud ks cluster config --cluster <cluster_name_or_ID>
```
Note: Every time you log in to the IBM Cloud® Kubernetes Service CLI to work with clusters, you must run these commands to set the path to the cluster's configuration file as a session variable. The Kubernetes CLI uses this variable to find a local configuration file and certificates that are necessary to connect with the cluster in IBM Cloud.
Remove the secret from your Kubernetes cluster. The Kubernetes secret contains the logging ingestion key. Run the following command:
```
kubectl delete secret logdna-agent-key -n ibm-observe
```
Remove the logging agent on every worker(node) of your Kubernetes cluster. The logging agent is responsible for collecting and forwarding your logs. Run the following command:
```
kubectl delete daemonset logdna-agent -n ibm-observe
```
Verify that the logging agent is deleted successfully. Run the following command:
```
kubectl get pods -n ibm-observe
```
You should not see any logging pods.

Configure your Kubernetes cluster with the new ingestion key

To configure your Kubernetes cluster in the us-south region to forward logs to your logging instance, complete the following steps from the command line:

Open a terminal. Then, log in to the IBM Cloud. Run the following command and follow the prompts:
```
ibmcloud login -a cloud.ibm.com
```
Select the account where you have provisioned the IBM Log Analysis instance.
Set up the cluster environment. Run the following commands:

First, get the command to set the environment variable and download the Kubernetes configuration files.
```
ibmcloud ks cluster-config <cluster_name_or_ID>
```
Add a secret to your Kubernetes cluster. Run the following command:
```
kubectl create secret generic logdna-agent-key --from-literal=logdna-agent-key=LOGDNA_INGESTION_KEY_FOR_YOUR_INSTANCE -n ibm-observe
```
The LOGDNA_INGESTION_KEY_FOR_YOUR_INSTANCE shows the logging ingestion key for your instance.

The Kubernetes secret contains the logging ingestion key. The logging ingestion key is used to authenticate the logging agent with the IBM Log Analysis service. It is used to open a secure web socket to the ingestion server on the logging back-end system.
Configure the logging agent on every worker(node) of your Kubernetes cluster. Run the following command:
```
kubectl create -f https://assets.us-south.logging.cloud.ibm.com/clients/agent-resources.yaml -n ibm-observe
```
The logging agent is responsible for collecting and forwarding your logs.

The agent collects automatically logs with extension *.log and extensionless files that are located under /var/log. By default, logs are collected from all namespaces, including the kube-system.
Verify that the logging agent is created successfully and its status. Run the following command:
```
kubectl get pods -n ibm-observe
```

Launch the logging web UI

You launch the web UI from the IBM Cloud Observability dashboard.

Complete the following steps to launch the web UI:

Log in to your IBM Cloud account.
Click the Menu icon > Observability to launch the Observability dashboard.
Click Logging.

The list of IBM Log Analysis instances that are available on IBM Cloud is displayed.
Select one instance. Then, click Open dashboard.

The logging web UI opens and displays your cluster logs.

View your logs

From the logging web UI, you can view your logs as they pass through the system. You view logs by using log tailing.

With the Lite service plan, you can only tail your latest logs.

Next steps

If you want to filter cluster logs, search cluster logs, define views, and configure alerts, you must upgrade the IBM Log Analysis plan to a paid plan.