Upgrading from logging agent version 2 to logging agent version 3

If your Kubernetes cluster version is 1.9+ or Red Hat OpenShift 4.5+, you can upgrade the logging agent to version 3 and run the agent as non-root.

As of 28 March 2024 the IBM Log Analysis and IBM Cloud Activity Tracker services are deprecated and will no longer be supported as of 30 March 2025. Customers will need to migrate to IBM Cloud Logs, which replaces these two services, prior to 30 March 2025. For information about IBM Cloud Logs, see the IBM Cloud Logs documentation.

Version 3 of the logging agent includes a number of features and enhancements, depending on the release level, including:

Running as a non-root user.
Configuring how existing files are handled on startup. For example, whether they are read from the beginning or end of the file.
The ability to monitor journald log files.
Logging Kubernetes events.
Enhancement of log file metadata with Kubernetes labels.
Enhanced log inclusion and exclusion rules using regex patterns.
Redaction of log lines using regex patterns.

When you upgrade the version of the agent, some logs may not be collected or duplicated depending on the period of time that you take to delete the current logging agent and deploy a new version of the logging agent in the cluster.

Complete the following steps from the command line to upgrade the logging agent version that is deployed in your Kubernetes cluster:

Step 1. Determine the latest available version of the agent

To see all the currently installed agent versions, run the following commands:

ibmcloud cr login
ibmcloud cr region-set global
ibmcloud cr images --restrict ext/logdna-agent

Make note of the image you want to update. The image will be named similar to 3.2.0-20210629.b52135aba711. You will need this information for Step 6.

Do not use the latest or stable versions.

The LogDNA release notes lists all LogDNA enhancements and fixes, including enhancements and fixes to the LogDNA Agent.

Step 2. Set the cluster context

Open a terminal to log in to IBM Cloud.
```
ibmcloud login -a cloud.ibm.com
```
List the clusters to find out in which region and resource group the cluster is available.
```
ibmcloud ks clusters
```
Set the resource group and region.
```
ibmcloud target -g RESOURCE_GROUP -r REGION
```
Where

RESOURCE_GROUP is the name of the resource group where the cluster is available, for example, default.

REGION is the region where the cluster is available, for example, us-south.
Set the cluster where you want to configure logging as the context for this session.
```
ibmcloud ks cluster config --cluster <cluster_name_or_ID>
```

Step 3. Backup the YAML file of the logging agent that is deployed in the cluster

Run the following command to backup the YAML file of the logging agent that is currently deployed in your cluster:

kubectl get ds logdna-agent -o yaml --namespace=<namespace> > logdna-agent-<clusterName>-<date>.yaml

Where

clusterName is the name of your cluster
date is the current date
namespace is the namespace where you deployed your agent. This is normally ibm-observe.

Step 4. Create the namespace ibm-observe

This step is only required if you installed the prior logging agent in a namespace that is different from ibm-observe.

To check if the namespace exists, run the following command:

kubectl get namespaces

Verify the ibm-observe namespace is active.

If you need to create the namespace, run the following command:

kubectl create namespace ibm-observe

Step 5. Delete the logging agent version 2

If you are running the logging agent 2.0 or earlier, do the following:

Delete the old agent by running the following command:
```
kubectl delete daemonset.apps/logdna-agent
```
Delete the secret in the default namespace by running the following command:
```
kubectl delete secret logdna-agent-key
```
Make sure you change to the default context before running this command.

If you are running the logging agent 2.1 or later, delete the old agent by running the following command:

kubectl delete -f <YOUR_FILE_NAME>

Where YOUR_FILE_NAME is the name of the yaml file you used to deploy the old agent. This is the file you obtained in [Step 3](# upgrade_log_analysis_agent_3_step3).

Step 6. Preparing the version 3 yaml file to run the agent as non-root

When you deploy a logging agent version 3 by using the yaml that is provided, you configure the agent with the feature stateful lookback enabled. This feature helps avoid log loss or duplication of data when the agent restarts. Stateful lookback requires the logging agent to run as root.

If you run in a secured environment or a highly regulated environment, the logging agent should run as non-root. To do this, you must edit the yaml you must edit the yaml file that is provided. Running as non-root requires additional configuration on the init container.

Complete the following steps to modify the yaml so that the logging agent runs as non-root:

Download the new version 3 yaml file. Starting with version 3, yaml files are stored in a version-specific directory.

For Kubernetes clusters, get the yaml file from the following location:

Public endpoint: https://assets.<REGION>.logging.cloud.ibm.com/clients/logdna-agent/<VERSION>/agent-resources.yaml

Private endpoint: https://assets.<REGION>.logging.cloud.ibm.com/clients/logdna-agent/<VERSION>/agent-resources-private.yaml

For OpenShift clusters, get the yaml file from the following location:

Public endpoint: https://assets.<REGION>.logging.cloud.ibm.com/clients/logdna-agent/<VERSION>/agent-resources-openshift.yaml

Private endpoint: https://assets.<REGION>.logging.cloud.ibm.com/clients/logdna-agent/<VERSION>/agent-resources-openshift-private.yaml

Where

<REGION> indicates the region where the logging instance is available. For more information about regions, see Locations.

<VERSION> indicates the version of the agent that you want to deploy. For more information on versions, see Getting information about Kubernetes logging agent images.

For example:
```
wget https://assets.us-south.logging.cloud.ibm.com/clients/logdna-agent/3.2.0/agent-resources-private.yaml -o agent-resources-private.yaml
```
You cannot use a version 1 or 2 yaml file. If you do, the agent will not run and you will get the following error in the agent log: kc logs logdna-agent-5sxhb -n ibm-observe standard_init_linux.go:211: exec user process caused "operation not permitted".
Edit the downloaded yaml file and make the following changes:
- Change the image tag to specify the image you obtained in Step 1. For example:
```
image: icr.io/ext/logdna-agent:3.2.0-20210629.b52135aba711
```
  Do not change any other instances of the version number in the yaml file.
- In the securityContext section, add the following lines to run the agent as non-root:
```
securityContext:
      # Add these 2 lines to run as non-root
      runAsUser: 5000
      runAsGroup: 5000
```
  This section needs to be in the spec/containers/name branch at the same indentation level as capabilities.
- Add any custom tags from your version 2 yaml file to the LOGDNA_TAGS.
- Save the updated yaml file as logging-agent-resources-private.yaml.

Step 7. Deploy the logging agent version 3 in the cluster

Install the agent by running the following command:

kubectl create -f <logging-agent-resources yaml file>

For example:

kubectl create -f logging-agent-resources-private.yaml

Step 8. Verify that the logging agent is deployed successfully

To verify that the logging agent is deployed successfully, run the following command:

kubectl get pod -n ibm-observe

The deployment is successful when you see one or more logging pods.

The number of logging pods equals the number of worker nodes in your cluster.
All pods must be in a Running state.