Managing the IBM® Cloud Logs Routing agent for IBM Cloud Kubernetes Service clusters

You can deploy an IBM Cloud Logs Routing agent to collect and route infrastructure and application logs from an IBM Cloud Kubernetes Service cluster to a target of your choice. Valid targets are Log Analysis instances.

Deploying the agent

Complete the following steps to deploy an agent on an IBM Cloud Kubernetes Service cluster:

Before you begin

Make sure you have access to Kubernetes cluster with permissions to create namespaces and deploy the agent.
Install the following CLIs:
- The IBM Cloud CLI to log in to the IBM Cloud and manage IBM Cloud services such as creating an API key.
- The Kubernetes CLI to manage the cluster by using kubectl commands. Learn more.
Read about Connecting to IBM® Cloud Logs Routing.

Step 1. Define the authentication method for the agent

Choose the type of identity and the authentication method for the agent. Then, create an API key if needed.

Complete the following steps:

Choose the type of identity: user, service ID, or trusted profile.

You can use a user, a service ID, or a trusted profile as the identity that is used by the agent to authenticate with the IBM® Cloud Logs Routing service.
Grant permissions for ingestion to the identity that you have chosen.

The role that is required for sending logs to IBM Cloud Logs Routing is Writer.

For more information, see Setting up IAM permissions for ingestion.
Generate an API Key for user authentication or for service ID authentication.

For authentication with trusted profiles, this step is not required.

For more information, see Generating an API Key for ingestion.

Step 2. Setting up and deploying the IBM Cloud Logs Routing agent configuration

Complete the following steps:

Access your cluster.
Configure the log collection behavior of the IBM Cloud Logs Routing agent, see Configuring whether logs are included or excluded.
Install the IBM Cloud Logs Routing agent. Use the following curl command in your terminal:
```
curl -sSL https://ibm.biz/logs-router-setup | bash -s -- \
  -v <agent_version> \
  -m <iam_auth_mode> \
  -i <trusted_profile_id> \
  -k <iam_api_key> \
  -t <cluster_type> \
  -r <region> \
  -p <ingester_target_port>
  -d <directory>
```
When sending logs to IBM Cloud Logs, you can optionally specify the following commandline options to add the application and subsystem name to each log line:
```
    -a <application_name> -s <subsystem_name>
```
(Optional) To send logs directly to IBM Cloud Logs, bypassing the IBM Cloud Logs Routing data plane, use the following option:
```
--send-directly-to-icl
```
Where:

-v

Agent version. Specify the version of the IBM Cloud Logs Routing Agent. To find the current supported versions, refer to Checking available agent versions.

-m

IAM authentication mode (TrustedProfile | IAMAPIKey).

-i

Trusted profile ID (required for TrustedProfile mode). Provide the Trusted Profile ID. When using trusted profiles, set to the ID configured in Setting up Permissions for Ingestion.

For more information on Trusted Profiles, see Creating a Trusted Profile.

-k

IAM API key (required for IAMAPIKey mode). Make sure you follow the instructions in Generating an API Key.

For more information about IAM API Keys, see Managing API Keys.

-t

Cluster type (OpenShift or Kubernetes). Specify if you are deploying the agent to an Red Hat OpenShift on IBM Cloud (OpenShift) or IBM Cloud Kubernetes Service (Kubernetes) cluster.

-r

Specify the region where the IBM Cloud Logs Routing Ingester endpoint is located (for example us-east).

-p

Target port for log ingestion. The default target port is 443, which corresponds to the ingester target port. - Ingester target port: The port must be 443 if you are connecting using a VPE gateway, or port 3443 when connecting using CSEs.

-d

Specify the directory containing the logger-agent-iks.yaml file configured in the previous step. For example, if your logger-agent-iks.yaml file is located in the /path/to/directory directory, you would simply specify the directory like this: -d /path/to/directory.

-a

Specify the application name that you want to see in your IBM Cloud Logs instance. If in the metadata, the application name defaults to namespace name. You can also use variables from the environment, for example '${NODE_NAME}'.

-s

Specify the subsystem name that you want to see in your IBM Cloud Logs instance. If in the metadata, the subsystem name defaults to container name. You can also use variables from the environment, for example '${NODE_NAME}'.

--send-directly-to-icl

(Optional) Directly send logs to IBM Cloud Logs instead of through the IBM Cloud Logs Routing data plane. This option is beneficial when managing multiple IBM Cloud Logs instances within the same account for advanced logging configurations.

For instance, this allows you to segregate logs from different environments (for example, development and production) into separate IBM Cloud Logs instances.

Ensure to specify the following details specific to your IBM Cloud Logs instance when using this option:
- -h <target_host>: The host for IBM Cloud Logs ingestion, found in the Endpoints section of your IBM Cloud Logs instance Overview. Use the ingress endpoint.
- -p <target_port>: Use 443.
  
  This option is particularly useful when managing multiple IBM Cloud Logs instances within the same account, since IBM Cloud Logs Routing supports only one account tenant per region.

Notice the agent can only send logs through private endpoints.

The following sample output shows a successful deployment of an agent in an Openshift cluster.

% curl -sSL https://ibm.biz/logs-router-setup | bash -s --   -v 1.0.5   -m IAMAPIKey   -k xxxxx   -t Kubernetes   -r eu-es   -p 443
Current kubectl context: mycluster-dallas/cm7g4rmd02o78j7gr670
========================================
       AGENT INSTALLATION DETAILS
========================================
Version               : 1.0.5
IAM Environment       : Production
IAM Auth Mode         : IAMAPIKey
Trusted Profile ID    :
IAM API Key           : **********
Cluster Type          : Kubernetes
Cluster Name          : mycluster-dallas
Region                : eu-es
Ingester Target Host  : ingester.private.eu-es.logs-router.cloud.ibm.com
Ingester Target Port  : 443
========================================
Downloading and applying configuration from https://logs-router-agent-config.s3.us.cloud-object-storage.appdomain.cloud/logger-agent-iks.yaml for agent version 1.0.5 in mycluster-dallas.
Warning: resource namespaces/ibm-observe is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
namespace/ibm-observe configured
serviceaccount/logger-agent-sa created
clusterrole.rbac.authorization.k8s.io/logger-agent-read-cr configured
clusterrolebinding.rbac.authorization.k8s.io/logger-agent-read-crb configured
secret/iam-api-key-secret created
configmap/logger-agent-config created
daemonset.apps/logger-agent-ds created
Configuration applied successfully.

====**** logger-agent version 1.0.5, installed successfully in mycluster-dallas ****====

Step 3. Verify the agent is successfully deployed

When the agent is deployed by using the default configuration, check the following resources are created:

A namespace ibm-observe.

Run kubectl get namespace to list the namespaces in the cluster. You can also run oc get namespace | grep ibm-observe to search for the ibm-observe namespace.
A config map logger-agent-config in the namespace ibm-observe.

Run kubectl get configmap logger-agent-config -n ibm-observe to view the agent config details. You can also use kubectl describe configmaps logger-agent-config -n ibm-observe.
A deamonset logger-agent-ds in the namespace ibm-observe.

Run kubectl get ds -n ibm-observe to view the deamonset.

Retrieve the list of agent pods by using the following command:

kubectl get pods -n ibm-observe -o wide

NAME                     READY     STATUS    RESTARTS AGE    IP              NODE           NOMINATED NODE   READINESS GATES
logger-agent-ds-4lwvt      1/1     Running   0          2d5h   172.17.61.181   192.168.16.4   <none>           <none>
logger-agent-ds-g7z87      1/1     Running   0          2d5h   172.17.0.48     192.168.32.4   <none>           <none>
logger-agent-ds-nw56s      1/1     Running   0          2d5h   172.17.32.232   192.168.0.10   <none>           <none>
logger-agent-ds-wcpbl      1/1     Running   0          2d5h   172.17.6.190    192.168.0.9    <none>           <none>

The READY column shows 1/1 for all pods, with a STATUS of Running. Verify that an agent pod is ready for each node in your cluster:

oc get nodes

NAME           STATUS   ROLES           AGE   VERSION
192.168.0.10   Ready    master,worker   8d    v1.20.0+558d959
192.168.0.9    Ready    master,worker   8d    v1.20.0+558d959
192.168.16.4   Ready    master,worker   8d    v1.20.0+558d959
192.168.32.4   Ready    master,worker   8d    v1.20.0+558d959

The number of items in each of these two lists need to be the same, and you can match the IP addresses in the node names with the values in the NODE column of the pod listing.

If your nodes are not named by their IP, you can append the -o wide option and compare the values in the INTERNAL-IP column instead.

Step 4. Verify logs are being delivered to your target destination

Complete the following steps:

Go to the web UI for your Log Analysis instance.
When your agent is correctly configured, you can see logs through the default dashboard view. The IBM Cloud Logs Routing agent tags log records with a meta object that includes the cluster name.
```
meta.cluster_name:<CLUSTER_NAME>
```
You can run the query meta.cluster_name:<YOUR_CLUSTER_NAME> in your Log Analysis instance to search for logs that are generated by your cluster.

Step 5. (Optional) Add additional metadata fields

You can add additional metadata fields to the routed logs.

Edit the logger-agent-config.yaml file in the logger-agent namespace.
In the fluent-bit.conf section add your custom metadata using this structure.
```
 [FILTER]
    Name record_modifier
    Match *
    Record <meta.key_name> <your_custom_value>
```
Where <meta.key_name> is the name of the metadata field to be added (for example, meta.env) and <your_custom_value> is the value to be assigned to the field (for example, the name of your environment).

For example, if you want to add an environment and region name as metadata, the configuration would be similar to this:
```
[FILTER]
    Name record_modifier
    Match *
    Record meta.cluster_name my-cluster
    Record meta.env production
    Record meta.region us-east
```
Save the configuration file.
Restart the daemon set to apply the changes by running the following:
```
kubectl rollout restart daemonset logger-agent-ds -n logger-agent
```

Uninstalling the agent

Complete the following steps to remove an agent:

Access your cluster.

Delete the daemonset.

kubectl delete ds logger-agent-ds -n ibm-observe

Delete the configmap.

kubectl delete cm logger-agent-config -n ibm-observe

Delete the secrets by deleting the service account.

kubectl delete sa logger-agent-sa -n ibm-observe

kubectl delete secret iam-api-key-secret -n ibm-observe