IBM Cloud Docs
Managing the Logging agent for IBM Cloud Kubernetes Service clusters

Managing the Logging agent for IBM Cloud Kubernetes Service clusters

You can deploy the Logging agent to collect and route infrastructure and application logs from an IBM Cloud Kubernetes Service cluster to an IBM Cloud Logs instance.

This topic is superseded by Deploying the Logging agent on Kubernetes using a Helm chart. While these instructions can still be used, deploying using the Helm chart is the preferred method.

Deploying the Logging agent

Complete the following steps to deploy the Logging agent on an IBM Cloud Kubernetes Service cluster:

Before you begin

  • Make sure you have access to Kubernetes cluster with permissions to create namespaces and deploy the agent.

  • Install the following CLIs:

    • The IBM Cloud CLI to log in to the IBM Cloud and manage IBM Cloud services such as creating an API key.

    • The Kubernetes CLI to manage the cluster by using kubectl commands. Learn more.

  • Read about the Logging agent.

  • Check the agent versions that are available. For more information, see Checking the available agent versions.

  • Download and install jq to process output and query desired results.

  • Download and install yq to read, write, and manipulate YAML files in a similar way to using jq for JSON files.

Step 1. Define the authentication method for the agent

Choose the type of identity and the authentication method for the agent. Then, create an API key if needed.

Complete the following steps:

  1. Choose the type of identity: user, service ID, or trusted profile.

    You can use a user, a service ID, or a trusted profile as the identity that is used by the agent to authenticate with the IBM® Cloud Logs service.

  2. Grant permissions for ingestion to the identity that you have chosen.

    The role that is required for sending logs to IBM Cloud Logs is Sender.

    For more information, see Setting up IAM permissions for ingestion.

  3. Generate an API Key for user authentication or for service ID authentication.

    For authentication with trusted profiles, this step is not required.

    For more information, see Generating an API Key for ingestion.

Step 2. Setting up and deploying the Logging agent configuration

Complete the following steps:

  1. Access your cluster.

  2. Configure the log collection behavior of the Logging agent, see Configuring whether logs are included or excluded.

  3. Install the Logging agent. Use the following curl command in your terminal:

    curl -sSL https://ibm.biz/logs-router-setup | bash -s -- \
  -v <agent_version> \
  -m <iam_auth_mode> \
  -i <trusted_profile_id> \
  -k <iam_api_key> \
  -t <cluster_type> \
  -d <directory> \
  -e PrivateProduction \
  -a <application_name> \
  -s <subsystem_name> \
  --send-directly-to-icl -h <target_host> -p <target_port>

    Where:

    -v <agent_version>

    Agent version. Specify the version of the Logging agent. To find the current supported versions, refer to Checking available agent versions.

    -m <iam_auth_mode>

    IAM authentication mode (TrustedProfile | IAMAPIKey).

    -i <trusted_profile_id>

    Trusted profile ID (required for TrustedProfile mode). Provide the Trusted Profile ID. When using trusted profiles, set to the ID configured in Setting up Permissions for Ingestion.

    For more information on Trusted Profiles, see Creating a Trusted Profile.

    -k <iam_api_key>

    IAM API key (required for IAMAPIKey mode). Make sure you follow the instructions in Generating an API Key.

    For more information about IAM API Keys, see Managing API Keys.

    -t <cluster_type>

    Cluster type (OpenShift or Kubernetes). Specify if you are deploying the agent to an Red Hat OpenShift on IBM Cloud (OpenShift) or IBM Cloud Kubernetes Service (Kubernetes) cluster.

    -d <directory>

    Specify the directory containing the logger-agent-iks.yaml file configured in the previous step. For example, if your logger-agent-iks.yaml file is located in the /path/to/directory directory, you would simply specify the directory like this: -d /path/to/directory.

    -a <application_name>

    Specify the application name that you want to see in your IBM Cloud Logs instance. If in the metadata, the application name defaults to namespace name. You can also use variables from the environment, for example '${NODE_NAME}'.

    -s <subsystem_name>

    Specify the subsystem name that you want to see in your IBM Cloud Logs instance. If in the metadata, the subsystem name defaults to container name. You can also use variables from the environment, for example '${NODE_NAME}'.

    -e PrivateProduction

    The agent needs access to IBM Cloud® Identity and Access Management (IAM). By detault the agent uses the public endpoint. To use the private endpoint, set this option. For more information, see IAM_Environment plug-in parameter value. All valid IAM_Environment plug-in parameter values can be specified for -e.

    --send-directly-to-icl

    Set this parameter to send logs directly to IBM Cloud Logs.

    -h <target_host>

    The host for IBM Cloud Logs ingestion, found in the Endpoints section of your IBM Cloud Logs instance Overview. Use the ingress endpoint.

    -p <target_port>

    Use 443.

Notice the agent can only send logs through private endpoints.

The following sample output shows a successful deployment of an agent in an Openshift cluster.

% curl -sSL https://ibm.biz/logs-router-setup | bash -s --   -v 1.0.5   -m IAMAPIKey   -k xxxxx   -t Kubernetes   -r eu-es   -p 443
Current kubectl context: mycluster-dallas/cm7g4rmd02o78j7gr670
========================================
       AGENT INSTALLATION DETAILS
========================================
Version               : 1.0.5
IAM Environment       : Production
IAM Auth Mode         : IAMAPIKey
Trusted Profile ID    :
IAM API Key           : **********
Cluster Type          : Kubernetes
Cluster Name          : mycluster-dallas
Region                : eu-es
Ingester Target Host  : ingester.private.eu-es.logs-router.cloud.ibm.com
Ingester Target Port  : 443
========================================
Downloading and applying configuration from https://logs-router-agent-config.s3.us.cloud-object-storage.appdomain.cloud/logger-agent-iks.yaml for agent version 1.0.5 in mycluster-dallas.
Warning: resource namespaces/ibm-observe is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
namespace/ibm-observe configured
serviceaccount/logger-agent-sa created
clusterrole.rbac.authorization.k8s.io/logger-agent-read-cr configured
clusterrolebinding.rbac.authorization.k8s.io/logger-agent-read-crb configured
secret/iam-api-key-secret created
configmap/logger-agent-config created
daemonset.apps/logger-agent-ds created
Configuration applied successfully.

====**** logger-agent version 1.0.5, installed successfully in mycluster-dallas ****====

Step 3. Verify the agent is successfully deployed

When the agent is deployed by using the default configuration, check the following resources are created:

  • A namespace ibm-observe.

    Run kubectl get namespace to list the namespaces in the cluster. You can also run oc get namespace | grep ibm-observe to search for the ibm-observe namespace.

  • A config map logger-agent-config in the namespace ibm-observe.

    Run kubectl get configmap logger-agent-config -n ibm-observe to view the agent config details. You can also use kubectl describe configmaps logger-agent-config -n ibm-observe.

  • A deamonset logger-agent-ds in the namespace ibm-observe.

    Run kubectl get ds -n ibm-observe to view the deamonset.

  • Retrieve the list of agent pods by using the following command:

    kubectl get pods -n ibm-observe -o wide

    NAME                     READY     STATUS    RESTARTS AGE    IP              NODE           NOMINATED NODE   READINESS GATES
logger-agent-ds-4lwvt      1/1     Running   0          2d5h   172.17.61.181   192.168.16.4   <none>           <none>
logger-agent-ds-g7z87      1/1     Running   0          2d5h   172.17.0.48     192.168.32.4   <none>           <none>
logger-agent-ds-nw56s      1/1     Running   0          2d5h   172.17.32.232   192.168.0.10   <none>           <none>
logger-agent-ds-wcpbl      1/1     Running   0          2d5h   172.17.6.190    192.168.0.9    <none>           <none>

    The READY column shows 1/1 for all pods, with a STATUS of Running. Verify that an agent pod is ready for each node in your cluster:

    oc get nodes

    NAME           STATUS   ROLES           AGE   VERSION
192.168.0.10   Ready    master,worker   8d    v1.20.0+558d959
192.168.0.9    Ready    master,worker   8d    v1.20.0+558d959
192.168.16.4   Ready    master,worker   8d    v1.20.0+558d959
192.168.32.4   Ready    master,worker   8d    v1.20.0+558d959

    The number of items in each of these two lists need to be the same, and you can match the IP addresses in the node names with the values in the NODE column of the pod listing.

    If your nodes are not named by their IP, you can append the -o wide option and compare the values in the INTERNAL-IP column instead.

Step 4. Verify logs are being delivered to your target destination

Complete the following steps:

  1. Go to the web UI for your IBM Cloud Logs instance..

  2. When your agent is correctly configured, you can see logs through the default dashboard view. The Logging agent tags log records with a meta object that includes the cluster name.

    meta.cluster_name:<CLUSTER_NAME>

    You can run the query meta.cluster_name:<YOUR_CLUSTER_NAME> in your IBM Cloud Logs instance to search for logs that are generated by your cluster.

Step 5. (Optional) Add additional metadata fields

You can add additional metadata fields to the routed logs.

  1. Edit the logger-agent-config.yaml file in the logger-agent namespace.

  2. In the fluent-bit.conf section add your custom metadata using this structure.

     [FILTER]
    Name record_modifier
    Match *
    Record <meta.key_name> <your_custom_value>

    Where <meta.key_name> is the name of the metadata field to be added (for example, meta.env) and <your_custom_value> is the value to be assigned to the field (for example, the name of your environment).

    For example, if you want to add an environment and region name as metadata, the configuration would be similar to this:

    [FILTER]
    Name record_modifier
    Match *
    Record meta.cluster_name my-cluster
    Record meta.env production
    Record meta.region us-east

  3. Save the configuration file.

  4. Restart the daemon set to apply the changes by running the following:

    kubectl rollout restart daemonset logger-agent-ds -n logger-agent

Uninstalling the agent

Complete the following steps to remove an agent:

  1. Access your cluster.

  2. Delete the daemonset.

    kubectl delete ds logger-agent-ds -n ibm-observe

  3. Delete the configmap.

    kubectl delete cm logger-agent-config -n ibm-observe

  4. Delete the secrets by deleting the service account.

    kubectl delete sa logger-agent-sa -n ibm-observe

    kubectl delete secret iam-api-key-secret -n ibm-observe