IBM Cloud Docs
Logging for clusters

Logging for clusters

Set up logging in IBM Cloud® Kubernetes Service to help you troubleshoot issues and improve the health and performance of your Kubernetes clusters and apps.

Continuous monitoring and logging is the key to detecting attacks on your cluster and troubleshooting issues as they arise. By continuously monitoring your cluster, you're able to better understand your cluster capacity and the availability of resources that are available to your app. With this insight, you can prepare to protect your apps against downtime.

You can configure custom settings for the logging agents that are deployed to your cluster such custom inclusion rules, logging directories, and more. For more information, see Logging agent.

Choosing a logging solution

By default, logs are generated and written locally for all the following IBM Cloud Kubernetes Service cluster components: worker nodes, containers, applications, persistent storage, Ingress application load balancer, Kubernetes API, and the kube-system namespace. Several logging solutions are available to collect, forward, and view these logs.

You can choose your logging solution based on which cluster components you need to collect logs for. A common implementation is to choose a logging service that you prefer based on its analysis and interface capabilities, such as IBM Log Analysis or a third-party service. You can then use IBM Cloud Activity Tracker to audit user activity in the cluster.

IBM® Log Analysis
Manage pod container logs by deploying an instance of IBM Log Analysis and configuring this instance for your cluster in Kubernetes Service. A logging agent collects logs with the extension *.log and extensionless files that are stored in the /var/log directory of your pod from all namespaces, including kube-system. The agent then forwards the logs to your IBM Log Analysis service instance. For more information about the service, see the IBM Log Analysis documentation. To enable IBM Log Analysis in your cluster, see Forwarding cluster and app logs to IBM Log Analysis.
IBM Cloud® Activity Tracker
To monitor user-initiated administrative activity made in your cluster, Kubernetes Service automatically generates cluster management events and forwards these event logs to IBM Cloud Activity Tracker. To access these logs, provision an instance of IBM Cloud Activity Tracker. For more information about the types of IBM Cloud Kubernetes Service events that you can track, see Activity Tracker events.
Fluentd with an external server
To collect, forward, and view logs for a cluster component, you can create a logging configuration by using Fluentd. When you create a logging configuration, the Fluentd cluster component collects logs from the paths for a specified source. Fluentd can then forward these logs to an external server that accepts a syslog protocol. To get started, see Understanding cluster and app log forwarding to syslog.

Forwarding cluster and app logs to IBM Log Analysis

Forward logs to an IBM Log Analysis instance.

Use the IBM Cloud Kubernetes Service observability plug-in to create a logging configuration for IBM Log Analysis in your cluster, and use this logging configuration to automatically collect and forward pod logs to IBM Log Analysis.

Considerations for using the IBM Cloud Kubernetes Service observability plug-in:

  • You can have only one logging configuration for IBM Log Analysis in your cluster at a time. If you want to use a different IBM Log Analysis service instance to send logs to, use the ibmcloud ob logging config replace command.
  • If you created a Log Analysis configuration in your cluster without using the IBM Cloud Kubernetes Service observability plug-in, you can use the ibmcloud ob logging agent discover command to make the configuration visible to the plug-in. Then, you can use the observability plug-in commands and functionality in the IBM Cloud console to manage the configuration.

Before you begin

To set up a logging configuration for your cluster,

  1. Create an IBM Log Analysis service instance and note the name of the instance. The service instance must belong to the same IBM Cloud account where you created your cluster, but can be in a different resource group and IBM Cloud region than your cluster.

  2. Set up a logging configuration for your cluster. When you create the logging configuration, a Kubernetes namespace ibm-observe is created and a Log Analysis agent is deployed as a daemon set to all worker nodes in your cluster. This agent collects logs with the extension *.log and extensionless files that are stored in the /var/log directory of your pod from all namespaces, including kube-system. The agent then forwards the logs to the IBM Log Analysis service.

    • From the console

      1. From the Kubernetes clusters console, select the cluster for which you want to create a Log Analysis configuration.
      2. On the cluster Overview page, click Connect.
      3. Select the region and the IBM Log Analysis service instance that you created earlier, and click Connect.

    • From the CLI

      1. Create the Log Analysis configuration. When you create the Log Analysis configuration, the ingestion key that was last added is retrieved automatically. If you want to use a different ingestion key, add the --logdna-ingestion-key <ingestion_key> option to the command.

        To use a different ingestion key after you created your logging configuration, use the ibmcloud ob logging config replace command.

        ibmcloud ob logging config create --cluster <cluster_name_or_ID> --instance <Log_Analysis_instance_name_or_ID>

        Example output

        Creating configuration...
OK

      2. Verify that the logging configuration was added to your cluster.

        ibmcloud ob logging config list --cluster <cluster_name_or_ID>

        Example output

        Listing configurations...

OK
Instance Name                Instance ID                            CRN   
IBM Cloud Log Analysis-opm   1a111a1a-1111-11a1-a1aa-aaa11111a11a   crn:v1:prod:public:logdna:us-south:a/a11111a1aaaaa11a111aa11a1aa1111a:1a111a1a-1111-11a1-a1aa-aaa11111a11a::

  3. Optional: Verify that the Log Analysis agent was set up successfully.

    1. If you used the console to create the Log Analysis configuration, log in to your cluster. For more information, see Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster..

    2. Verify that the daemon set for the Log Analysis agent was created and all instances are listed as AVAILABLE.

      kubectl get daemonsets -n ibm-observe

      Example output

      NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
logdna-agent   9         9         9       9            9           <none>          14m

      The number of daemon set instances that are deployed equals the number of worker nodes in your cluster.

    3. Review the ConfigMap that was created for your Log Analysis agent.

      kubectl describe configmap -n ibm-observe

  4. Access the logs for your pods from the Log Analysis dashboard.

    1. From the Kubernetes clusters console, select the cluster that you configured.
    2. On the cluster Overview page, click Launch. The Log Analysis dashboard opens.
    3. Review the pod logs that the Log Analysis agent collected from your cluster. It might take a few minutes for your first logs to show.

  5. Review how you can search and filter logs in the Log Analysis dashboard.

Forwarding cluster and app logs to an external server

Configure log forwarding for IBM Cloud Kubernetes Service standard clusters to an external server.

Understanding log forwarding to an external server

When you create a logging configuration for a source in your cluster to forward to an external server, a Fluentd component is created in your cluster. Fluentd collects the logs from that source's paths and forwards the logs to an external server. The traffic from the source to the logging service on the ingestion port is encrypted.

What are the sources that I can configure log forwarding for?
In the following image, you can see the location of the sources that you can configure logging for.

Log sources in your cluster.
Figure 1. Log sources in your cluster

  1. worker: Information that is specific to the infrastructure configuration that you have for your worker node. Worker logs are captured in syslog and contain operating system events. In auth.log you can find information on the authentication requests that are made to the OS.

    Paths

    • /var/log/syslog
    • /var/log/auth.log

  2. container: Information that is logged by a running container.Paths: Anything that is written to STDOUT or STDERR.

  3. application: Information about events that occur at the application level. This could be a notification that an event took place such as a successful login, a warning about storage, or other operations that can be performed at the app level.Paths: You can set the paths that your logs are forwarded to. However, in order for logs to be sent, you must use an absolute path in your logging configuration or the logs can't be read. If your path is mounted to your worker node, it might have created a symbolic link. Example: If the specified path is /usr/local/spark/work/app-0546/0/stderr but the logs actually go to /usr/local/spark-1.0-hadoop-1.2/work/app-0546/0/stderr, then the logs can't be read.

  4. storage: Information about persistent storage that is set up in your cluster. Storage logs can help you set up problem determination dashboards and alerts as part of your DevOps pipeline and production releases. Note: The paths /var/log/kubelet.log and /var/log/syslog also contain storage logs, but logs from these paths are collected by the kubernetes and worker log sources.

    Paths
    /var/log/ibmc-s3fs.log
    /var/log/ibmc-block.log
    Pods**
    portworx-***
    ibmcloud-block-storage-attacher-***
    ibmcloud-block-storage-driver-***
    ibmcloud-block-storage-plugin-***
    ibmcloud-object-storage-plugin-***

  5. kubernetes: Information from the kubelet, the kube-proxy, and other Kubernetes events that happen in the kube-system namespace of the worker node.

    Paths
    /var/log/kubelet.log
    /var/log/kube-proxy.log
    /var/log/event-exporter/1..log

  6. ingress: Information about the network traffic that comes into a cluster through the Ingress ALB.

    Paths
    /var/log/alb/ids/*.log
    /var/log/alb/ids/*.err
    /var/log/alb/customerlogs/*.log
    /var/log/alb/customerlogs/*.err

  7. kube-audit: Information about cluster-related actions that is sent to the Kubernetes API server, including the time, the user, and the affected resource. The kube-audit source can be configured with a webhook. For more information, see Forwarding Kubernetes API audit logs to an external server.

Am I responsible for keeping Fluentd updated?
To change your logging or filter configurations, the Fluentd logging component must be at the latest version. By default, automatic updates to the add-on are enabled. To disable automatic updates, see Updating cluster components: Fluentd for logging.
Can I forward some logs, but not others, from one source in my cluster?
Yes. For example, if you have a particularly chatty pod, you might want to prevent logs from that pod from taking up log storage space, while still allowing other pods' logs to be forwarded. To prevent logs from a specific pod from being forwarded, see Filtering logs.

Forwarding cluster and app logs

Create a configuration for cluster and app logging. You can differentiate between the different logging options by using options.

The following table shows the different options that you have when you configure logging and their descriptions.

Table 1. Understanding logging configuration options
Parameter Description
<cluster_name_or_ID> The name or ID of the cluster.
--logsource The source that you want to forward logs from. Accepted values are container, application, worker, kubernetes, ingress, and storage. This option supports a comma-separated list of log sources to apply to the configuration. If you don't provide a log source, logging configurations are created for container and ingress log sources.
--type syslog The value syslog forwards your logs to an external server.
--namespace Optional: The Kubernetes namespace that you want to forward logs from. Log forwarding is not supported for the ibm-system and kube-system Kubernetes namespaces. This value is valid only for the container log source. If you don't specify a namespace, then all namespaces in the cluster use this configuration.
--hostname Specify the hostname or IP address of the log collector service.
--port The ingestion port. If you don't specify a port, then the standard port 9091 is used. For syslog, specify the port of the log collector server. If you don't specify a port, then the standard port 514 is used.
--app-containers Optional: To forward logs from apps, you can specify the name of the container that contains your app. You can specify more than one container by using a comma-separated list. If no containers are specified, logs are forwarded from all the containers that contain the paths that you provided.
--app-paths The path on a container that the apps log to. To forward logs with source type application, you must provide a path. To specify more than one path, use a comma-separated list; for example, /var/log/myApp1/*,/var/log/myApp2/*
--syslog-protocol When the logging type is syslog<, the transport layer protocol. You can use the following protocols: udp, tls, or tcp. When forwarding to a rsyslog server with the udp protocol, logs that are over 1KB are truncated.
--ca-cert Required: When the logging type is syslog and the protocol is tls, the Kubernetes secret name that contains the certificate authority certificate.
--verify-mode When the logging type is syslog and the protocol is tls, the verification mode. Supported values are verify-peer and the default verify-none.
--skip-validation Optional: Skip the validation of the org and space names when they are specified. Skipping validation decreases processing time, but an invalid logging configuration does not correctly forward logs.

Forwarding logs to your own server over the udp or tcp protocols

  1. Ensure that you have the Editor or Administrator IBM Cloud IAM platform access role.

  2. For the cluster where the log source is located: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

  3. Set up a server that accepts a syslog protocol in 1 of 2 ways:

    Set up and manage your own server or have a provider manage it for you. If a provider manages the server for you, get the logging endpoint from the logging provider.

    Run syslog from a container. For example, you can use this deployment .yaml file to fetch a Docker public image that runs a container in your cluster. The image publishes the port 514 on the public cluster IP address, and uses this public cluster IP address to configure the syslog host.

    You can see your logs as valid JSON by removing syslog prefixes. To do so, add the following code to the beginning your etc/rsyslog.conf file where your rsyslog server runs: $template customFormat,"%msg%\n"$ActionFileDefaultTemplate customFormat

  4. Create a log forwarding configuration. For more information about the parameters, see the Understanding logging configuration options table.

    ibmcloud ks logging config create --cluster <cluster_name_or_ID> --logsource <log_source> --namespace <kubernetes_namespace> --hostname <log_server_hostname_or_IP> --port <log_server_port> --type syslog --app-containers <container1,2> --app-paths <paths_to_logs> --syslog-protocol <protocol> --skip-validation

Forwarding logs to your own server over the tls protocol

The following steps are general instructions. Before you use the container in a production environment, be sure that any security requirements are met.

  1. Ensure that you have the following IBM Cloud IAM roles:

    • Editor or Administrator platform access role for the cluster
    • Writer or Manager service access role for the kube-system namespace

  2. For the cluster where the log source is located: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

  3. Set up a server that accepts a syslog protocol in 1 of 2 ways:

    • Set up and manage your own server or have a provider manage it for you. If a provider manages the server for you, get the logging endpoint from the logging provider.

    • Run syslog from a container. For example, you can use this deployment .yaml file to fetch a Docker public image that runs a container in your cluster. The image publishes the port 514 on the public cluster IP address, and uses this public cluster IP address to configure the syslog host. You need to inject the relevant certificate authority and server-side certificates and update the syslog.conf to enable tls on your server.

  4. Save your certificate authority certificate to a file named ca-cert. It must be that exact name.

  5. Create a secret in the kube-system namespace for the ca-cert file. When you create your logging configuration, use the secret name for the --ca-cert option.

    kubectl -n kube-system create secret generic --from-file=ca-cert

  6. Create a log forwarding configuration. For more information about the parameters, see the Understanding logging configuration options table.

    ibmcloud ks logging config create --cluster <cluster name or id> --logsource <log source> --type syslog --syslog-protocol tls --hostname <ip address of syslog server> --port <port for syslog server, 514 is default> --ca-cert <secret name> --verify-mode <defaults to verify-none>

Filtering logs that are forwarded

You can choose which logs to forward to your external server by filtering out specific logs for a period of time. You can differentiate between the different filtering options by using options.

Table 2. Understanding the options for log filtering
Parameter Description
<cluster_name_or_ID> Required: The name or ID of the cluster that you want to filter logs for.
<log_type> The type of logs that you want to apply the filter to. Currently all, container, and host are supported.
<configs> Optional: A comma-separated list of your logging configuration IDs. If not provided, the filter is applied to all the cluster logging configurations that are passed to the filter. You can view log configurations that match the filter by using the --show-matching-configs option.
<kubernetes_namespace> Optional: The Kubernetes namespace that you want to forward logs from. This option applies only when you are using log type container.
<container_name> Optional: The name of the container from which you want to filter logs.
<logging_level> Optional: Filters out logs that are at the specified level and less. Acceptable values in their canonical order are fatal, error, warn/warning, info, debug, and trace. As an example, if you filtered logs at the info level, debug, and trace are also filtered. Note: You can use this option only when log messages are in JSON format and contain a level field. To display your messages in JSON, append the --output json option to the command.
<message> Optional: Filters out logs that contain a specified message that is written as a regular expression.
<filter_ID> Optional: The ID of the log filter.
--show-matching-configs Optional: Show the logging configurations that each filter applies to.
--all Optional: Delete all your log forwarding filters.

  1. Create a logging filter.

    ibmcloud ks logging filter create --cluster <cluster_name_or_ID> --type <log_type> --logging-configs <configs> --namespace <kubernetes_namespace> --container <container_name> --level <logging_level> --regex-message <message>

  2. View the log filter that you created.

    ibmcloud ks logging filter get --cluster <cluster_name_or_ID> --id <filter_ID> --show-matching-configs

  3. Update the log filter that you created.

    ibmcloud ks logging filter update --cluster <cluster_name_or_ID> --id <filter_ID> --type <server_type> --logging-configs <configs> --namespace <kubernetes_namespace --container <container_name> --level <logging_level> --regex-message <message>

  4. Delete a log filter that you created.

    ibmcloud ks logging filter rm --cluster <cluster_name_or_ID> --id <filter_ID> [--all]

Verifying, updating, and deleting log forwarding

Verifying log forwarding

You can verify that your configuration is set up correctly in 1 of 2 ways:

  • To list all the logging configurations in a cluster:
    ibmcloud ks logging config get --cluster <cluster_name_or_ID>
  • To list the logging configurations for one type of log source:
    ibmcloud ks logging config get --cluster <cluster_name_or_ID> --logsource <source>

Updating log forwarding

You can update a logging configuration that you already created:

ibmcloud ks logging config update --cluster <cluster_name_or_ID> --id <log_config_id> --namespace <namespace> --type <server_type> --syslog-protocol <protocol> --logsource <source> --hostname <hostname_or_ingestion_URL> --port <port> --app-containers <container1,2> --app-paths <paths_to_logs>

Deleting log forwarding

You can stop forwarding logs by deleting one or all the logging configurations for a cluster:

  • To delete one logging configuration:
    ibmcloud ks logging config rm --cluster <cluster_name_or_ID> --id <log_config_ID>
  • To delete all the logging configurations:
    ibmcloud ks logging config rm --cluster <my_cluster> --all