--- name: containers-cs_health_log title: Logging for clusters description: Set up logging in IBM Cloud® Kubernetes Service to help you troubleshoot issues and improve the health and performance of your Kubernetes clusters and apps. last-updated: 2026-07-27 --- > ## Documentation Index > The table of contents for this documentation set is at https://cloud.ibm.com/docs/containers?format=markdown > The index for all IBM Cloud docs is at: https://cloud.ibm.com/docs/llms.txt > Use these files to discover more information as needed. # Logging for clusters {: #health} Set up logging in IBM Cloud® Kubernetes Service to help you troubleshoot issues and improve the health and performance of your Kubernetes clusters and apps. {: shortdesc} Continuous monitoring and logging is the key to detecting attacks on your cluster and troubleshooting issues as they arise. By continuously monitoring your cluster, you're able to better understand your cluster capacity and the availability of resources that are available to your app. With this insight, you can prepare to protect your apps against downtime. ## Choosing a logging solution {: #logging_overview} By default, logs are generated and written locally for all the following IBM Cloud Kubernetes Service cluster components: worker nodes, containers, applications, persistent storage, Ingress application load balancer, Kubernetes API, and the `kube-system` namespace. Several logging solutions are available to collect, forward, and view these logs. {: shortdesc} IBM Cloud Logs : Manage pod container logs by deploying an instance of IBM Cloud Logs and configuring this instance for your cluster in Kubernetes Service. A logging agent collects logs with the extension `*.log` and extensionless files that are stored in the `/var/log` directory of your pod from all namespaces, including `kube-system`. The agent then forwards the logs to your service instance. You can also track user-initiated administrative activity made in your cluster.Kubernetes Service automatically generates cluster management events and forwards these event logs to IBM Cloud Logs. For more information, see [Getting started with IBM Cloud Logs](https://cloud.ibm.com/docs/cloud-logs?topic=cloud-logs-getting-started&format=markdown). To deploy a logging agent to your cluster, see [Managing the Logging agent for Red Hat OpenShift on IBM Cloud clusters](https://cloud.ibm.com/docs/cloud-logs?topic=cloud-logs-agent-helm-os-deploy&format=markdown) or [Managing the Logging agent for IBM Cloud Kubernetes Service clusters](https://cloud.ibm.com/docs/cloud-logs?topic=cloud-logs-agent-helm-kube-deploy&format=markdown). Fluentd with an external server : To collect, forward, and view logs for a cluster component, you can create a logging configuration by using Fluentd. When you create a logging configuration, the [Fluentd](https://www.fluentd.org/){: external} cluster component collects logs from the paths for a specified source. Fluentd can then forward these logs to an external server that accepts a `syslog` protocol. To get started, see [Understanding log forwarding to an external server](#logging-external). ## Migrating logging and monitoring agents to Cloud Logs {: #logging_forwarding} The observability CLI plug-in `ibmcloud ob` and the `v2/observe` endpoints are no longer supported. There is no direct replacement, but you can now manage your logging and monitoring integrations through the [IBM Cloud Kubernetes Service extension](https://cloud.ibm.com/docs/cloud-logs?topic=cloud-logs-extensions-kubernetes&format=markdown) or by [Sending IBM Cloud Kubernetes Service log data to IBM Cloud Logs](https://cloud.ibm.com/docs/cloud-logs?topic=cloud-logs-kube2logs&format=markdown). {: deprecated} You can no longer use the `ob` plug-in, Terraform, or API to install observability agents on a cluster or to modify your existing configuration. Sysdig agents continue to send metrics to the specified IBM Cloud Monitoring instance. LogDNA agents can no longer send logs since IBM Cloud Log Analysis is replaced by IBM Cloud Logs. ### Removing the observability plug-in agents {: #ob-remove} * After support for the `ob plugin` ends, you must delete each component individually. 1. Clean up the daemonsets and configmaps. ```sh kubectl delete daemonset logdna-agent -n ibm-observe kubectl delete daemonset sysdig-agent -n ibm-observe kubectl delete configmap -n ibm-observe kubectl delete configmap -n ibm-observe ``` {: pre} 1. Optional: Delete the namespace. After no other resources are running in the namespace. ```sh kubectl delete namespace ibm-observe ``` {: pre} After the plug-in has been removed, reinstall Logging and Monitoring agents in your cluster using the Cluster dashboard, Terraform, or manually. For more information, see the following links: - [Sending IBM Cloud Kubernetes Service log data to IBM Cloud Logs](https://cloud.ibm.com/docs/cloud-logs?topic=cloud-logs-kube2logs&format=markdown) - [Monitoring a Kubernetes cluster](https://cloud.ibm.com/docs/monitoring?topic=monitoring-kubernetes_cluster&format=markdown) ## Forwarding cluster and app logs to an external server {: #configuring} Configure log forwarding for IBM Cloud Kubernetes Service standard clusters to an external server. {: shortdesc} ### Understanding log forwarding to an external server {: #logging-external} When you create a logging configuration for a source in your cluster to forward to an external server, a [Fluentd](https://www.fluentd.org/){: external} component is created in your cluster. Fluentd collects the logs from that source's paths and forwards the logs to an external server. The traffic from the source to the logging service on the ingestion port is encrypted. {: shortdesc} What are the sources that I can configure log forwarding for? : In the following image, you can see the region of the sources that you can configure logging for. ![Log sources in your cluster.](images/log_sources.svg "Log sources in your cluster"){: caption="Log sources in your cluster" caption-side="bottom"} 1. `worker`: Information that is specific to the infrastructure configuration that you have for your worker node. Worker logs are captured in `syslog` and contain operating system events. In `auth.log` you can find information on the authentication requests that are made to the OS. Paths - `/var/log/syslog` - `/var/log/auth.log` 2. `container`: Information that is logged by a running container.**Paths**: Anything that is written to `STDOUT` or `STDERR`. 3. `application`: Information about events that occur at the application level. This could be a notification that an event took place such as a successful login, a warning about storage, or other operations that can be performed at the app level.**Paths**: You can set the paths that your logs are forwarded to. However, in order for logs to be sent, you must use an absolute path in your logging configuration or the logs can't be read. If your path is mounted to your worker node, it might have created a symbolic link. Example: If the specified path is `/usr/local/spark/work/app-0546/0/stderr` but the logs actually go to `/usr/local/spark-1.0-hadoop-1.2/work/app-0546/0/stderr`, then the logs can't be read. 4. `storage`: Information about persistent storage that is set up in your cluster. Storage logs can help you set up problem determination dashboards and alerts as part of your DevOps pipeline and production releases. **Note**: The paths `/var/log/kubelet.log` and `/var/log/syslog` also contain storage logs, but logs from these paths are collected by the `kubernetes` and `worker` log sources. Paths : `/var/log/ibmc-s3fs.log` : `/var/log/ibmc-block.log` Pods : `portworx-***` : `ibmcloud-block-storage-attacher-***` : `ibmcloud-block-storage-driver-***` : `ibmcloud-block-storage-plugin-***` : `ibmcloud-object-storage-plugin-***` 5. `kubernetes`: Information from the kubelet, the kube-proxy, and other Kubernetes events that happen in the kube-system namespace of the worker node. Paths : `/var/log/kubelet.log` : `/var/log/kube-proxy.log` : `/var/log/event-exporter/1..log` 6. `ingress`: Information about the network traffic that comes into a cluster through the Ingress ALB. Paths : `/var/log/alb/ids/*.log` : `/var/log/alb/ids/*.err` : `/var/log/alb/customerlogs/*.log` : `/var/log/alb/customerlogs/*.err` 7. `kube-audit`: Information about cluster-related actions that is sent to the Kubernetes API server, including the time, the user, and the affected resource. The `kube-audit` source can be configured with a webhook. For more information, see [Forwarding Kubernetes API audit logs to an external server](https://cloud.ibm.com/docs/containers?topic=containers-health-audit&format=markdown#audit-api-server-external). Am I responsible for keeping Fluentd updated? : To change your logging or filter configurations, the Fluentd logging component must be at the latest version. By default, automatic updates to the add-on are enabled. To disable automatic updates, see [Updating cluster components: Fluentd for logging](https://cloud.ibm.com/docs/containers?topic=containers-update&format=markdown#logging-up). Can I forward some logs, but not others, from one source in my cluster? : Yes. For example, if you have a particularly chatty pod, you might want to prevent logs from that pod from taking up log storage space, while still allowing other pods' logs to be forwarded. To prevent logs from a specific pod from being forwarded, see [Filtering logs](#filter-logs). ### Forwarding cluster and app logs {: #enable-forwarding} Create a configuration for cluster and app logging. You can differentiate between the different logging options by using options. {: shortdesc} The following table shows the different options that you have when you configure logging and their descriptions. | Parameter | Description | | --- | ---------- | | `` | The name or ID of the cluster. | | `--logsource` | The source that you want to forward logs from. Accepted values are `container`, `application`, `worker`, `kubernetes`, `ingress`, and `storage`. This option supports a comma-separated list of log sources to apply to the configuration. If you don't provide a log source, logging configurations are created for `container` and `ingress` log sources. | | `--type syslog` | The value `syslog` forwards your logs to an external server. | | `--namespace` | Optional: The Kubernetes namespace that you want to forward logs from. Log forwarding is not supported for the `ibm-system` and `kube-system` Kubernetes namespaces. This value is valid only for the `container` log source. If you don't specify a namespace, then all namespaces in the cluster use this configuration. | | `--hostname` | Specify the hostname or IP address of the log collector service. | | `--port` | The ingestion port. If you don't specify a port, then the standard port `9091` is used. For syslog, specify the port of the log collector server. If you don't specify a port, then the standard port `514` is used. | | `--app-containers` | Optional: To forward logs from apps, you can specify the name of the container that contains your app. You can specify more than one container by using a comma-separated list. If no containers are specified, logs are forwarded from all the containers that contain the paths that you provided. | | `--app-paths` | The path on a container that the apps log to. To forward logs with source type `application`, you must provide a path. To specify more than one path, use a comma-separated list; for example, `/var/log/myApp1/*,/var/log/myApp2/*` | | `--syslog-protocol` | When the logging type is `syslog<`, the transport layer protocol. You can use the following protocols: `udp`, `tls`, or `tcp`. When forwarding to a `rsyslog` server with the `udp` protocol, logs that are over 1KB are truncated. | | `--ca-cert` | Required: When the logging type is `syslog` and the protocol is `tls`, the Kubernetes secret name that contains the certificate authority certificate. | | `--verify-mode` | When the logging type is `syslog` and the protocol is `tls`, the verification mode. Supported values are `verify-peer` and the default `verify-none`. | | `--skip-validation` | Optional: Skip the validation of the org and space names when they are specified. Skipping validation decreases processing time, but an invalid logging configuration does not correctly forward logs. | {: caption="Understanding logging configuration options" caption-side="bottom"} ### Forwarding logs to your own server over the `udp` or `tcp` protocols {: #enable-forwarding-udp-tcp} 1. Ensure that you have the [**Editor** or **Administrator** IBM Cloud IAM platform access role](https://cloud.ibm.com/docs/containers?topic=containers-iam-platform-access-roles&format=markdown). 2. For the cluster where the log source is located: [Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster&format=markdown) 3. Set up a server that accepts a `syslog` protocol in 1 of 2 ways: : Set up and manage your own server or have a provider manage it for you. If a provider manages the server for you, get the logging endpoint from the logging provider. : Run `syslog` from a container. For example, you can use this [deployment .yaml file](https://github.com/IBM-Cloud/kube-samples/blob/master/deploy-apps-clusters/deploy-syslog-from-kube.yaml){: external} to fetch a Docker public image that runs a container in your cluster. The image publishes the port `514` on the public cluster IP address, and uses this public cluster IP address to configure the syslog host. You can see your logs as valid JSON by removing `syslog` prefixes. To do so, add the following code to the beginning your `etc/rsyslog.conf` file where your `rsyslog` server runs: `$template customFormat,"%msg%\n"$ActionFileDefaultTemplate customFormat` {: tip} 4. Create a log forwarding configuration. For more information about the parameters, see the [Understanding logging configuration options table](#enable-forwarding). ```sh ibmcloud ks logging config create --cluster CLUSTER_NAME_OR_ID --logsource LOG_SOURCE --namespace KUBERNETES_NAMESPACE --hostname LOG_SERVER_HOSTNAME_OR_IP --port LOG_SERVER_PORT --type syslog --app-containers CONTAINER1,2 --app-paths PATHS_TO_LOGS --syslog-protocol PROTOCOL ``` {: pre} ### Forwarding logs to your own server over the `tls` protocol {: #enable-forwarding-tls} 1. Ensure that you have the following [IBM Cloud IAM roles](https://cloud.ibm.com/docs/containers?topic=containers-iam-platform-access-roles&format=markdown): - **Editor** or **Administrator** platform access role for the cluster - **Writer** or **Manager** service access role for the `kube-system` namespace 2. For the cluster where the log source is located: [Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster&format=markdown) 3. Set up a server that accepts a `syslog` protocol in 1 of 2 ways: - Set up and manage your own server or have a provider manage it for you. If a provider manages the server for you, get the logging endpoint from the logging provider. - Run `syslog` from a container. For example, you can use this [deployment .yaml file](https://github.com/IBM-Cloud/kube-samples/blob/master/deploy-apps-clusters/deploy-syslog-from-kube.yaml){: external} to fetch a Docker public image that runs a container in your cluster. The image publishes the port `514` on the public cluster IP address, and uses this public cluster IP address to configure the `syslog` host. You need to inject the relevant certificate authority and server-side certificates and update the `syslog.conf` to enable `tls` on your server. 4. Save your certificate authority certificate to a file named `ca-cert`. It must be that exact name. 5. Create a secret in the `kube-system` namespace for the `ca-cert` file. When you create your logging configuration, use the secret name for the `--ca-cert` option. ```sh kubectl -n kube-system create secret generic --from-file=ca-cert ``` {: pre} 6. Create a log forwarding configuration. For more information about the parameters, see the [Understanding logging configuration options table](#enable-forwarding). ```sh ibmcloud ks logging config create --cluster --logsource --type syslog --syslog-protocol tls --hostname --port --ca-cert --verify-mode ``` {: pre} ### Filtering logs that are forwarded {: #filter-logs} You can choose which logs to forward to your external server by filtering out specific logs for a period of time. You can differentiate between the different filtering options by using options. {: shortdesc} | Parameter | Description | | --- | ------- | | `` | Required: The name or ID of the cluster that you want to filter logs for. | | `` | The type of logs that you want to apply the filter to. Currently `all`, `container`, and `host` are supported. | | `` | Optional: A comma-separated list of your logging configuration IDs. If not provided, the filter is applied to all the cluster logging configurations that are passed to the filter. You can view log configurations that match the filter by using the `--show-matching-configs` option. | | `` | Optional: The Kubernetes namespace that you want to forward logs from. This option applies only when you are using log type `container`. | | `` | Optional: The name of the container from which you want to filter logs. | | `` | Optional: Filters out logs that are at the specified level and less. Acceptable values in their canonical order are `fatal`, `error`, `warn/warning`, `info`, `debug`, and `trace`. As an example, if you filtered logs at the `info` level, `debug`, and `trace` are also filtered. **Note**: You can use this option only when log messages are in JSON format and contain a level field. To display your messages in JSON, append the `--output json` option to the command. | | `` | Optional: Filters out logs that contain a specified message that is written as a regular expression. | | `` | Optional: The ID of the log filter. | | `--show-matching-configs` | Optional: Show the logging configurations that each filter applies to. | | `--all` | Optional: Delete all your log forwarding filters. | {: caption="Understanding the options for log filtering" caption-side="bottom"} 1. Create a logging filter. ```sh ibmcloud ks logging filter create --cluster CLUSTER_NAME_OR_ID --type LOG_TYPE --logging-configs CONFIGS --namespace KUBERNETES_NAMESPACE --container CONTAINER_NAME --level LOGGING_LEVEL --regex-message MESSAGE ``` {: pre} 2. View the log filter that you created. ```sh ibmcloud ks logging filter get --cluster CLUSTER_NAME_OR_ID --id FILTER_ID --show-matching-configs ``` {: pre} 3. Update the log filter that you created. ```sh ibmcloud ks logging filter update --cluster CLUSTER_NAME_OR_ID --id FILTER_ID --type SERVER_TYPE --logging-configs CONFIGS --namespace KUBERNETES_NAMESPACE --container CONTAINER_NAME --level LOGGING_LEVEL --regex-message MESSAGE ``` {: pre} 4. Delete a log filter that you created. ```sh ibmcloud ks logging filter rm --cluster CLUSTER_NAME_OR_ID --id FILTER_ID [--all] ``` {: pre} ### Verifying, updating, and deleting log forwarding {: #verifying-log-forwarding} #### Verifying log forwarding {: #verify-log-forwarding} You can verify that your configuration is set up correctly in 1 of 2 ways: - To list all the logging configurations in a cluster: ```sh ibmcloud ks logging config get --cluster CLUSTER_NAME_OR_ID ``` {: pre} - To list the logging configurations for one type of log source: ```sh ibmcloud ks logging config get --cluster CLUSTER_NAME_OR_ID --logsource SOURCE ``` {: pre} #### Updating log forwarding {: #updating-log-forwarding} You can update a logging configuration that you already created: ```sh ibmcloud ks logging config update --cluster CLUSTER_NAME_OR_ID --id LOG_CONFIG_ID --namespace NAMESPACE --type SERVER_TYPE --syslog-protocol PROTOCOL --logsource SOURCE --hostname HOSTNAME_OR_INGESTION_URL --port PORT --app-containers CONTAINER1,2 --app-paths PATHS_TO_LOGS ``` {: pre} #### Deleting log forwarding {: #deleting-log-forwarding} You can stop forwarding logs by deleting one or all the logging configurations for a cluster: - To delete one logging configuration: ```sh ibmcloud ks logging config rm --cluster CLUSTER_NAME_OR_ID --id LOG_CONFIG_ID ``` {: pre} - To delete all the logging configurations for a namespace: ```sh ibmcloud ks logging config rm --cluster MY_CLUSTER --namespace KUBERNETES_NAMESPACE ``` {: pre}