IBM Cloud Docs
Debugging Calico components

Debugging Calico components

Virtual Private Cloud Classic infrastructure

In version 1.29 and later, the Calico Operator determines the number of calico-typha pods based on the number of cluster workers and does not consider tainted nodes. If you have fewer than 3 untainted nodes in your cluster, or have a very large cluster with a small number of untainted nodes, you might have one or more calico-typha pods are stuck in Pending state because the pod can't find an untainted node to run on. Usually, this doesn't cause an issue as long as there is at least one calico-typha pod in Running state. However, for high availability it is recommended to have at least two calico-typha pods always running. As a best practice, make sure that there are enough untainted nodes to run all the calico-typha pods that are created by the Calico operator.

You experience issues with Calico components such as pods that don't deploy or intermittent networking issues.

Increase the logging level of Calico components to gather more information about the issue.

Increasing the log level for the calico-typha components

Complete the following steps to increase the log level for the calico-typha component.

  1. Run the following command to edit the calico-typha deployment.

    For 1.29 and later:

    kubectl edit deploy calico-typha -n calico-system

    For 1.28 and earlier:

    kubectl edit deploy calico-typha -n kube-system

  2. Change the TYPHA_LOGSEVERITYSCREEN environment variable from info to debug.

          containers:
    - env:
      - name: TYPHA_LOGSEVERITYSCREEN
        value: debug

  3. Save and close the file to apply the changes and restart the calico-typha deployment.

Increasing the log level for the calico-cni components

Complete the following steps to increase the log level for the calico-cni component.

  1. Run the following command to edit the calico-config ConfigMap.

    kubectl edit cm -n kube-system calico-config

  2. Change the cni_network_config > plugins > log_level environment variable to debug.

      cni_network_config: |-
  {
    "name": "k8s-pod-network",
    "cniVersion": "0.3.1",
    "plugins": [
      {
        "type": "calico",
        "log_level": "debug",

  3. Save and close the file. The change won't take effect until the calico-node pods are restarted.

  4. Restart the calico-node pods to apply the changes.

    kubectl rollout restart daemonset/calico-node -n kube-system

    Example output

    daemonset.apps/calico-node restarted

Increasing the log level for the calico-node components

Complete the following steps to increase the log level for the calico-node component.

  1. Run the following command:

    For 1.29 and later:

    kubectl edit ds calico-node -n calico-system

    For 1.28 and earlier:

    kubectl edit ds calico-node -n kube-system

  2. Under the FELIX_USAGEREPORTINGENABLED name and value pair (or after any of the FELIX_* environment variable name value pairs), add the following entry.

    - name: FELIX_LOGSEVERITYSCREEN
  value: Debug

  3. Save the change. After saving your changes, all the pods in the calico-node daemonset complete a rolling update that applies the changes. The calico-cni also applies any changes to logging levels in the kube-system/calico-config ConfigMap.

Increasing the log level for the calico-kube-controllers components

Complete the following steps to increase the log level for the calico-kube-controllers component.

  1. Edit the daemonset by running the following command.

    For 1.29 and later:

    kubectl edit ds calico-node -n calico-system

    For 1.28 and earlier:

    kubectl edit ds calico-node -n kube-system

  2. Under the DATASTORE_TYPE name and value pair, add the following entry.

    - name: LOG_LEVEL
  value: debug

  3. Save the change. The calico-kube-controllers pod restarts and applies the changes.

Gathering Calico logs

  1. List the pods and nodes in your cluster and make a node of the pod name, pod IP address, and worker node that has the issue.

  2. Get the logs for the calico-node pod on the worker node where the problem occurred.

    For 1.29 and later:

    kubectl logs calico-typha-aaa11111a-aaaaa -n calico-system

    For 1.28 and earlier:

    kubectl logs calico-typha-aaa11111a-aaaaa -n kube-system

  3. Get logs for the calico-kube-controllers pod.

    For 1.29 and later:

    kubectl logs calico-kube-controllers-11aaa11aa1-a1a1a -n calico-system

    For 1.28 and earlier:

    kubectl logs calico-kube-controllers-11aaa11aa1-a1a1a -n kube-system

  4. Follow the instructions for Debugging by using kubectl exec to get /var/log/syslog, containerd.log, kubelet.log, and kern.log from the worker node.