Steps to update the cluster master

Before you begin, make sure that you have the Operator or Administrator IAM platform access role. If you're unsure of your access role, go to Manage → Access (IAM) → Users in the IBM Cloud console, or ask your account administrator.

If a certificate authority (CA) certificate rotation is in progress, the master update is blocked until the rotation completes. Check the status of any in-progress rotation before you begin.

To update the Red Hat OpenShift master major or minor version:

1. Review the Red Hat OpenShift on IBM Cloud version information and make any updates marked Update before master.

2. Review any Kubernetes helpful warnings, such as deprecation notices.

3. Check the add-ons and plug-ins that are installed in your cluster for any impact that might be caused by updating the cluster version.

   • Checking add-ons

     1. List the add-ons in the cluster.

        ibmcloud oc cluster addon ls --cluster CLUSTER
        

     2. Check the supported Red Hat OpenShift version for each add-on that is installed.

        ibmcloud oc addon-versions
        

     3. If the add-on must be updated to run in the Red Hat OpenShift version that you want to update your cluster to, update the add-on.

   • Checking plug-ins

     1. In the Helm catalog, find the plug-ins that you installed in your cluster.
     2. From the side menu, expand the SOURCES & TAR FILE section.
     3. Download and open the source code.
     4. Check the README.md or RELEASENOTES.md files for supported versions.
     5. If the plug-in must be updated to run in the Red Hat OpenShift version that you want to update your cluster to, update the plug-in by following the plug-in instructions.

4. Update your API server and associated master components by using the IBM Cloud console or running the CLI ibmcloud oc cluster master update command.

5. Wait a few minutes, then confirm that the update is complete. Review the API server version on the IBM Cloud clusters dashboard or run ibmcloud oc cluster ls.

6. Install the version of the oc cli that matches the API server version that runs in the master. Kubernetes does not support oc client versions that are two or more versions apart from the server version (n +/- 2). To refresh your local configuration, run ibmcloud oc cluster config -c CLUSTER_NAME_OR_ID, then verify with oc version --client.

When the master update is complete, update your worker nodes. The method depends on your infrastructure type:

1. Updating classic worker nodes — uses a rolling update controlled by a ConfigMap and the worker update command.
2. Updating VPC worker nodes — VPC bare metal workers are updated in place using worker reload; VPC VSI workers must use worker replace --update. The ConfigMap rolling update procedure is not yet supported for VPC worker nodes.

Prerequisites

Before you update your classic infrastructure worker nodes, complete the following prerequisite steps.

During a worker node update, the worker node machine is reimaged and all data that is not stored on persistent storage is permanently deleted. Verify that any data you need to retain is stored outside the worker node before you begin.

If you have Portworx installed in your cluster, you must update your Portworx configuration before you update worker nodes.

Updating classic worker nodes in the CLI with a configmap

Use a ConfigMap to perform a rolling update of your classic worker nodes. The ConfigMap lets you control how many nodes can be unavailable at a time, per zone or region. If the default 20% unavailability rule is acceptable for your cluster, you can skip steps 3 and 4 (ConfigMap creation) and proceed directly to step 5 to apply the update using the default behavior.

1. Complete the prerequisite steps.

2. List available worker nodes and note their private IP address.

   ibmcloud oc worker ls --cluster CLUSTER
   

3. View the labels of a worker node. You can find the worker node labels in the Labels section of your CLI output. Every label consists of a NodeSelectorKey and a NodeSelectorValue.

   oc describe node PRIVATE-WORKER-IP
   

   Example output

   NAME:               10.184.58.3
   Roles:              <none>
   Labels:             arch=amd64
                   beta.kubernetes.io/arch=amd64
                   beta.kubernetes.io/os=linux
                   failure-domain.beta.kubernetes.io/region=us-south
                   failure-domain.beta.kubernetes.io/zone=dal12
                   ibm-cloud.kubernetes.io/encrypted-docker-data=true
                   ibm-cloud.kubernetes.io/iaas-provider=softlayer
                   ibm-cloud.kubernetes.io/machine-type=u3c.2x4.encrypted
                   kubernetes.io/hostname=10.123.45.3
                   privateVLAN=2299001
                   publicVLAN=2299012
   Annotations:        node.alpha.kubernetes.io/ttl=0
                   volumes.kubernetes.io/controller-managed-attach-detach=true
   CreationTimestamp:  Tue, 03 Apr 2022 15:26:17 -0400
   Taints:             <none>
   Unschedulable:      false
   

4. Create a config map and define the unavailability rules for your worker nodes. The ConfigMap supports up to 15 named checks. Each check targets a set of worker nodes by label and sets the maximum percentage of those nodes that can be unavailable at one time. The following example shows a zone check (zonecheck.json), a region check (regioncheck.json), a default fallback check (defaultcheck.json), and a template for custom checks. For every check, choose one of the worker node labels that you retrieved in the previous step to identify the target nodes.

   For every check, you can set only one value for NodeSelectorKey and NodeSelectorValue. If you want to set rules for more than one region, zone, or other worker node labels, create a new check. Define up to 15 checks in a config map. If you add more checks, only 1 worker node is reloaded at a time until all workers requested are updated.

   Example

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: ibm-cluster-update-configuration
     namespace: kube-system
   data:
     drain_timeout_seconds: "120"
     zonecheck.json: |
       {
         "MaxUnavailablePercentage": 30,
         "NodeSelectorKey": "failure-domain.beta.kubernetes.io/zone",
         "NodeSelectorValue": "dal13"
       }
     regioncheck.json: |
       {
         "MaxUnavailablePercentage": 20,
         "NodeSelectorKey": "failure-domain.beta.kubernetes.io/region",
         "NodeSelectorValue": "us-south"
       }
     defaultcheck.json: |
       {
         "MaxUnavailablePercentage": 20
       }
     <check_name>: |
       {
         "MaxUnavailablePercentage": <value_in_percentage>,
         "NodeSelectorKey": "<node_selector_key>",
         "NodeSelectorValue": "<node_selector_value>"
       }
   

   drain_timeout_seconds

   Optional: The timeout in seconds to wait for the drain to complete. Draining a worker node safely removes all existing pods from the worker node and reschedules the pods onto other worker nodes in the cluster. Accepted values are integers in the range 1 - 180. The default value is 30.

   zonecheck.json and regioncheck.json

   Two checks that define a rule for a set of worker nodes that you can identify with the specified NodeSelectorKey and NodeSelectorValue. The zonecheck.json identifies worker nodes based on their zone label, and the regioncheck.json uses the region label that is added to every worker node during provisioning. In the example, 30% of all worker nodes that have dal13 as their zone label and 20% of all the worker nodes in us-south can be unavailable during the update.

   defaultcheck.json

   If you don't create a config map or the map is configured incorrectly, the Kubernetes default is applied. By default, only 20% of the worker nodes in the cluster can be unavailable at a time. You can override the default value by adding the default check to your config map. In the example, every worker node that is not specified in the zone and region checks (dal13 or us-south) can be unavailable during the update.

   MaxUnavailablePercentage

   The maximum number of nodes that are allowed to be unavailable for a specified label key and value, which is specified as a percentage. A worker node is unavailable during the deploying, reloading, or provisioning process. The queued worker nodes are blocked from updating if it exceeds any defined maximum unavailable percentages.

   NodeSelectorKey

   The label key of the worker node for which you want to set a rule. You can set rules for the default labels that are provided by IBM, as well as on worker node labels that you created. If you want to add a rule for worker nodes that belong to one worker pool, you can use the ibm-cloud.kubernetes.io/machine-type label.

   NodeSelectorValue

   The label value that the worker node must have to be considered for the rule that you define.

5. Create the configuration map in your cluster.

   oc apply -f <filepath/configmap.yaml>
   

6. Verify that the config map is created.

   oc get configmap --namespace kube-system
   

7. Update the worker nodes.

   ibmcloud oc worker update --cluster CLUSTER --worker WORKER-NODE-1-ID --worker WORKER-NODE-2-ID
   

8. Optional: Verify the events that are triggered by the config map and any validation errors that occur. The events can be reviewed in the Events section of your CLI output.

   oc describe -n kube-system cm ibm-cluster-update-configuration
   

9. Confirm that the update is complete by reviewing the Kubernetes version of your worker nodes.

   oc get nodes
   

10. Verify that you don't have duplicate worker nodes. Sometimes, older clusters list duplicate worker nodes with a NotReady status after an update. To remove duplicates, see troubleshooting.

Updating classic worker nodes in the console

After you set up the ConfigMap for the first time, you can update worker nodes by using the IBM Cloud console. The console respects the unavailability rules that you defined in the ConfigMap.

1. Complete the prerequisite steps and set up a ConfigMap to control how your worker nodes are updated.
2. From the IBM Cloud console menu , click Containers > Clusters.
3. From the Clusters page, click your cluster.
4. From the Worker Nodes tab, select the checkbox for each worker node that you want to update. An action bar is displayed over the table header row.
5. From the action bar, click Update.

If you have Portworx installed in your cluster, you must restart the Portworx pods on the updated worker nodes. For more information, see Portworx limitations.

Prerequisites

Before you update your VPC infrastructure worker nodes, complete the following prerequisite steps.

For VPC VSI workers, the worker node is deleted and replaced with a new node. For VPC bare metal workers, the node is reloaded in place. In both cases, data that is not stored on persistent storage is permanently deleted. Verify that any data you need to retain is stored outside the worker node before you begin.

If you have Portworx deployed in your cluster, follow the steps to update VPC worker nodes with Portworx volumes instead of the steps on this page.

Firmware updates during VPC bare metal worker reload

When you reload a VPC bare metal worker node, IBM Cloud infrastructure automatically checks whether a firmware update is pending for that server and applies it as part of the reload process. No additional action is required to trigger the firmware update.

Be aware of the following considerations when you reload a VPC bare metal worker node:

Extended reload time

If a firmware update is applied during the reload, the total reload time can increase significantly — by 30 minutes or more — beyond the typical reload duration. Plan your maintenance windows accordingly.

No advance visibility into pending updates

There is no visibility into whether a firmware update is pending for a worker node before you issue the reload command.

Data loss risk

As with all VPC bare metal worker reloads, data on local disks is deleted during the reload regardless of whether a firmware update is applied. Back up any data that is not stored on persistent storage before you reload.

Reload failure due to firmware update

In some cases, a firmware update can fail, which causes the worker node to enter a reload_failed state (Failed to reload worker) with status detail The infrastructure firmware update has failed. (P4056). If this occurs:

1. Wait a few minutes, then retry the reload by running ibmcloud oc worker reload --cluster CLUSTER --worker WORKER-NODE-ID again.
2. If the error persists after 2–3 attempts, open an IBM Cloud support case.

Before you begin

• Access your Red Hat OpenShift cluster.
• Verify that any data you need to retain is stored on persistent storage outside the worker node. Data stored only on the worker node is lost and cannot be recovered.
• Make sure that you have the Operator or Administrator IAM platform access role. If you're unsure of your access role, go to Manage → Access (IAM) → Users in the IBM Cloud console, or ask your account administrator.

To update flavors

1. List available worker nodes and note their private IP address.

   1. List available worker pools in your cluster.

      ibmcloud oc worker-pool ls --cluster CLUSTER
      

   2. List the worker nodes in the worker pool. Note the ID and Private IP.

      ibmcloud oc worker ls --cluster CLUSTER --worker-pool WORKER-POOL
      

   3. Get the details for a worker node. In the output, note the zone and either the private and public VLAN ID for classic clusters or the subnet ID for VPC clusters.

      ibmcloud oc worker get --cluster CLUSTER --worker WORKER-ID
      

2. List available flavors in the zone.

   ibmcloud oc flavors --zone <zone>
   

3. Create a worker node with the new machine type.

   1. Create a worker pool with the number of worker nodes that you want to replace.
      • Classic clusters:

        ibmcloud oc worker-pool create classic --name WORKER-POOL --cluster CLUSTER --flavor FLAVOR --size-per-zone NUMBER-OF-WORKERS-PER-ZONE
        

      • VPC Generation 2 clusters:

        ibmcloud oc worker-pool create vpc-gen2 --name NAME --cluster CLUSTER --flavor FLAVOR --size-per-zone NUMBER-OF-WORKERS-PER-ZONE --label LABEL
        

   2. Verify that the worker pool is created.

      ibmcloud oc worker-pool ls --cluster CLUSTER
      

   3. Add the zone to your worker pool that you retrieved earlier. When you add a zone, the worker nodes that are defined in your worker pool are provisioned in the zone and considered for future workload scheduling. If you want to spread your worker nodes across multiple zones, choose a classic or VPC multizone location.
      • Classic clusters:

        ibmcloud oc zone add classic --zone ZONE --cluster CLUSTER --worker-pool WORKER-POOL --private-vlan PRIVATE-VLAN-ID --public-vlan PUBLIC-VLAN-ID
        

      • VPC clusters:

        ibmcloud oc zone add vpc-gen2 --zone ZONE --cluster CLUSTER --worker-pool WORKER-POOL --subnet-id VPC-SUBNET-ID
        

4. Wait for the worker nodes to be deployed. When the worker node state changes to Normal, the deployment is finished.

   ibmcloud oc worker ls --cluster CLUSTER
   

5. Remove the old worker pool. If you are removing a Classic bare metal flavor (which is billed monthly), you are charged for the entire month even if you remove it mid-month. VPC workers, including bare metal, are billed hourly.

   1. Remove the worker pool with the old machine type. Removing a worker pool removes all worker nodes in the pool in all zones. This process might take a few minutes to complete.

      ibmcloud oc worker-pool rm --worker-pool WORKER-POOL --cluster CLUSTER
      

   2. Verify that the worker pool is removed.

      ibmcloud oc worker-pool ls --cluster CLUSTER
      

6. Verify that the worker nodes are removed from your cluster.

   ibmcloud oc worker ls --cluster CLUSTER
   

7. Repeat these steps to update other worker pools or stand-alone worker nodes to different flavors.

Managing automatic updates for Fluentd

When you create a logging configuration for a source in your cluster to forward to an external server, a Fluentd component is created in your cluster. To change your logging or filter configurations, the Fluentd component must be at the latest version. By default, automatic updates to the component are enabled.

To run the following commands, you must have the Administrator IBM Cloud IAM platform access role for the cluster.

You can manage automatic updates of the Fluentd component in the following ways.

• Check whether automatic updates are enabled by running the ibmcloud oc logging autoupdate get --cluster CLUSTER command.
• Disable automatic updates by running the ibmcloud oc logging autoupdate disable command.
• If automatic updates are disabled, but you need to change your configuration, you have two options:

  • Turn on automatic updates for your Fluentd pods.

    ibmcloud oc logging autoupdate enable --cluster CLUSTER
    

  • Force a one-time update when you use a logging command that includes the --force-update option. Your pods update to the latest version of the Fluentd component, but Fluentd does not update automatically going forward. Example command

    ibmcloud oc logging config update --cluster CLUSTER --id LOG-CONFIG-ID --type LOG-TYPE --force-update
    

Managing automatic updates for Ingress ALBs

Control when the Ingress application load balancer (ALB) component is updated. For information about keeping ALBs up-to-date, see Managing the Ingress ALB lifecycle.