Updating classic worker nodes in the CLI with a ConfigMap

Use a ConfigMap to perform a rolling update of your classic worker nodes. The ConfigMap lets you control how many nodes can be unavailable at a time, per zone or region. If the default 20% unavailability rule is acceptable for your cluster, you can skip steps 3 through 6 (ConfigMap creation and apply) and proceed directly to step 7 to update the worker nodes by using the default behavior.

1. Complete the prerequisite steps.

2. List available worker nodes and note their private IP address.

   ibmcloud oc worker ls --cluster CLUSTER
   

3. View the labels of a worker node. You can find the worker node labels in the Labels section of your CLI output. Every label consists of a NodeSelectorKey and a NodeSelectorValue.

   oc describe node PRIVATE-WORKER-IP
   

   Example output

   NAME:               10.184.58.3
   Roles:              <none>
   Labels:             arch=amd64
                   beta.kubernetes.io/arch=amd64
                   beta.kubernetes.io/os=linux
                   failure-domain.beta.kubernetes.io/region=us-south
                   failure-domain.beta.kubernetes.io/zone=dal12
                   ibm-cloud.kubernetes.io/encrypted-docker-data=true
                   ibm-cloud.kubernetes.io/iaas-provider=softlayer
                   ibm-cloud.kubernetes.io/machine-type=u3c.2x4.encrypted
                   kubernetes.io/hostname=10.123.45.3
                   privateVLAN=2299001
                   publicVLAN=2299012
   Annotations:        node.alpha.kubernetes.io/ttl=0
                   volumes.kubernetes.io/controller-managed-attach-detach=true
   CreationTimestamp:  Tue, 03 Apr 2022 15:26:17 -0400
   Taints:             <none>
   Unschedulable:      false
   

4. Create a ConfigMap and define the unavailability rules for your worker nodes. The ConfigMap supports up to 15 named checks. Each check targets a set of worker nodes by label and sets the maximum percentage of those nodes that can be unavailable at one time. The following example shows a zone check (zonecheck.json), a region check (regioncheck.json), a default fallback check (defaultcheck.json), and a template for custom checks. For every check, choose one of the worker node labels that you retrieved in the previous step to identify the target nodes.

   For every check, you can set only one value for NodeSelectorKey and NodeSelectorValue. If you want to set rules for more than one region, zone, or other worker node labels, create a new check. Define up to 15 checks in a ConfigMap. If you add more checks, only 1 worker node is reloaded at a time until all workers requested are updated.

   Example

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: ibm-cluster-update-configuration
     namespace: kube-system
   data:
     drain_timeout_seconds: "120"
     zonecheck.json: |
       {
         "MaxUnavailablePercentage": 30,
         "NodeSelectorKey": "failure-domain.beta.kubernetes.io/zone",
         "NodeSelectorValue": "dal13"
       }
     regioncheck.json: |
       {
         "MaxUnavailablePercentage": 20,
         "NodeSelectorKey": "failure-domain.beta.kubernetes.io/region",
         "NodeSelectorValue": "us-south"
       }
     defaultcheck.json: |
       {
         "MaxUnavailablePercentage": 20
       }
     CHECK_NAME: |
       {
         "MaxUnavailablePercentage": VALUE_IN_PERCENTAGE,
         "NodeSelectorKey": "NODE_SELECTOR_KEY",
         "NodeSelectorValue": "NODE_SELECTOR_VALUE"
       }
   

   drain_timeout_seconds

   Optional: The timeout in seconds to wait for the drain to complete. Draining a worker node safely removes all existing pods from the worker node and reschedules the pods onto other worker nodes in the cluster. Accepted values are integers in the range 1 - 180. The default value is 30.

   zonecheck.json and regioncheck.json

   Two checks that define a rule for a set of worker nodes that you can identify with the specified NodeSelectorKey and NodeSelectorValue. The zonecheck.json identifies worker nodes based on their zone label, and the regioncheck.json uses the region label that is added to every worker node during provisioning. In the example, 30% of all worker nodes that have dal13 as their zone label and 20% of all the worker nodes in us-south can be unavailable during the update.

   defaultcheck.json

   If you don't create a ConfigMap or the ConfigMap is configured incorrectly, the Kubernetes default is applied. By default, only 20% of the worker nodes in the cluster can be unavailable at a time. You can override the default value by adding the default check to your ConfigMap. In the example, every worker node that is not specified in the zone and region checks (dal13 or us-south) can be unavailable during the update.

   MaxUnavailablePercentage

   The maximum number of nodes that are allowed to be unavailable for a specified label key and value, which is specified as a percentage. A worker node is unavailable during the deploying, reloading, or provisioning process. The queued worker nodes are blocked from updating if it exceeds any defined maximum unavailable percentages.

   NodeSelectorKey

   The label key of the worker node for which you want to set a rule. You can set rules for the default labels that are provided by IBM, as well as on worker node labels that you created. If you want to add a rule for worker nodes that belong to one worker pool, you can use the ibm-cloud.kubernetes.io/machine-type label.

   NodeSelectorValue

   The label value that the worker node must have to be considered for the rule that you define.

5. Create the configuration map in your cluster.

   oc apply -f CONFIGMAP_FILE_PATH
   

6. Verify that the ConfigMap is created.

   oc get configmap --namespace kube-system
   

7. Update the worker nodes.

   ibmcloud oc worker update --cluster CLUSTER --worker WORKER-NODE-1-ID --worker WORKER-NODE-2-ID
   

8. Optional: Verify the events that are triggered by the ConfigMap and any validation errors that occur. The events can be reviewed in the Events section of your CLI output.

   oc describe -n kube-system cm ibm-cluster-update-configuration
   

9. Confirm that the update is complete by reviewing the Kubernetes version of your worker nodes. For worker node states, see Worker node states.

   oc get nodes
   

10. Verify that you don't have duplicate worker nodes. Sometimes, older clusters list duplicate worker nodes with a NotReady status after an update. To remove duplicates, see troubleshooting.

Updating classic worker nodes in the console

After you set up the ConfigMap for the first time, you can update worker nodes by using the IBM Cloud console. The console respects the unavailability rules that you defined in the ConfigMap.

1. Complete the prerequisite steps and set up a ConfigMap to control how your worker nodes are updated.
2. From the IBM Cloud console menu , click Containers > Clusters.
3. From the Clusters page, click your cluster.
4. From the Worker Nodes tab, select the checkbox for each worker node that you want to update. An action bar is displayed over the table header row.
5. From the action bar, click Update.

If you have Portworx installed in your cluster, you must restart the Portworx pods on the updated worker nodes. For more information, see Portworx limitations.