Updating hosts that are assigned as worker nodes

Checking if a version update is available with the IBM Cloud CLI

1. Log in to IBM Cloud. Include the --sso option if you have a federated account.

   ibmcloud login [--sso]
   

2. List the Satellite clusters in your account.

   ibmcloud ks cluster ls --provider satellite
   

3. List the worker nodes in the cluster that you want to update the version for. In the output, check for an asterisk * with a message that indicates a version update is available.

   ibmcloud ks worker ls -c <cluster_name_or_ID>
   

   Example output

   ID                Primary IP       Flavor   State    Status   Zone     Version   
   sat-worker-<ID>   <IP_address>     upi      normal   Ready    zone-1   4.5.35_1534_openshift*   
   
   * To update to 4.5.37_1537_openshift version, run 'ibmcloud ks worker replace'. Review and make any required version changes before you update: 'https://ibm.biz/upworker'
   

Checking if a version update is available from the IBM Cloud console

1. Log in to the Satellite console.
2. Click the location with the hosts that you want to update.
3. Click the Hosts tab.
4. From the host list, click the link to the Cluster for the host that you want to update. A new tab opens for the Red Hat OpenShift on IBM Cloud cluster details.
5. Click the Worker nodes tab.
6. In the Version column, check for an information icon that says Update available when you click on the icon. If no update is available, no icon is present.
7. Determine if the version update is a major, minor, or patch update.

Apply version updates to worker node hosts one at a time

1. Optional: Attach and assign extra hosts to the service cluster to handle the compute capacity while your existing hosts are updating.

2. Identify your worker node hosts. The worker node hosts do not have infrastructure listed in the Cluster column of the output, but instead have the name of the cluster.

3. Update your worker nodes individually by running the ibmcloud ks worker update command.

   ibmcloud ks worker update -c CLUSTER_NAME_OR_ID --worker WORKER_ID
   

4. Confirm that the update is complete by reviewing the Kubernetes version of your worker nodes.

   kubectl get nodes
   

   If the update failed, you must apply version updates by replacing hosts.

Apply version updates to your worker node hosts with a ConfigMap

You can roll out updates to all your worker node hosts with a ConfigMap. Specify which nodes to update by using labels. You can also specify

1. Optional: Attach and assign extra hosts to the service cluster to handle the compute capacity while your existing hosts are updating.

2. Identify your worker node hosts. Your worker node hosts are not listed as Infrastructure.

3. View the labels of a worker node. You can find the worker node labels in the Labels section of your CLI output. Every label consists of a NodeSelectorKey and a NodeSelectorValue. You can use the labels to specify which worker nodes to update.

   kubectl get nodes -o yaml
   

   Example output

   labels:
     arch: amd64
     beta.kubernetes.io/arch: amd64
     beta.kubernetes.io/instance-type: upi
     beta.kubernetes.io/os: linux
     failure-domain.beta.kubernetes.io/region: us-east
     failure-domain.beta.kubernetes.io/zone: us-east-2
     ibm-cloud.kubernetes.io/iaas-provider: upi
     ibm-cloud.kubernetes.io/internal-ip: 10.241.0.4
     ibm-cloud.kubernetes.io/machine-type: upi
     ibm-cloud.kubernetes.io/os: REDHAT_7_64
     ibm-cloud.kubernetes.io/region: us-east
     ibm-cloud.kubernetes.io/worker-id: sat-satellitei-5bd83963cdfedc7668c9040879c88113a134c4e7
     ibm-cloud.kubernetes.io/worker-pool-id: cbtljodw089nltg8k210-9a3f763
     ibm-cloud.kubernetes.io/worker-pool-name: default
     ibm-cloud.kubernetes.io/worker-version: 4.7.59_1583_openshift
     ibm-cloud.kubernetes.io/zone: us-east-2
     kubernetes.io/arch: amd64
     kubernetes.io/hostname: satellite-ibm-host-3
     kubernetes.io/os: linux
     node-role.kubernetes.io/master: ""
     node-role.kubernetes.io/worker: ""
     node.kubernetes.io/instance-type: upi
     node.openshift.io/os_id: rhel
     privateVLAN: "1"
     topology.kubernetes.io/region: us-east
     topology.kubernetes.io/zone: us-east-2
   

4. Create a ConfigMap and define the unavailability rules for your worker nodes. The following example shows two checks, the defaultcheck.json and a check template. You can use this example check to define rules for all worker nodes that don't match any of the checks that you defined in the ConfigMap (defaultcheck.json). Use the check template to create your own check. For every check, to identify a worker node, you must choose one of the worker node labels that you retrieved in the previous step.

   For every check, you can set only one value for NodeSelectorKey and NodeSelectorValue. Define up to 10 checks in a ConfigMap. If you add more checks, they are ignored.

   Example

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: ibm-cluster-update-configuration
     namespace: kube-system
   data:
     drain_timeout_seconds: "120"
     defaultcheck.json: |
       {
         "MaxUnavailablePercentage": 20
       }
     <check_name>: |
       {
         "MaxUnavailablePercentage": <value_in_percentage>,
         "NodeSelectorKey": "<node_selector_key>",
         "NodeSelectorValue": "<node_selector_value>"
       }
   

   drain_timeout_seconds

   Optional: The timeout in seconds to wait for the drain to complete. Draining a worker node safely removes all existing pods from the worker node and reschedules the pods onto other worker nodes in the cluster. Accepted values are integers in the range 1 - 180. The default value is 30.

   defaultcheck.json

   When you update worker nodes in Satellite hosts, only 20% of the worker nodes in the cluster can be unavailable at a time.

   MaxUnavailablePercentage

   The maximum number of nodes that are allowed to be unavailable for a specified label key and value, which is specified as a percentage. A worker node is unavailable during the deploying, reloading, or provisioning process. The queued worker nodes are blocked from updating if it exceeds any defined maximum unavailable percentages.

   NodeSelectorKey

   The label key of the worker node for which you want to set a rule. You can set rules for the default labels that are provided by IBM, as well as on worker node labels that you created.

   NodeSelectorValue

   The label value that the worker node must have to be considered for the rule that you define.

5. Create the ConfigMap in your cluster.

   kubectl apply -f <filepath/configmap.yaml>
   

6. Verify that the ConfigMap is created.

   kubectl get configmap --namespace kube-system
   

7. Update the worker nodes by listing them by ID.

   ibmcloud ks worker update --cluster <cluster_name_or_ID> --worker <worker_node1_ID> --worker <worker_node2_ID>
   

8. Optional: Verify the events that are triggered by the ConfigMap and any validation errors that occur. The events can be reviewed in the Events section of your CLI output.

   kubectl describe -n kube-system cm ibm-cluster-update-configuration
   

9. Confirm that the update is complete by reviewing the Kubernetes version of your worker nodes.

   kubectl get nodes
   

   If the update failed, you must apply version updates by replacing hosts.

10. Verify that you don't have duplicate worker nodes. Sometimes, older clusters might list duplicate worker nodes with a NotReady status after an update. To remove duplicates, see troubleshooting.