Rotating CA certificates in your cluster

Revoke existing certificate authority (CA) certificates in your cluster and issue new CA certificates.

Regularly rotating your CA certificates is recommended to keep your setup secure. It is good practice to rotate your CA certificates whenever you update your worker nodes, as the longest step of certificate rotation includes reloading or replacing your worker nodes.

By default, certificate authority (CA) certificates are administered to secure access to various components of your cluster, such as the master API server. As you use your cluster, you might want to revoke the certificates issued by the existing CA. For example, the administrators of your team might use a certificate signing request (CSR) to manually generate certificates that are signed by the cluster's CA for worker nodes in the cluster. If an administrator leaves your organization, you can ensure that they no longer have admin access to your cluster by creating a new CA and certificates for your cluster, and removing the old CA and certificates.

1. Create a CA for your cluster. Certificates that are signed by this new CA are issued for the cluster master components, and the API server is refreshed.

   ibmcloud ks cluster ca create -c CLUSTER
   

2. Ensure that your cluster's master health is normal, the API server refresh is complete, and any master updates are complete. It might take several minutes for the master API server to refresh.

   ibmcloud ks cluster get --cluster CLUSTER
   

3. Check the status of the CA creation. In the output, note the timestamp in the Action Completed field. For a description of each status, see Certificate states and timestamps.

   ibmcloud ks cluster ca status -c CLUSTER
   

   Example output

   Status:             CA certificate created. Reload your worker nodes then begin the CA certificate rotation.
   Action Started:     2024-08-30T16:17:56+0000
   Action Completed:   2024-08-30T16:21:13+0000
   

4. Download the updated Kubernetes configuration data and certificates in your cluster's kubeconfig file.

   ibmcloud ks cluster config -c CLUSTER --admin --network
   

5. Verify that the timestamps in your new certificates are later than the timestamp that you found in step 3. To check the date on your certificates, you can use a tool such as KeyCDN.

6. Update any tooling or webhooks that rely on the previous certificates. For example, you might need to update one or more of the following.

   • If you use the certificate from your cluster's kubeconfig file in your own service, such as Jenkins.
   • If you use calicoctl to manage Calico network policies, update your services and automation to use the new certificates.
   • If you forward audit logs to IBM Cloud Logs, update the certificates for your master audit webhook.
   • If you forward audit logs over the private network, update the certificates for your master audit webhook.

7. Reload your classic worker nodes or replace your VPC worker nodes to pick up the certificates that are signed by the new CA.

8. Rotate the old certificates with the new certificates. The old CA certificates in your cluster are removed.

   ibmcloud ks cluster ca rotate -c CLUSTER
   

9. Check the status of the CA certificate rotation.

   ibmcloud ks cluster ca status -c CLUSTER
   

   Example output

   Status:             CA certificate rotation complete.
   Action Started:     2024-08-30T16:37:56+0000
   Action Completed:   2024-08-30T16:41:13+0000
   

10. Because some clients might not allow expired CA certificates in the certificate bundle, download the updated Kubernetes configuration and certificates in your cluster's kubeconfig file.

    ibmcloud ks cluster config -c CLUSTER --admin --network
    

11. Update any tooling that relies on the previous certificates. If you use the certificate from your cluster's kubeconfig file in your own service such as Travis or Jenkins, or if you use calicoctl to manage Calico network policies, update your services and automation to use the new certificates.