Why can't I SSH into my worker node?

Debugging by using oc debug

Use the oc debug node command to deploy a pod with a privileged securityContext to a worker node that you want to troubleshoot.

The debug pod is deployed with an interactive shell so that you can access the worker node immediately after the pod is created. For more information about how the oc debug node command works, see this Red Hat blog post.

1. Get the name of the worker node that you want to access. For CoreOS worker nodes, the name is the hostname of the worker. For all other worker nodes, the worker node name is the private IP address.

   oc get nodes -o wide
   

2. Create a debug pod that has host access. When the pod is created, the pod's interactive shell is automatically opened. If the oc debug node command fails, continue to option 2.

   oc debug node/<NODE_NAME>
   

   If the oc debug node/<NODE_NAME> command fails, there might be security group, ACL, or firewall rules that prevent the default container image from being pulled. Try the command again with the --image=us.icr.io/armada-master/network-alpine:latest option, which uses an image from IBM Cloud Container Registry that is accessible over the private network .

3. Run debug commands to help you gather information and troubleshoot issues. Commands that you might use to debug, such as tcpdump, curl, ip, ifconfig, nc, ping, and ps, are already available in the shell. You can also install other tools, such as mtr and conntrack, by running yum install <tool>.

Debugging by using kubectl exec

If you are unable to use the oc debug node command, you can create an Alpine pod with a privileged securityContext and use the kubectl exec command to run debug commands from the pod's interactive shell.

1. Get the name of the worker node that you want to access. For CoreOS worker nodes, the name is the hostname of the worker. For all other worker nodes, the worker node name is the private IP address.

   oc get nodes -o wide
   

2. Export the name in an environment variable.

   export NODE=<NODE_NAME>
   

3. Create a debug pod on the worker node. The Docker alpine image here is used as an example. If the worker node doesn't have public network access, you can maintain a copy of the image for debugging in your own ICR repository or build a customized image with other tools to fit your needs.

   kubectl apply -f - << EOF
   apiVersion: v1
   kind: Pod
   metadata:
     name: debug-${NODE}
     namespace: default
   spec:
     tolerations:
     - operator: "Exists"
     hostNetwork: true
     containers:
     - args: ["-c", "sleep 20d"]
       command: ["/bin/sh"]
       image: us.icr.io/armada-master/network-alpine:latest
       imagePullPolicy: Always
       name: debug
       securityContext:
         privileged: true
       volumeMounts:
       - mountPath: /host
         name: host-volume
     volumes:
     - name: host-volume
       hostPath:
         path: /
     nodeSelector:
       kubernetes.io/hostname: ${NODE}
     restartPolicy: Never
   EOF
   

4. Log in to the debug pod. The pod's interactive shell is automatically opened. If the kubectl exec command fails, continue to option 3.

   kubectl exec -it debug-${NODE} -- sh
   

   To get logs or other file from a worker node, use the **kubectl cp** command in the following format. The following example gets the /var/log/messages file from the host file system of the worker node.

   oc cp default/debug-${NODE}:/host/var/log/messages ./messages
   

   Get the following logs to look for issues on the worker node.

   /var/log/messages 
   /var/log/kubelet.log
   /var/log/crio.log
   /var/log/calico/cni/cni.log
   

5. Run debug commands to help you gather information and troubleshoot issues. Commands that you might use to debug, such as dig, tcpdump, mtr, curl, ip, ifconfig, nc, ping, and ps, are already available in the shell. You can also install other tools, such as conntrack, by running apk add <tool>. For example, to add conntrack, run apk add conntrack-tools.

6. Delete the host access pod that you created for debugging.

   kubectl delete pod debug-${NODE}
   

Debugging by enabling root SSH access on a worker node

If you are unable to use the oc debug node or kubectl exec commands, such as if the VPN connection between the cluster master and worker nodes is down, you can create a pod that enables root SSH access and copies a public SSH key to the worker node for SSH access.

Allowing root SSH access is a security risk. Only allow SSH access when it is required and no other option is available to troubleshoot worker node issues. When you finish troubleshooting, be sure to follow the steps in the Cleaning up after debugging section to disable SSH access.

1. Choose an existing or create a new public SSH key.

   ssh-keygen -f /tmp/id_rsa_cluster_worker -t rsa -b 4096 -C temp-worker-ssh-key -P ''
   ls /tmp
   

   id_rsa_cluster_worker id_rsa_cluster_worker.pub
   

   cat /tmp/id_rsa_cluster_worker.pub
   

2. Get the name of the worker node that you want to access. For CoreOS worker nodes, the name is the hostname of the worker. For all other worker nodes, the worker node name is the private IP address.

   oc get nodes -o wide
   

3. Create the following YAML file for a debug pod, and save the file as enable-ssh.yaml. Replace <NODE_NAME> with the worker node name and replace the example value for SSH_PUBLIC_KEY with your public SSH key. The Docker alpine image here is used as an example. If the worker node doesn't have public network access, you can keep a copy of the image for debugging in your own ICR repository or build a customized image with other tools to fit your needs.

   apiVersion: v1
   kind: Pod
   metadata:
     name: enable-ssh-<NODE_NAME>
     labels:
       name: enable-ssh
   spec:
     tolerations:
     - operator: "Exists"
     hostNetwork: true
     hostPID: true
     hostIPC: true
     containers:
     - image: us.icr.io/armada-master/network-alpine:latest
       env:
       - name: SSH_PUBLIC_KEY
         value: "<ssh-rsa AAA...ZZZ temp-worker-ssh-key>"
       args: ["-c", "echo $(SSH_PUBLIC_KEY) | tee -a /root/.ssh/authorized_keys && sed -i 's/^#*PermitRootLogin.*/PermitRootLogin yes/g' /host/etc/ssh/sshd_config && sed -i 's/^#*PermitRootLogin.*/PermitRootLogin yes/g' /host/etc/ssh/sshd_config.d/40-rhcos-defaults.conf || true && killall -1 sshd || yes n | ssh-keygen -f /host/etc/ssh/ssh_host_rsa_key -t rsa -b 4096 -C temp-server-ssh-key -P '' && while true; do sleep 86400; done"]
       command: ["/bin/sh"]
       name: enable-ssh
       securityContext:
         privileged: true
       volumeMounts:
       - mountPath: /host
         name: host-volume
       - mountPath: /root/.ssh
         name: ssh-volume
     volumes:
     - name: host-volume
       hostPath:
         path: /
     - name: ssh-volume
       hostPath:
         path: /root/.ssh
     nodeSelector:
       kubernetes.io/hostname: <NODE_NAME>
     restartPolicy: Never
   

4. Create the pod in your cluster. When this pod is created, your public key is added to the worker node and SSH is configured to allow root SSH login.

   oc apply -f enable-ssh.yaml
   

5. Use the private or public network to access the worker node by using your SSH key.