Location error messages

By default, IBM Cloud Satellite monitors the health of your locations and tries to resolve issues automatically for you. For issues that cannot be resolved automatically, you can debug the location by reviewing the provided health information.

Reviewing error messages and logs

View your locations in the console or list your locations in the CLI, and review the Status. If the status is not healthy, continue to the next step. For more information, see Viewing location health.

ibmcloud sat location ls

Example output

Name         ID                     Status            Ready   Created      Hosts (used/total)   Managed From   
Port-North   aaaaa1a11aaaaaa111aa   action required   no      6 days ago   3 / 5                Washington DC

Find the details for your location, and review the State, Ready for deployment and Message sections. From the console, you can click the location and hover over the tooltip in the title with the location name and health.

ibmcloud sat location get --location <location_name_or_ID>

Example output:

Name:                           Port-NewYork   
ID:                             aaaaa1a11aaaaaa111aa   
Created:                        2020-06-05 13:50:58 -0400 (6 days ago)   
Creator:                        name@email.com   
Managed From:                   Washington DC   
State:                          action required   
Ready for deployments:          no   
Message:                        R0015: Could not assign hosts because no hosts are available. Attach more hosts to the location and try again. For more information, see the docs: 'http://ibm.biz/sat-loc'

Find more details about the error message and affected components by setting up Log Analysis to review Satellite location logs.
Review the location state and message for steps to resolve the issue. For a description of the location state, see Cluster States in the Red Hat OpenShift on IBM Cloud documentation. For a description of the location message, review the following sections. If your location state is warning or critical and the location is ready for deployments, open a support ticket.

R0001: Ready location

Location message: The Satellite location is ready for operations.
Steps to resolve: Your Satellite location has no critical alerts, and the IBM monitoring component in the location control plane is monitoring the health of your location. You might still see some warning messages for actions that you can take to improve the state of resources in your location, such as hosts.

R0002, R0018, R0020, R0029, R0037, R0039, R0042: Wait for location to be ready

Location message: R0002: The Satellite location has issues that IBM Cloud Support is working to resolve. Check back later.; R0018: Satellite is attempting to recover.; R0020: Wait while Satellite completes a recovery action.; R0029: Successfully initiated recovery action.; R0037: The Satellite location has clusters that are in a failed state. IBM Cloud Support is working to resolve. Check back later.; R0039: The Satellite location control plane is currently unhealthy. IBM Cloud Support is working to resolve. Check back later.; R0042: IBM Cloud Support is resolving link api failures. Check back later. If this issue persists, raise a support case.
Steps to resolve: Check back later to see if the issue is resolved. If the issue persists for a while, you can open a support case.

To find more details about your issue, set up Log Analysis for Satellite location.

Set up Log Analysis for Satellite location platform logs.
Search the platform logs for the error code for more details, such as failed API method due to a permissions error.
If the details indicate a permission error:
1. As the account administrator, log in to the IBM Cloud CLI and target the resource group and region that the location is in.
```
ibmcloud login -g <resource_group> -r <region>
```
2. Reset the API key that is used for permissions.
```
ibmcloud ks api-key reset
```

R0009: Unable to recover

Location message: R0009: Satellite is unable to recover from issues.
Steps to resolve: Satellite tried unsuccessfully to automatically resolve the issue. Review any further messages to troubleshoot further, such as adding more hosts to the location. If the issue persists for a while, you can open a support case.

R0010, R0030, R0031, R0032: Control plane needs hosts

Location message: R0010: Assign more hosts to the location control plane, or replace unhealthy hosts.; R0030: A zone for the Satellite location control plane is reaching a critical capacity. If critical capacity is reached, you cannot add more clusters to location. Add more hosts to the control plane zone, or replace unhealthy hosts.; R0031: A zone for the Satellite location control plane is reaching a warning capacity. Add more hosts to the control plane zone, or replace unhealthy hosts.; R0032: Manually assign hosts to the control plane across all 3 zones.
Steps to resolve: Your location has no available hosts for Satellite to automatically assign to the location control plane, and might be reaching capacity limits. You can choose from the following options.

Attach and assign more hosts to the location control plane. Keep in mind that when you scale up the location control plane, scale evenly in multiples of 3, and assign the hosts evenly across zones.
Remove and reattach the host.

R0011, R0040, R0041: Issues with the control plane hosts

Location message

R0011: Make sure that all hosts for your Satellite location are in a normal state. If you still have issues, contact IBM Cloud Support and include your Satellite location ID.

R0040: The Satellite location data plane is currently unhealthy. To debug the host, see 'http://ibm.biz/sat-host-debug'. If you still have issues, contact IBM Cloud Support and include your Satellite location ID.

R0041: Unknown issues are detected with the Satellite location control plane hosts. Ensure that hosts meet the minimum requirements, http://ibm.biz/sat-host-reqs. If you still have issues, contact IBM Cloud Support and include your Satellite location ID.

Steps to resolve

Check the Status of your hosts. ibmcloud sat host ls --location <location_name_or_ID>
If you have no hosts, attach hosts to your location.
Make sure that you have at least 6 hosts (2 hosts per zone across 3 zones) that are assigned to the infrastructure cluster for the location, to run location control plane operations.
If your hosts have no status and are unassigned, log in to debug the host machines.
Review the host status to resolve the host issue.

R0012: Hosts are needed in all 3 zones

Location message

R0012: The location control plane does not have hosts in all 3 zones. Add available hosts to your location for the control plane.

Steps to resolve

If you just assigned hosts to the control plane, wait a while for the bootstrapping process to complete. Otherwise, assign at least one host to each of the three zones for the location itself, to run control plane operations.

If you did assign at least 2 hosts in each of the 3 zones, check the CPU and memory size of the hosts. The hosts must have at least 4 vCPU and 16 memory.
If you did assign at least 2 hosts per zone, make sure that the hosts meet the minimum requirements to use in Satellite, such as operating system and networking configuration.
If you did assign at least 2 hosts in each of the 3 zones but the bootstrapping process failed, log in to debug the host machines.

R0013: Unavailable zone

Location message

R0013: A zone in the location control plane is unavailable. Attach more hosts to the location and assign the hosts to the zone, or replace unhealthy hosts.

Steps to resolve

Assign at least 2 hosts to each of the 3 zones for the location itself, to run control plane operations. If you did assign at least 2 hosts in each of the 3 zones, follow these steps.

Check the CPU and memory size of the hosts. The hosts must have at least 4 vCPU and 16 memory.
Make sure that the hosts meet the minimum requirements to use in Satellite, such as operating system and networking configuration.
Log in to debug the host machines.
Remove and reattach the host. When you remove a host, the host is unassigned from the location control plane, and you must assign another host to the zone.

R0014: DNS record for control plane

Location message

R0014: Verify that the Satellite location has a DNS record for load balancing requests to the location control plane.

Steps to resolve

Verify that all hosts in your Satellite control plane show a State of assigned and a Status of Ready by running ibmcloud sat host ls --location <location_ID_or_name>.
If all hosts show the correct state and status, the DNS record for your location is not yet created. This process can take up to 30 minutes to complete after all hosts are successfully assigned to your location.
If one or more hosts do not show the correct state or status, see Debugging host health.

R0015, R0016: Host issues

Location message: R0015: Could not assign hosts because no hosts are available. Attach more hosts to the location and try again. For more information, see the docs: 'http://ibm.biz/sat-loc'; R0016: Unexpected error occurred after assigning host. To debug the host, see 'http://ibm.biz/sat-host-debug'. If you still have issues, contact IBM Cloud Support and include your Satellite location ID.
Steps to resolve: Attach more hosts to the location. If you attached hosts that are not showing up as available, see Debugging host health.

R0023, R0101: Wait for location to be ready

Location message: R0023: Wait while Satellite sets up the location control plane.; R0101: The Satellite location has clusters in the middle of an operation. Wait for them to finish and check back later.
Steps to resolve: Wait for the location control plane to finish setting up and check back later.

R0024, R0025: Cluster issues

Location message

R0024: The Satellite location has Red Hat OpenShift clusters in warning health.

R0025: The Satellite location has Red Hat OpenShift clusters in critical health.

Steps to resolve

Wait to see if another message is returned, such as a message about host capacity.
If a host message is returned, try Debugging hosts.
If no further message is returned, try Debugging your Red Hat OpenShift on IBM Cloud clusters.

R0026: Host disk space

Location message

R0026: Hosts in the location control plane are running out of disk space. Assign more hosts to the location control plane or reload the hosts with disk space issues.

Steps to resolve

List the hosts that are assigned to the control plane. by running ibmcloud sat host ls --location <location_name_or_ID> | grep infrastructure.
Check the details of the hosts by running ibmcloud sat host get --host <host_ID> --location <location_name_or_ID>.
In the infrastructure provider for the host, check the disk space of your host machine. Make sure that each host meets the minimum requirements. Remove and reattach the host.
If debugging and reattaching the host do not resolve the issue, the location control plane needs more compute resources to continue running. Assign more hosts to the location control plane.

R0033, R0034, R0035: Control plane capacity issues

Location message

R0033: Hosts in the location control plane have critical memory usage issues. Add more hosts to the location control plane and wait for the location to return to normal.

R0034: Hosts in the location control plane have critical CPU usage issues. Add more hosts to the location control plane and wait for the location to return to normal.

R0035: The location control plane is running at max capacity and cannot support any more workloads. Add hosts to each zone and wait for the location to return to normal.

Steps to resolve

In each zone, check the CPU and memory size of the hosts.
- Across all hosts in a zone, at least 3 CPU total must be available.
- Across all hosts in a zone, at least 4 GB memory total must be available.
Attach 3 more hosts to the location.
Assign at least one host to each of the three zones to add capacity for control plane operations. Keep in mind that when you scale up the location control plane, scale evenly in multiples of 3, and assign the hosts evenly across zones.
Refresh the cluster by running the ibmcloud ks cluster master refresh --cluster <cluster-id> command.

R0036: Location subdomain traffic routing

Location message: R0036: The location subdomains are not correctly routing traffic to your control plane hosts. Verify that the location subdomains are registered with the correct IP addresses for your control plane hosts with the ibmcloud sat location dns commands.
Steps to resolve: See Why does the location subdomain not route traffic to control plane hosts?.

R0038, R0101: Location has cluster operations in progress

Location message: R0038: The Satellite location has clusters in the middle of an operation. Wait for them to finish and check back later.; R0101: The Satellite location has clusters in the middle of an operation. Wait for them to finish and check back later.
Steps to resolve: Wait for the clusters to finish their operations and check back later.

R0043: Layer 3 connectivity

Location message: R0043: The location does not meet the following requirement: Hosts must have TCP/UDP/ICMP Layer 3 connectivity for all ports across hosts. If you still have issues, contact IBM Cloud Support and include your Satellite location ID.
Steps to resolve: Hosts must have TCP/UDP/ICMP Layer 3 connectivity for all ports across hosts. You cannot block access to certain ports that might block communication across hosts. Review Host network requirements and unblock the ports on the host in your infrastructure provider. This error can also mean that the host does not have the required RHEL packages installed, or the required CPU, memory, disk space and firewall ports opened up. After you verify that your hosts meet all the requirements, take a look at the platform logs. For more information, see Setting up Log Analysis for Satellite location platform logs.

To test TCP/UDP/ICMP Layer 3 connectivity for all ports across hosts,

SSH into a host that is attached to your location but that is not assigned to any resources.

You can only SSH into the machine if you did not assign the host to a cluster, or if the assignment failed. Otherwise, Satellite disables the ability to log in to the host via SSH for security purposes. You can remove the host and reload the operating system to restore the ability to SSH into the machine.
To check TCP connectivity, verify that netcat receives a response from all other hosts on port 10250. If the operation times out, review Host network requirements to unblock the ports on the host in your infrastructure provider.
```
nc -zv <host_IP> 10250
```
To check ICMP connectivity, verify that a ping is successful to all other hosts. Repeat this step for each IP address of the hosts that are attached to your location. If the ping times out, review Host network requirements to unblock the ports on the host in your infrastructure provider.
```
ping <host_IP>
```
If the TCP and ICMP connectivity checks do not reveal any issues, reboot all control plane hosts by rebooting one host at a time. Do not reboot control plane hosts concurrently, which can prevent etcd from running on the control plane hosts.

R0044: DNS issues

Location message: R0044: DNS issues have been detected on one or more hosts. Verify that your DNS solution is working as expected. If you still have issues, contact IBM Cloud Support and include your Satellite location ID.
Steps to resolve: One or more hosts in your locations is unable to resolve DNS queries or a search domain is causing unexpected issues. Verify that your DNS solution is working as expected and that all hosts meet the network host requirements.

To test DNS resolution,

SSH into a host that is attached to your location but that is not assigned to any resources.

You can only SSH into the machine if you did not assign the host to a cluster, or if the assignment failed. Otherwise, Satellite disables the ability to log in to the host via SSH for security purposes. You can remove the host and reload the operating system to restore the ability to SSH into the machine.

Ensure that DNS resolution is working properly.

dig +short +timeout=5 +nocookie cloud.ibm.com

Ensure that localhost with any appended search domains in your DNS configuration either do not resolve to anything, or resolve only to 127.0.0.1. In the /etc/resolv.conf file that manages DNS resolution for each host, multiple search domains, such as search ibm.com, might be listed. Calico Typha pods on each host run a health check that uses localhost resolution. However, some search domains might be appended when the health check attempts to resolve localhost, which causes the health check to fail. To ensure that the health check can run properly, make sure that none of the listed search domains resolve to anything other than the 127.0.0.1 IP address when appended to localhost.

R0045: Host read-only file system issues

Location message

R0045: A read-only file system has been detected on one or more hosts. Replace the affected host(s).

Steps to resolve

Set up Log Analysis for Satellite location platform logs for more information about which hosts are affected.
Remove the affected hosts and reattach new hosts.
If you still have issues, open a support case and include your Satellite location ID.

R0046: NTP issues

Location message: R0046: An NTP issue has been detected on one or more hosts. Verify that your NTP solution is working as expected.
Steps to resolve: One or more hosts in your locations has Network Time Protocol (NTP) issues that must be resolved.

To test NTP on your hosts,

SSH into a host that is attached to your location but that is not assigned to any resources.

You can only SSH into the machine if you did not assign the host to a cluster, or if the assignment failed. Otherwise, Satellite disables the ability to log in to the host via SSH for security purposes. You can remove the host and reload the operating system to restore the ability to SSH into the machine.
Ensure that the time that is reported by the host does not differ from the actual time by more than 3 minutes. If the time differs by more than 3 minutes, verify your NTP solution with your infrastructure provider.
```
date +%s
```
Repeat these steps to identify any hosts that have NTP issues.

R0047: Location health checking

Location message: R0047: IBM Cloud is unable to use the health check endpoint to check the location's health.
Steps to resolve: See Why is IBM Cloud unable to check my location's health?.

R0048: etcd backup failure

Location message: R0048: The etcd backup for a cluster in your location failed to complete within the past day.
Steps to resolve: The etcd data is backed up every 8 hours from your Satellite location control plane to a bucket in your IBM Cloud Object Storage instance. If this backup consecutively fails 3 times over 24 hours, problems might exist with the Object Storage bucket or service instance, or with your Satellite location's connection to the Object Storage instance.

To determine where your problem exists,

Ensure that the hosts that you assigned to the location control plane are able to access the IBM Cloud Object Storage endpoint for the IBM Cloud region that your location is managed from. For example, in your host firewall, you must allow outbound connectivity from your control plane hosts to the following endpoints.

Required outbound connectivity for hosts to Object Storage endpoints
Region	Object Storage endpoint
`wdc`	`s3.us.cloud-object-storage.appdomain.cloud`
`lon`	`s3.eu.cloud-object-storage.appdomain.cloud`

Verify that the Object Storage service instance and bucket that back up your etcd data are available and were not deleted.
1. From the Satellite console, click the name of your location.
2. In the details section of your location overview, copy the name of the Object Storage bucket.
3. In the IBM Cloud console, navigate to your IBM Cloud resource list.
4. Expand the Storage row.
5. Look for the Object Storage instance where you created the bucket. If you did not specify a bucket name during location creation, check each Object Storage instance until you find the autogenerated bucket for your location.
6. Click the instance's name. The Buckets list page opens.
7. Verify that the bucket for your control plane etcd backup exists.
8. If either the service instance or bucket were deleted, open a support case and include your Satellite location ID.
If the control plane hosts can reach the Object Storage endpoint, and the Object Storage service instance and bucket exist, open a support case to investigate backup failures and include your Satellite location ID.

R0049: Satellite Link IAM API key issue

Location message

The Link tunnel client is experiencing authentication issues. Contact IBM Cloud Support and include your Satellite location ID.

This error is reported because the IAM API key that is set for the region or resource group that the location is in does not have the required permissions in Satellite or Kubernetes Service, usually because the permissions of the API key owner changed or the API owner is no longer in the account.

Steps to resolve

If you have a Red Hat OpenShift cluster in a Satellite location and you are the account owner or a user with Administrator permission for all Satellite components, you can resolve this issue by resetting the API key. Note that when you reset the API key, the old key is deleted. Be sure to check if other services are using this API key.

Log in to IBM Cloud: ibmcloud login.
Target the region that the location is managed from: ibmcloud target -r <region>.
Target the resource group that the location is in: ibmcloud target -g <resource-group>.
Reset the IAM API key for that region or resource group: ibmcloud ks api-key reset --region <region>.
Review that the API key was set: ibmcloud ks api-key info --cluster <roks_cluster_in_location>.
Open a support case and ask that the location that contains the cluster to be refreshed. Include your location ID, which you can find by running the ibmcloud sat location ls command.

If your location does not have any clusters, then you cannot reset the API key yourself. Instead, open a support case and ask for your location to be refreshed. Include your location ID, which you can find by running the ibmcloud sat location ls command.

R0050, R0051: Satellite Link connector issues

Location message: The Link tunnel client is experiencing token authentication issues. Contact IBM Cloud Support and include your Satellite location ID.; The Link tunnel client cannot retrieve the location ID. Contact IBM Cloud Support and include your Satellite location ID.
Steps to resolve: Open a support case and include your Satellite location ID.

R0052: Ingress certificate generation issues

Location message: Ingress certificates have not been generated for the location endpoints.
Steps to resolve: IBM Cloud Support is notified and is working to resolve the issue. Try again later.

R0056: Pod status stuck in `terminating`

Location message: Pods have been stuck in a terminating state on the location control plane node for over an hour.
Steps to resolve: Pods that remain in a terminating state for an hour or more indicate that the location control plane is not healthy. Restart the location control plane hosts and see if the problem is resolved. If the problem still exists, follow the steps to replace the location control plane hosts.

When you update or replace control plane hosts, do not assign or remove multiple hosts at the same time as doing so may break the control plane. You must wait for a host assignment or removal to complete before assigning or removing another host. To avoid possible service disruptions, make sure you attach and assign additional hosts to the control plane before removing any hosts.

For each host you want to remove from the control plane, attach an additional host.
Assign the attached hosts to your location. Make sure that you only assign hosts one at a time and that each assignment completes before you assign another host.
Remove the original hosts from your Satellite location. Make sure that you only remove hosts one at a time and that each removal completes before you remove another host.

For additional information about the affected components, set up Log Analysis and review the R0056 error logs.

R0057: Outbound traffic to IAM is failing

Location message: Outbound traffic to IBM Cloud IAM is failing. To ensure that all host requirements are met, see Host system requirements. More information is available in the IBM Cloud Platform Logs. If the issue persists, contact IBM Cloud Support and include your Satellite location ID.
Steps to resolve: Check the health status and ensure that the host system requirements are met.

Run the following command to check the health status.
```
curl https://iam.cloud.ibm.com/healthz 
```
If the output from the previous step indicates a failure, check that your hosts meet all system requirements.
If you have met all the system requirements and the issue persists, open a support case and include your Satellite location ID. You can find your location ID by running the ibmcloud sat location ls command.

For additional information about the affected components, set up Log Analysis and review the R0057 error logs.

R0058: DNS registration is failing

Location message: A location with this name was recently deleted. If you want to reuse the location name of a recently deleted location, DNS registration might take a week to complete.
Steps to resolve: If you don't need to reuse this location name, delete this location and create a location with a unique name. If the issue persists, open a support case and include your Satellite location ID.

R0059: Outbound traffic to IBM Cloud Container Registry is failing.

Location message: Outbound traffic to IBM Cloud Container Registry is failing. To ensure that all host requirements are met, see the Host system requirements. More information is available in the IBM Cloud Platform Logs. If the issue persists, contact IBM Cloud Support and include your Satellite location ID.
Steps to resolve: Check the health status and ensure that the host system requirements are met.

Run the following command to check the health status.
```
curl https://iam.cloud.ibm.com/healthz 
```
If the output from the previous step indicates a failure, check that your hosts meet all system requirements.
If you have met all the system requirements and the issue persists, open a support case and include your Satellite location ID. You can find your location ID by running the ibmcloud sat location ls command.

For additional information about the affected components, set up Log Analysis and review the R0059 error logs.

R0060: Outbound traffic to LaunchDarkly is failing.

Location message: Outbound traffic to LaunchDarkly is failing. To ensure that all host requirements are met, see the Host system requirements. More information is available in the IBM Cloud Platform Logs. If the issue persists, contact IBM Cloud Support and include your Satellite location ID.
Steps to resolve: Check the health status and ensure that the host system requirements are met.

Run the following command to check the health status.
```
curl https://iam.cloud.ibm.com/healthz 
```
If the output from the previous step indicates a failure, check that your hosts meet all system requirements.
If you have met all the system requirements and the issue persists, open a support case and include your Satellite location ID. You can find your location ID by running the ibmcloud sat location ls command.

For additional information about the affected components, set up Log Analysis and review the R0060 error logs.

R0061: A Satellite cluster API server is unreachable from IBM Cloud.

Location message: A Satellite cluster API server is unreachable from IBM Cloud. For more information, see the IBM Cloud Platform Logs. If the issue persists, contact IBM Cloud Support and include your Satellite location ID.
Steps to resolve: Take the following steps to resolve this issue.

Check the IBM Cloud platform logs for more details. For more information, see set up Log Analysis and review the R0061 error logs.
Check that your hosts meet all system requirements, specifically for outbound connectivity to connect to IBM and Link tunnel clients.
If you have met all the system requirements and the issue persists, open a support case and include your Satellite location ID. You can find your location ID by running the ibmcloud sat location ls command.

Location error messages

Reviewing error messages and logs

R0001: Ready location

R0002, R0018, R0020, R0029, R0037, R0039, R0042: Wait for location to be ready

R0009: Unable to recover

R0010, R0030, R0031, R0032: Control plane needs hosts

R0011, R0040, R0041: Issues with the control plane hosts

R0012: Hosts are needed in all 3 zones

R0013: Unavailable zone

R0014: DNS record for control plane

R0015, R0016: Host issues

R0023, R0101: Wait for location to be ready

R0024, R0025: Cluster issues

R0026: Host disk space

R0033, R0034, R0035: Control plane capacity issues

R0036: Location subdomain traffic routing

R0038, R0101: Location has cluster operations in progress

R0043: Layer 3 connectivity

R0044: DNS issues

R0045: Host read-only file system issues

R0046: NTP issues

R0047: Location health checking

R0048: etcd backup failure

R0049: Satellite Link IAM API key issue

R0050, R0051: Satellite Link connector issues

R0052: Ingress certificate generation issues

R0056: Pod status stuck in terminating

R0057: Outbound traffic to IAM is failing

R0058: DNS registration is failing

R0059: Outbound traffic to IBM Cloud Container Registry is failing.

R0060: Outbound traffic to LaunchDarkly is failing.

R0061: A Satellite cluster API server is unreachable from IBM Cloud.

R0056: Pod status stuck in `terminating`