Troubleshooting error message reference
This reference lists all documented error messages and error codes across troubleshooting topics for Red Hat OpenShift on IBM Cloud. Entries are grouped by component and link to the full troubleshooting topic.
Clusters and masters
| Error message | Troubleshooting topic |
|---|---|
Cannot complete cluster master operations because the cluster has a broken webhook application. |
Why do cluster master operations fail due to a broken webhook? |
Cannot complete cluster master upgrade because the Upgradeable status condition is set to False. |
Why do I see a Cannot complete cluster master upgrade message? |
The master is approaching its allotted memory resource limit (93%). |
Why does my cluster master status say it is approaching its resource limit? |
etcd database size is approaching the maximum |
Why do I see an etcd database size is approaching the maximum error? |
The 'configuration' field is not a valid Kubernetes PodSecurityConfiguration setting. |
Why do I get an error that my PodSecurityConfiguration is not valid? |
No VPC is available. Create a VPC. |
VPC: Why is no VPC available when I create a cluster in the console? |
Your cluster can't pull images from the 'icr.io' domains because an IAM access policy could not be created. |
Why can't the cluster pull images from IBM Cloud Container Registry during creation? |
Image security enforcement update canceled. CAE008: can't enable Portieris image security enforcement because the cluster already has a conflicting image admission controller installed. |
Why is my Portieris cluster image security enforcement installation canceled? |
incorrect account for worker, Worker deploy failed due to network communications failing, Unable to connect to the IBM Cloud account. |
Why can't I create or delete clusters or worker nodes? |
Unable to create cluster. The 'vpc-gen2' infrastructure operation failed with the message: the provided token is not authorized to view the specified subnet |
Why do I get an infrastructure operation failed error when creating a VPC cluster? |
No resources found., connection timed out, dial tcp: connect: connection timed out |
Debugging common CLI issues with clusters |
Encrypted storage cannot be configured. Review the customer root key configuration for the worker pool. |
Why can't I create a VPC cluster with encrypted worker nodes? |
Pending security group creation |
When I create a VPC cluster, my worker nodes are stuck in Pending security group creation |
Infrastructure instance status is 'failed': Can't start instance because provisioning failed. |
Why do I see DNS failures after adding a custom DNS resolver? |
Could not store the cloud object storage bucket and IAM service key. |
Why do I get an error about a cloud object storage bucket when I create a cluster? |
Could not find user. |
Why do I see a Could not find user error when I try to access the web console? |
No resources found. |
After logging in to my cluster, why do I see a no resources found message? |
VPN server configuration update failed. |
Why does the cluster master return a VPN server error? |
Version shows as not up to date in oc get clusterversion output |
Why does OpenShift show the cluster version is not up to date? |
| Image streams not populated on a secure by default cluster | Why don't image streams populate on a secure by default cluster? |
sysdig-agent pods in CrashLoopBackOff on private-only RHCOS cluster |
Why are sysdig-agent pods in CrashLoopBackOff on a private-only RHCOS cluster? |
Worker nodes
| Error message | Troubleshooting topic |
|---|---|
The worker node instance ID changed. Reload the worker node if bare metal hardware was serviced. |
Classic: Why is the bare metal instance ID inconsistent with worker records? |
The dedicated hosts for the zone 'eu-de-2' are not ready. |
VPC: Why can't I create worker nodes on dedicated hosts? |
SoftLayerAPIError(SoftLayer_Exception_Public): Could not obtain network VLAN with id #123456. |
Classic: Why can't I add worker nodes with an invalid VLAN ID? |
Registration failed – The plan containers.kubernetes.vpc.gen2.roks is not available in <region>. |
Why do I see a Registration failed error when I try to provision or reload worker nodes? |
A VSI with this profile will put user over quota. |
VPC worker nodes fail to provision due to quota limits |
warning: Container container-00 is unable to start due to an error: Back-off pulling image "registry.redhat.io/rhel8/support-tools" |
After creating a version 4.15 cluster, my app no longer works |
Network health check (NHC) errors
The following error codes appear in the output of the ibmcloud oc cluster health issues command.
| Error code | Severity | Description | Troubleshooting topic |
|---|---|---|---|
NHC001 |
Warning | Tigera operator has been reporting that Calico is in 'progressing' state for over an hour. | Why does the Network status show an NHC001 error? |
NHC003 |
Warning | Some worker nodes in the cluster can not reach container image registries to pull images. | Why does the Network status show an NHC003 error? |
NHC004 |
Warning | Some worker nodes in the cluster can not resolve VPE gateway hostnames. | Why does the Network status show an NHC004 error? |
NHC005 |
Warning | Tigera operator is reporting that Calico is in 'degraded' state. | Why does the Network status show an NHC005 error? |
NHC006 |
Warning | One or more DNS resolvers are not reachable from certain worker nodes. | Why does the Network status show an NHC006 error? |
NHC007 |
Warning | One or more DNS resolvers are not reachable from certain worker nodes. | Why does the Network status show an NHC007 error? |
NHC009 |
Error | The IAM token exchange request failed. | Why does the Network status show an NHC009 error? |
NHC010 |
Error | Exceeded security group rules related quota. | Why does the Network status show an NHC010 error? |
NHC011 |
Error | Exceeded security group related quota. | Why does the Network status show an NHC011 error? |
Ingress status errors (ERR and ESS codes)
The following error codes appear in the output of the ibmcloud oc ingress status-report get command. Shared codes appear in both IBM Cloud Kubernetes Service and Red Hat OpenShift on IBM Cloud.
| Error code | Error message | Troubleshooting topic |
|---|---|---|
ERRDSIA |
The subdomain has incorrect addresses registered. | Ingress error: ERRDSIA |
ERRDRISS |
The subdomain has DNS resolution issues. | Ingress error: ERRDRISS |
ERRDSAISS |
The external provider for the given subdomain has authorization issues. | Ingress error: ERRDSAISS |
ERRDSISS |
The subdomain has TLS secret issues. | Ingress error: ERRDSISS |
ERRSAM |
The load balancer service address is missing. | Ingress error: ERRSAM |
ESSDNE |
The secret is not present on the cluster or is in the wrong namespace. | Ingress error: ESSDNE |
ESSEC |
The certificate for TLS secret expired or will expire soon. | Ingress error: ESSEC |
ESSEF |
The Opaque secret field expired or will expire soon. | Ingress error: ESSEF |
ESSSMG |
Could not find the secret group. | Ingress error: ESSSMG |
ESSSMI |
Could not access Secrets Manager instance. | Ingress error: ESSSMI |
ESSSMINF |
The Secrets Manager instance is not found. | Ingress error: ESSSMINF |
ESSVC |
The CRN does not match the default secret with the same domain. | Ingress error: ESSVC |
ESSWS |
The secret status shows a warning. | Ingress error: ESSWS |
ERRESNF |
The external service is missing. | Ingress error: ERRESNF |
ERRIODEG |
The Ingress Operator is in a degraded state. | Ingress error: ERRIODEG |
ERRIONF |
The Ingress Operator is missing from the cluster. | Ingress error: ERRIONF |
ERRRNA |
One or more routes not admitted. | Ingress error: ERRRNA |
ERRSAMO |
The load balancer service address is missing. | Ingress error: ERRSAMO |
ERRSEIPM |
The service is missing one or more worker IPs. | Ingress error: ERRSEIPM |
XXX.us-south.containers.appdomain.cloud: dial tcp: ... can't marshal DNS message |
Why does the DNS Operator show a RouteHealthDegraded or can't marshal DNS message error? |
Load balancers
| Error message | Troubleshooting topic |
|---|---|
The VPC load balancer that routes requests to this Kubernetes LoadBalancer service is offline. |
VPC clusters: Why can't my app connect via load balancer? |
The subnet with ID(s) '<subnet_id>' has insufficient available ipv4 addresses. |
VPC clusters: Why does a Kubernetes LoadBalancer service fail with no IPs? |
The load balancer was created in zone <zone>. This setting cannot be changed. |
VPC Clusters: My VPC NLB has a zone error and does not update |
Warning CreatingCloudLoadBalancerFailed ... Failed ensuring LoadBalancer: FindLoadBalancer failed ... 401 Unauthorized ... BXNIM0430E |
Why do I see SyncLoadBalancerFailed errors when creating a VPC cluster? |
| Security group protocol mismatch events on load balancer creation or update | VPC clusters: Security group protocol error creating or updating a LoadBalancer |
CAE003: Unable to determine the ingress IP address for the network load balancer. |
Classic clusters: Why does the master status have an ingress IP address for NLB error? |
| VPC load balancer health status shows only 2 of N instances as passing | VPC clusters: Why do I see VPC load balancer health status failures? |
Error on cloud load balancer ... Service and associated VPC load balancer do not match ... hostname.invalid |
Why does my Private Path NLB contain a hostname.invalid error? |
| Ingress subdomain DNS issue | Why does my Ingress subdomain have a DNS issue? |
Apps and services
| Error message | Troubleshooting topic |
|---|---|
Failed to create pod sandbox: rpc error: ... failed to request 1 IPv4 addresses. IPAM allocated only 0 |
Why don't my containers start? |
ImagePullBackOff or image pull authorization errors |
Why do images fail to pull from registry with ImagePullBackOff or authorization errors? |
pull QPS exceeded errors during image pulls |
Why do pods show pull QPS exceeded errors during image pulls? |
Error: failed to download "<helm_repo>/<chart_name>" |
Troubleshooting helm chart installation updated configuration values |
This service doesn't support creation of keys |
Resolving service binding errors in IBM Cloud clusters |
Pod remains in Pending state |
Why do pods remain in pending state? |
| Pod repeatedly fails to restart or is unexpectedly removed | Why do pods repeatedly fail to restart or are unexpectedly removed? |
error: build error: After retrying 2 times, Pull image still failed due to error: unauthorized: authentication required |
Why does my build error due to image pull authentication? |
received unexpected HTTP status: 504 Gateway Time-out |
Why does pushing to the internal registry time out? |
error: build error: Failed to push image: error copying layers and metadata |
Why does my pod not build with a permission denied error because of security context constraint (SCC)? |
dial tcp 161.26.0.28:443: connect: network is unreachable |
Why can't I push images to the internal registry from outside the VPC network? |
Pod is in CrashLoopBackOff status |
Why are my pods in a CrashLoopBackOff status? |
oc debug fails with a container is unable to start error |
Why does the oc debug command fail with a container is unable to start error? |
The entitlement 'ocp_entitled' was not found. |
Why do I see a license or entitlement error when creating a worker pool? |
Error: Unable to find a match: kernel-headers-VERSION kernel-devel-VERSION |
Why does my NVIDIA GPU driver installation fail on RHEL 9 worker nodes? |
Failed to get StorageCluster","error":"no matches for kind "StorageCluster" in version "ocs.openshift.io/v1" |
Why are no pods listed in the openshift-storage namespace? |
Pods in openshift-marketplace namespace in ImagePullBackOff |
Pods in the openshift-marketplace namespace are in ImagePullBackOff |
failed to set feature gates error on worker node upgrade |
Why do I see a failed to set feature gates error when upgrading a worker node? |
Confidential container scheduling failure: Insufficient kata.peerpods.io/vm |
How do I troubleshoot confidential containers? |
Permissions and credentials
| Error message | Troubleshooting topic |
|---|---|
User doesn't have permissions to create or manage Storage |
What permissions do I need to manage storage and create PVCs? |
Secure by default (SBD)
| Error message | Troubleshooting topic |
|---|---|
warning: Container container-00 is unable to start due to an error: Back-off pulling image "registry.redhat.io/rhel8/support-tools" |
After creating a version 4.15 cluster, my app no longer works |
Pending security group creation |
When I create a VPC cluster, my worker nodes are stuck in Pending security group creation |
Infrastructure instance status is 'failed': Can't start instance because provisioning failed. |
Why do I see DNS failures after adding a custom DNS resolver? |
| Nodeport apps not working after updating to version 4.15 | Fixing nodeport apps after updating cluster version 4.15 or later |
| Other clusters in the VPC failing after creating a version 4.15 cluster | After creating a version 4.15 cluster, applications running in other clusters in my VPC are failing |
| VSIs cannot access VPE gateway | Why can't my VSIs access VPE gateway? |
File Storage
Block Storage
Object Storage
| Error message | Troubleshooting topic |
|---|---|
pvc:...:can't access bucket <bucket_name>: NotFound: Not Found |
Why can't my PVC access an existing bucket? |
Error: symlink ... helm-ibmc: file exists |
Why does installing the Object storage Helm plug-in fail? |
d--------- 1 root root 0 Jan 1 1970 <file_name> (non-root user cannot access files) |
Resolving non-root user access issues to files in IBM Cloud |
EPERM: operation not permitted |
Why does my app pod fail with an Operation not permitted error? |
chown: changing ownership of '<volume_mount_path>': Input/output error |
Why can't the ownership of the mount path be changed? |
Error: rendered manifest contains a resource that already exists. ... existing_kind: storageClass |
Why does installing the IBM Cloud Object Storage plug-in fail? |
Bad value for ibm.io/object-store-endpoint ... scheme is missing. |
Why do I see wrong s3fs or IAM API endpoints when I create a PVC? |
SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. |
Why do I see wrong credentials or access denied messages when I create a PVC? |
| Object Storage PVC remains in pending state | Why does my PVC remain in a pending state? |
can't get credentials: can't get secret tsecret-key: secrets "secret-key" not found |
Why does PVC or pod creation fail due to not finding the Kubernetes secret? |
Transport endpoint is not connected. |
Why is the transport endpoint not connected? |
Error mounting volume: s3fs mount failed: s3fs: error while loading shared libraries: libfuse.so.2 |
Why do I see a volume mounting error when using the IBM Cloud Object Storage plug-in? |
| Transport endpoint not connected errors when using the COS cluster add-on | Why do I see transport endpoint not connected errors when using the IBM Cloud Object Storage cluster add-on? |
Portworx Storage
| Error message | Troubleshooting topic |
|---|---|
kp.Error: ... msg='Unauthorized: The user does not have access to the specified resource' |
Why does encryption fail with an invalid KMS endpoint? |
Red Hat OpenShift on IBM Cloud Data Foundation (ODF)
| Error message | Troubleshooting topic |
|---|---|
Failed to get StorageCluster","error":"no matches for kind "StorageCluster" in version "ocs.openshift.io/v1" |
Why are no pods listed in the openshift-storage namespace? |
ODF pods stuck at Pending |
Why are the OpenShift Data Foundation pods stuck at Pending? |
ODF storage cluster stuck at Progressing |
Why is the status of my OpenShift Data Foundation storage cluster stuck at Progressing? |
ceph-cluster-controller: failed to reconcile ... must be no more than 63 characters |
Why is the status of my OpenShift Data Foundation storage cluster stuck at Failed to reconcile? |