Why does my PVC remain in a pending state?
This troubleshooting topic applies only to Red Hat OpenShift clusters that run version 3.11.
When you create a PVC and you run oc get pvc <pvc_name>, your PVC remains in a Pending state, even after waiting for some time.
During the PVC creation and binding, many different tasks are executed by the IBM Cloud Object Storage plug-in. Each task can fail and cause a different error message.
Describe your PVC and review the common error messages.
- Find the root cause for why the PVC remains in a Pending state.oc describe pvc <pvc_name> -n <namespace>
- Review common error message descriptions and resolutions.
| Error message | Description | Steps to resolve | 
|---|---|---|
| can't get credentials: can't get secret <secret_name>: secrets "<secret_name>" not found | The Kubernetes secret that holds your IBM Cloud Object Storage service credentials does not exist in the same namespace as the PVC or pod. | See PVC or pod creation fails due to not finding the Kubernetes secret. | 
| can't get credentials: Wrong Secret Type.Provided secret of type XXXX.Expected type ibm/ibmc-s3fs | The Kubernetes secret that you created for your IBM Cloud Object Storage service instance does not include the type: ibm/ibmc-s3fs. | Edit the Kubernetes secret that holds your IBM Cloud Object Storage credentials to add or change the typetoibm/ibmc-s3fs. | 
| Bad value for ibm.io/object-store-endpoint XXXX: scheme is missing. Must be of the form http://<hostname> or https://<hostname>Bad value for ibm.io/iam-endpoint XXXX: scheme is missing. Must be of the form http://<hostname> or https://<hostname> | The s3fs API or IAM API endpoint has the wrong format, or the s3fs API endpoint could not be retrieved based on your cluster location. | See PVC creation fails due to wrong s3fs API endpoint. | 
| object-path can't be set when auto create is enabled | You specified an existing subdirectory in your bucket that you want to mount to your PVC by using the ibm.io/object-pathannotation. If you set a subdirectory, you must disable the bucket auto create feature. | In your PVC, set ibm.io/auto-create-bucket: "false"and provide the name of the existing bucket inibm.io/bucket. | 
| bucket auto-create must be enabled when bucket auto-delete is enabled | In your PVC, you set ibm.io/auto-delete-bucket: trueto automatically delete your data, the bucket, and the PV when you remove the PVC. This option requiresibm.io/auto-create-bucketto be set totrue,
                andibm.io/bucketto be set to""at the same time. | In your PVC, set ibm.io/auto-create-bucket: trueandibm.io/bucket: ""so that your bucket is automatically created with a name with the formattmp-s3fs-xxxx. | 
| bucket can't be set when auto-delete is enabled | In your PVC, you set ibm.io/auto-delete-bucket: trueto automatically delete your data, the bucket, and the PV when you remove the PVC. This option requiresibm.io/auto-create-bucketto be set totrue,
                andibm.io/bucketto be set to""at the same time. | In your PVC, set ibm.io/auto-create-bucket: trueandibm.io/bucket: ""so that your bucket is automatically created with a name with the formattmp-s3fs-xxxx. | 
| can't create bucket using API key without service-instance-id | If you want to use IAM API keys to access your IBM Cloud Object Storage service instance, you must store the API key and the ID of the IBM Cloud Object Storage service instance in a Kubernetes secret. | See Creating a secret for the object storage service credentials. | 
| object-path “<subdirectory_name>” not found inside bucket <bucket_name> | You specified an existing subdirectory in your bucket that you want to mount to your PVC by using the ibm.io/object-pathannotation. This subdirectory could not be found in the bucket that you specified. | Verify that the subdirectory that you specified in ibm.io/object-pathexists in the bucket that you specified inibm.io/bucket. | 
| BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again. | You set ibm.io/auto-create-bucket: trueand specified a bucket name at the same time, or you specified a bucket name that already exists in IBM Cloud Object Storage. Bucket names must be unique across all service instances and
                regions in IBM Cloud Object Storage. | Make sure that you set ibm.io/auto-create-bucket: falseand that you provide a bucket name that is unique in IBM Cloud Object Storage. If you want to use the IBM Cloud Object Storage plug-in to automatically create a bucket
                name for you, setibm.io/auto-create-bucket: trueandibm.io/bucket: "". Your bucket is created with a unique name in the formattmp-s3fs-xxxx. If you want to specify a bucket name for the
                auto created bucket, setibm.io/auto-create-bucket: trueandibm.io/auto-delete-bucket: falseandibm.io/bucket: "<bucket_name>". | 
| can't access bucket <bucket_name>: NotFound: Not Found | You tried to access a bucket that you did not create, or the storage class and s3fs API endpoint that you specified don't match the storage class and s3fs API endpoint that were used when the bucket was created. | See can't access an existing bucket. | 
| Put https://s3-api.dal-us-geo.objectstorage.service.networklayer.com/<bucket_name>: net/http: invalid header field value "AWS4-HMAC-SHA256 Credential=1234a12a123a123a1a123aa1a123a123   \n/20190412/us-standard/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=12aa1abc123456aabb12aas12aa123456sb123456abc" for key Authorization | The values in your Kubernetes secret are not correctly encoded to base64. | Review the values in your Kubernetes secret and encode every value to base64. You can also use the oc create secretcommand to create a new
                secret and let Kubernetes automatically encode your values to base64. | 
| can't access bucket <bucket_name>: Forbidden: Forbidden | You specified ibm.io/auto-create-bucket: falseand tried to access a bucket that you did not create, the secret access key or access key ID of your IBM Cloud Object Storage HMAC credentials are incorrect, or the regional endpoint
                of the bucket has changed. | You can't access a bucket that you did not create. Automatically create a bucket instead by setting ibm.io/auto-create-bucket: trueandibm.io/bucket: "". If you are the owner of the bucket, see PVC creation fails due to wrong credentials or access denied                to check your credentials. | 
| can't create bucket <bucket_name>: AccessDenied: Access Denied | You specified ibm.io/auto-create-bucket: trueto automatically create a bucket in IBM Cloud Object Storage, but the credentials that you provided in the Kubernetes secret are assigned the Reader IAM service
                access role. This role does not allow bucket creation in IBM Cloud Object Storage. | See PVC creation fails due to wrong credentials or access denied. | 
| can't create bucket <bucket_name>: AccessForbidden: Access Forbidden | You specified ibm.io/auto-create-bucket: trueand provided a name of an existing bucket inibm.io/bucket. In addition the credentials that you provided in the Kubernetes secret are assigned the Reader                IAM service access role. This role does not allow bucket creation in IBM Cloud Object Storage. | To use an existing bucket, set ibm.io/auto-create-bucket: falseand provide the name of your existing bucket inibm.io/bucket. To automatically create a bucket by using your existing Kubernetes secret, setibm.io/bucket: ""and follow PVC creation fails due to wrong credentials or access denied to verify the credentials in your Kubernetes secret. | 
| can't create bucket <bucket_name>: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. For more information, see REST Authentication and SOAP Authentication for details | The IBM Cloud Object Storage secret access key of your HMAC credentials that you provided in your Kubernetes secret is not correct. | See PVC creation fails due to wrong credentials or access denied. | 
| can't create bucket <bucket_name>: InvalidAccessKeyId: The AWS Access Key ID you provided does not exist in our records | The IBM Cloud Object Storage access key ID or the secret access key of your HMAC credentials that you provided in your Kubernetes secret is not correct. | See PVC creation fails due to wrong credentials or access denied. | 
| can't create bucket <bucket_name>: CredentialsEndpointError: failed to load credentialscan't access bucket <bucket_name>: CredentialsEndpointError: failed to load credentials | The IBM Cloud Object Storage API key of your IAM credentials and the GUID of your IBM Cloud Object Storage service instance are not correct. | See PVC creation fails due to wrong credentials or access denied. | 
| TokenManagerRetrieveError: error retrieving the token | This error occurs when you create a PVC with IAM credentials on a cluster that does not have public outbound access. | If your cluster does not have public outbound access, create an IBM Cloud Object Storage instance that uses HMAC credentials. | 
| set-access-policy not supported for classic cluster | This error occurs when you install the ibm-object-storage-pluginin a Classic cluster and set thebucketAccessPolicy=trueoption. ThebucketAccessPolicy=trueoption is only used with VPC clusters. | Install the plug-in and set the bucketAccessPolicy=falseoption. |