FAQ - General

How can I find out the total size of my bucket by using the API?

You can use the Resource Configuration API to get the bytes used for a given bucket.

How can I view my buckets?

You can view and navigate your buckets using the console, CLI or the API.

For example, the CLI command ibmcloud cos buckets will list all buckets associated with the targeted service instance.

Can I migrate data from AWS S3 into IBM Cloud Object Storage?

Yes, you can use your existing tools to read and write data into IBM Cloud Object Storage. You need to configure HMAC credentials allow your tools to authenticate. Not all S3-compatible tools are currently unsupported. For details, see Using HMAC credentials.

Can I use AWS S3 SDKs with IBM Cloud Object Storage?

IBM Cloud Object Storage supports the most commonly used subset of Amazon S3 API operations. IBM makes a sustained best effort to ensure that the IBM Cloud Object Storage APIs stay compatible with the industry standard S3 API. IBM Cloud Object Storage also produces several native core COS SDKs that are derivatives of publicly available AWS SDKs. These core COS SDKs are explicitly tested on each new IBM Cloud Object Storage upgrade. When using AWS SDKs, use HMAC authorization and an explicit endpoint. For details, see About IBM COS SDKs.

Is there a 100-bucket limit to an account? What happens if I need more?

Yes, 100 is the current bucket limit. Generally, prefixes are a better way to group objects in a bucket, unless the data needs to be in a different region or storage class. For example, to group patient records, you would use one prefix per patient. If this is not a workable solution and you require additional buckets, contact IBM customer support.

If I want to store my data by using IBM Cloud Object Storage Vault or Cold Vault, do I need to create another account?

No, storage classes (and regions as well) are defined at the bucket level. Simply create a new bucket that is set to the wanted storage class.

When I create a bucket by using the API, how do I set the storage class?

The storage class (for example, us-smart) is assigned to the LocationConstraint configuration variable for that bucket. This is because of a key difference between the way AWS S3 and IBM Cloud Object Storage handle storage classes. Object Storage sets storage classes at the bucket level, while AWS S3 assigns a storage class to an individual object. For a list of valid provisioning codes for LocationConstraint, see the Storage Classes guide.

Can the storage class of a bucket be changed? For example, if you have production data in 'standard', can we easily switch it to 'vault' for billing purposes if we are not using it frequently?

You can change the storage class by manually moving or copying the data from one bucket to another bucket with the wanted storage class.

Can the location of a bucket be changed?

To change a location, create a new bucket in the desired location and move existing data to the new bucket.

Does data consistency in Object Storage come with a performance impact?

Consistency with any distributed system comes with a cost, because the efficiency of the IBM Cloud Object Storage dispersed storage system is not trivial, but is lower compared to systems with multiple synchronous copies.

Aren't there performance implications if my application needs to manipulate large objects?

For performance optimization, objects can be uploaded and downloaded in multiple parts, in parallel.

How does IBM Cloud® Object Storage delete expired data?

Deletion of an object undergoes various stages to prevent data from being accessible (both before and after deletion). For details, see Data deletion.

How many objects can fit in a single bucket?

There is no practical limit to the number of objects in a single bucket.

Can I nest buckets inside one another?

No, buckets cannot be nested. If a greater level of organization is required within a bucket, the use of prefixes is supported: {endpoint}/{bucket-name}/{object-prefix}/{object-name}. The object's key remains the combination {object-prefix}/{object-name}.

What is the difference between 'Class A' and 'Class B' requests?

'Class A' requests are operations that involve modification or listing. This includes creating buckets, uploading or copying objects, creating or changing configurations, listing buckets, and listing the contents of buckets.'Class B' requests are those related to retrieving objects or their associated metadata/configurations from the system. There is no charge for deleting buckets or objects from the system.

What is the best way to structure your data by using Object Storage so you can 'look' at it and find what you are looking for? Without a directory structure, having thousands of files at one level is difficult to view.

You can use metadata that is associated with each object to find the objects you are looking for. The biggest advantage of Object Storage is the metadata that is associated with each object. Each object can have up to 4 MB of metadata in Object Storage. When offloaded to a database, metadata provides excellent search capabilities. Many (key, value) pairs can be stored in 4 MB. You can also use Prefix searching to find what you are looking for. For example, if you use buckets to separate each customer data, you can use prefixes within buckets for organization. For example: /bucket1/folder/object where 'folder/' is the prefix.

Can you confirm that Object Storage is ‘immediately consistent’, as opposed to ‘eventually consistent’?

Object Storage is ‘immediately consistent’ for data and ‘eventually consistent’ for usage accounting.

Can Object Storage partition the data automatically using HDFS, so I can read the partitions in parallel, for example, with Spark?

Object Storage supports a ranged GET on the object, so an application can do a distributed striped-read-type operation. Doing the striping is managed by the application.

Can I restore a bucket from a specific back-up file?

It is possible to overwrite an existing bucket. Restore options depend on the capabilities provided by the back-up tool you use; check with your back-up provider. As described in Your responsibilities when using IBM Cloud Object Storage, you are responsible for ensuring data back-ups if necessary. IBM Cloud® Object Storage does not provide a back-up service.

Can a web browser display the content of files stored in IBM Cloud Object Storage?

Web browsers can display web content in IBM Cloud Object Storage files, using the COS endpoint as the file location. To create a functioning website, however, you need to set up a web environment; for example, elements such as a CNAME record. IBM Cloud Object Storage does not support automatic static website hosting. For information, see Static websites and this tutorial.

If I set an archive policy on an existing bucket, does the policy apply to existing files?

The policy applies to the new objects uploaded but does not affect existing objects on a bucket. For details, see Add or manage an archive policy on a bucket.

Can I unzip a file after I upload it?

A feature to unzip or decompress files is not part of the service. For large data transfer, consider using Aspera high-speed transfer, multi-part uploads, or threads to manage multi-part uploads. See Store large objects.

Can I create a bucket, in the same or different region, with a deleted bucket name?

A bucket name can be reused as soon as 15 minutes after the contents of the bucket have been deleted and the bucket has been deleted. Then, the objects and bucket are irrevocably deleted and can not be restored.

If you do not first empty and then delete the bucket, and instead delete or schedule the Object Storage service instance for deletion, the bucket names will be held in reserve for a default period of seven (7) days until the account reclamation process is completed. Until the reclamation process is complete, it is possible to restore the instance, along with the buckets and objects. After reclamation is complete, all buckets and objects will be irrevocably deleted and can not be restored, although the bucket names will be made available for new buckets to reuse.

Why do CredentialRetrievalError occur while uploading data to Object Storage or while retrieving credentials? 

CredentialRetrievalError can occur due to the following reasons:

• The API key is not valid.
• The IAM endpoint is incorrect.

However, if the issue persists, contact IBM customer support.

How do I ensure communication with Object Storage?

You can check the communication with Object Storage by using one of the following:

• Use a COS API HEAD call to a bucket that will return the headers for that bucket. See api-head-bucket.

• Use SDK : See headbucket property.

Can I migrate a bucket from one COS instance to another?

Yes, You can achieve the same by creating a bucket in the target Object Storage instance and perform a sync. For details see cloud-object-storage-region-copy.

My COS service is locked. How do I reactivate the COS service?

Exceeding the data limit for the Lite account is one of the reasons why your account is locked or deactivated. The COS support team can help to unlock your account.

• The lite plan account can be activated only once.

• Upon enablement, reduce your storage to less than 25GB within a week to prevent it from getting disabled again.

After deleting a Object Storage instance, is it possible to reuse the same bucket names that were part of the deleted COS instance?

When an empty bucket is deleted, the name of the bucket is held in reserve by the system for 10 minutes after the delete operation.  After 10 minutes the name is released for re-use.

How can I archive and restore objects in Object Storage?

Archived objects must be restored before you can access them. While restoring, specify the time limit the objects should remain available before being re-archived. For details, see archive-restore data.

Can I enable Object Storage replication between two different regions for DR purposes?

Yes, it is possible to configure buckets for automated replication of objects to a destination bucket.

How can I track events in Object Storage?

The Object Storage Activity Tracker service records user-initiated activities that change the state of a service in Object Storage. For details, see IBM Cloud Activity Tracker.

How can I setup notifications when objects are updated or written to a bucket?

You can use Code Engine to receive events about actions taken on your bucket.

Does Object Storage have rate limits when writing to or reading from buckets?

Yes, Object Storage has rate limiting. For details, see COS support.

Is Object Storage HIPAA compliant to host PHI data?

Yes, Object Storage is HIPAA compliant.

Is there any option in Object Storage to enable accelerate data transfer?

Object Storage offers Aspera service for high speed data transfer.

How can I compare various attributes of an object in two different buckets?

Use rclone. It enables you to compare various attributes.

What is the default retention period for buckets?

There is no default retention period applied. You can set it while creating the bucket.

Can we add a retention policy to an existing bucket?

Yes, Retention policies can be added to an existing bucket; however, the retention period can only be extended. It cannot be decreased from the currently configured value.

Why is there a "legal hold" concept on top of the "retention period"?

A legal hold prevents an object from being overwritten or deleted. However, a legal hold does not have to be associated with a retention period and remains in effect until the legal hold is removed. For details, see Legal hold and retention period.

How can I access a private COS endpoint in a data center from another date center?

Use Object Storage Direct Link Connection to create a global direct link.

How does frequency of data access impact the pricing of Object Storage?

Storage cost for Object Storage is determined by the total volume of data stored, the amount of public outbound bandwidth used, and the total number of operational requests processed by the system. For details, see cloud-object-storage-billing.

What are the considerations for choosing the correct storage class in Object Storage?  

You can choose the correct storage class based on your requirement. For details, see billing-storage-classes.

How to invoke IBM Cloud Object Storage bucket operations using cURL?

You have the most power by using the command line in most environments with IBM Cloud Object Storage and cURL. However using cURL assumes a certain amount of familiarity with the command line and Object Storage. For details, see Using cURL.

Is encryption applied to a bucket by default?

Yes, by default, all objects stored in Object Storage are encrypted using randomly generated keys and an all-or-nothing-transform (AONT). You can get the encryption details using IBM Cloud UI/CLI. For details, see Cloud Storage Encryption.

How can I list all permissions of a bucket? 

The IAM feature creates a report at the instance level which may extend to their buckets. It does not specifically report at the bucket level. For details, see Account Access Report.

How can I monitor Object Storage resources?

Use the Activity Tracker service to capture and record Object Storage activities and monitor the activity of your IBM Cloud account. Activity Tracker is used to track how users and applications interact with Object Storage.

Does an object in a bucket get overwritten if the same object name is used again in the same bucket?

Yes, the object is overwritten.

How do I get bucket information without using the web console? 

Use the Object Storage Resource Configuration API to get bucket information. For details, see COS configuration and COS Integration.

How can I manage service credentials for Object Storage instances?

When a service credential is created, the underlying Service ID is granted a role on the entire instance of Object Storage. For details, see Managing Service credentials.

Why are parts of my credentials hidden or not viewable?

There may be an issue where the viewer does not have sufficient roles to view the credential information. For more information, see the account credentials documentation.

Is there a way to enable Key Protect on a Object Storage bucket after the bucket is created?

No, it is impossible to add Key Protect after creating a bucket. Key Protect can only be added while creating the bucket.

How can I move data into archive tier?

You can archive objects using the web console, REST API, and third-party tools that are integrated with IBM Cloud Object Storage. For details, see COS Archive.

Can I use the same Object Storage instance across multiple regions?

Yes, the Object Storage instance is a global service. Once an instance is created, you can choose the region while creating the bucket.

Are files scanned for viruses, while being uploaded to COS?

While there is no built in antivirus scanning in Object Storage, customers could enable a scanning workflow employing their own anti-virus technology that is deployed on Code Engine(/docs/codeengine?topic=codeengine-getting-started).

Is it possible to form a Hadoop cluster using Object Storage?

No, Object Storage is used for the object storage service. For a Hadoop cluster, you need the processing associated with each unit of storage. You may consider the Hadoop-as-a-Service setup.

Can I generate a "Pre-signed URL" to download a file and review?

A Pre-signed URL is not generated using the IBM Cloud UI; however, you can use CyberDuck to generate the “pre-signed URL”. It is free. For details, see this Slack channel.

How can I generate a Auth Token using the IAM API Key using REST?

For more information on working with the API, see Creating IAM token for API Key and Configuration Authentication.

How can I use the Object Storage web console to download and upload large objects?

You can use IBM Cloud CLI or the API to download large objects. Alternatively, plugins such as Aspera /rclone can be used.

What are the libraries that the Object Storage SDK supports?

Object Storage provides SDKs for Java, Python, NodeJS, and Go featuring capabilities to make the most of IBM Cloud Object Storage. For information about the features supported by each SDK, see the feature list.

When a file is uploaded to a cross region bucket using the ‘us-geo’ endpoint, how long is the delay before the file is available at the other US sites?

The data are spread immediately without delay and the uploaded files are available once the write is successful.

How do I access the reclaimed resources?

Create a new set of credentials to access the restored resources.

Can I host a website using a Object Storage bucket?

You can use Object Storage bucket to host a static website. For details, see Hosting Website using COS.

Are REST and cURL commands supported for Object Storage bucket creation using HMAC credentials?

Yes, you should setup an authorization header. For details, see Using HMAC Signature.

What kind of IAM authorization is required to edit a bucket's authorized IPs list? 

You must have 'Manager' privilege on the bucket to manage the firewall and to set the authorizations.

Can I convert a single region Object Storage bucket to cross region without having to copy objects?

No, you must copy objects to the target bucket. For details, see COS Region Copy.

Is there a way to verify an object’s integrity during an upload to Object Storage?

Object Storage supports object integrity and ensures that the payload is not altered during transit.

How can I set a notification when usage in a Object Storage instance is near a certain billing amount?

You can use a "soft" bucket quota feature by integrating with Metrics Monitoring and configuring for notifications. For details on establishing a hard quota that prevents usage beyond a set bucket size, see Using Bucket Quota.

Why am I unable to delete a Object Storage instance?

It isn't possible to delete an instance if the API key or Service ID being used is locked. You'll need to navigate in the console to Manage > Access (IAM) and unlock the API Key or Service ID. The error provided may seem ambiguous but is intended to increase security:

An error occurred during an attempt to complete the operation. Try fixing the issue or try the operation again later. Description: 400

This is intentionally vague to prevent any useful information from being conveyed to a possible attacker. For more information on locking API keys or Service IDs, see the IAM documentation.

How do I download the Root CA certificate for Object Storage?

Object Storage root CA certificates can be downloaded from https://www.digicert.com/kb/digicert-root-certificates.htm. Please download PEM or DER/CRT format from "DigiCert TLS RSA SHA256 2020 CA1" that is located under "Other intermediate certificates."

How do I delete a non-empty bucket when I do not see any objects in it?

There may be versioned objects or incomplete multipart uploads that are still within the bucket but aren't being displayed. Both of these can be cleaned up by setting an expiry policy to delete stale data.

Also, you can delete multipart uploads directly using the Minio client command: mc rm s3/ -I -r --force

How to I find my current active Object Storage instance/resources?

Login to the IBM Cloud shell: https://cloud.ibm.com/shell and enter at the prompt ibmcloud resource search "service_name:cloud-object-storage AND type:resource-instance".

The response you receive includes information for the name of your instance, location, family, resource type, resource group ID, CRN, tags, service tags, and access tags.

Why do I receive an error when I try to create a bucket?

Check IAM permissions because a user must have "Writer" permissions to create buckets.

Content-based restrictions may be preventing the user from acting on the service.