FAQ - Data management
Frequently asked questions can produce helpful answers and insight into best practices for working with IBM Cloud® Object Storage.
Can I migrate data from AWS S3 into IBM Cloud Object Storage?
Yes, you can use your existing tools to read and write data into IBM Cloud Object Storage. You need to configure HMAC credentials allow your tools to authenticate. Not all S3-compatible tools are currently unsupported. For details, see Using HMAC credentials.
How does IBM Cloud® Object Storage delete expired data?
Deletion of an object undergoes various stages to prevent data from being accessible (both before and after deletion). For details, see Data deletion.
What is the best way to structure your data by using Object Storage so you can 'look' at it and find what you are looking for?
You can use metadata that is associated with each object to find the objects you are looking for. The biggest advantage of Object Storage is the metadata that is associated with each object. Each object can have up to 4 MB of metadata in Object Storage. When offloaded to a database, metadata provides excellent search capabilities. Many (key, value) pairs can be stored in 4 MB. You can also use Prefix searching to find what you are looking for. For example, if you use buckets to separate each customer data, you can use prefixes within buckets for organization. For example: /bucket1/folder/object where 'folder/' is the prefix.
Can Object Storage partition the data automatically using HDFS, so I can read the partitions in parallel, for example, with Spark?
Object Storage supports a ranged GET on the object, so an application can do a distributed striped-read-type operation. Doing the striping is managed by the application.
Can I unzip a file after I upload it?
A feature to unzip or decompress files is not part of the service. For large data transfer, consider using Aspera high-speed transfer, multi-part uploads, or threads to manage multi-part uploads. See Store large objects.
How can I archive and restore objects in Object Storage?
Archived objects must be restored before you can access them. While restoring, specify the time limit the objects should remain available before being re-archived. For details, see archive-restore data.
Does an object in a bucket get overwritten if the same object name is used again in the same bucket?
Yes, the object is overwritten.
Are files scanned for viruses, while being uploaded to COS?
While there is no built in antivirus scanning in Object Storage, customers could enable a scanning workflow employing their own anti-virus technology that is deployed on Code Engine(/docs/codeengine?topic=codeengine-getting-started).
How can I use the Object Storage web console to download and upload large objects?
You can use IBM Cloud CLI or the API to download large objects. Alternatively, plugins such as Aspera /rclone can be used.
How do I access the reclaimed resources?
Create a new set of credentials to access the restored resources.
Is there a way to verify an object’s integrity during an upload to Object Storage?
Object Storage supports object integrity and ensures that the payload is not altered during transit.