IBM Cloud Docs
Access a storage bucket by using a dynamic secret

Access a storage bucket by using a dynamic secret

In this tutorial, you learn how to use IBM Cloud® Secrets Manager to create and lease an IAM credential that can be used to access a bucket in Cloud Object Storage.

As an enterprise developer, you might be looking for ways to improve the security of your application secrets. When it comes to managing API keys, you want the ability to create your credentials dynamically so that they exist only when you need them to. You also want to lease an API key to someone else on your team and ensure that it is automatically revoked after a time duration that you specify.

With Secrets Manager, you can create a dynamic secretA unique value, such as a password or an API key, that is created dynamically and leased to an application that requires access to a protected resource. After a dynamic secret reaches the end of its lease, access to the protected resource is revoked and the secret is deleted automatically. that you can use to access a protected resource, such as deployment logs that you store in a Cloud Object Storage bucket. For example, consider the following scenario.

The diagram shows the basic flow between the Secrets Manager and Cloud Object Storage services.
IAM credential flow

  1. As an admin user, you want to create a dynamic secret that your team can use to access a Cloud Object Storage bucket in your account. You send a request to create IAM credentials in Secrets Manager.
  2. Secrets Manager creates the secret and validates it against your defined IAM access policies.
  3. Later, a developer wants to access the contents of your storage bucket. The developer sends a request to retrieve the value of your IAM credential.
  4. Secrets Manager validates the request and generates a single-use API key that the developer can use to authenticate to Cloud Object Storage. After the API key reaches the end of its lease, the API key is revoked automatically.

Before you begin

Before you get started, be sure that you have Administrator platform access so that you can provision services, create access groups, and customize access policies for others. You also need the following prerequisites:

Set up your environment

To work with Secrets Manager and Cloud Object Storage, you need to create instances of both services in your IBM Cloud account. You also need to configure permissions so that you can run operations against both services.

In this step, you set up an access environment by creating an access group, a service ID, and an IBM Cloud API key. At the end of the tutorial, you can easily remove both instances if you no longer need them.

You can have only one Secrets Manager service instance in your IBM Cloud account.

Create Secrets Manager and Cloud Object Storage instances

Start by creating test instances of Secrets Manager and Cloud Object Storage in your IBM Cloud account.

  1. On the command line, log in to IBM Cloud through the IBM Cloud CLI.

    ibmcloud login
    

    If the login fails, run the ibmcloud login --sso command to try again. The --sso parameter is required when you log in with a federated ID. If this option is used, go to the link listed in the CLI output to generate a one-time passcode.

  2. Select the account, region, and resource group where you want to create a Secrets Manager service instance.

    In this tutorial, you interact with the Dallas region. If you're logged in to a different region, be sure to set Dallas as your target region by running the following command.

    ibmcloud target -r us-south -g default
    
  3. Create a Cloud Object Storage instance.

    ibmcloud resource service-instance-create test-cos-instance-tutorial cloud-object-storage lite us-south
    
  4. Create a Secrets Manager instance.

    ibmcloud resource service-instance-create test-sm-instance-tutorial secrets-manager lite us-south
    

    The provisioning process for Secrets Manager takes 5 - 15 minutes to complete.

  5. Export environment variables with the IDs of both service instances.

    export COS_INSTANCE_ID=`ibmcloud resource service-instance "test-cos-instance-tutorial" --output json | jq -r ".[].guid"`; echo $COS_INSTANCE_ID
    
    export SM_INSTANCE_ID=`ibmcloud resource service-instance "test-sm-instance-tutorial" --output json | jq -r ".[].guid"`; echo $SM_INSTANCE_ID
    
  6. (Optional) Verify that your services were created successfully by clicking the Menu icon Menu icon > Resource List in your IBM Cloud console.

Set up access permissions

Next, define the access hierarchy that you need to be able to run operations against the Secrets Manager and Cloud Object Storage services.

  1. Create a test access group in your IBM Cloud account.

    By creating an access group, you're able to assign access to the set of users and service IDs that require access to your Secrets Manager and Cloud Object Storage instances. From the command line, run the following command to create a new access group called test-cos-admin-group.

    export ACCESS_GROUP_ID=`ibmcloud iam access-group-create test-storage-admin-group -d "An access group for testing Secrets Manager and Cloud Object Storage." --output json | jq -r ".id"`; echo $ACCESS_GROUP_ID
    
  2. Set up your account credentials.

    Create a service ID and set it as an environment variable.

    export SERVICE_ID=`ibmcloud iam service-id-create test-service-id-secrets --description "A service ID for testing Secrets Manager." --output json | jq -r ".id"`; echo $SERVICE_ID
    

    Assign the service ID permissions to create and manage other service IDs.

    ibmcloud iam service-policy-create $SERVICE_ID --roles Operator --service-name "IAM Identity Service"
    

    Assign the service ID permissions to view and update access groups in your account.

    ibmcloud iam access-group-policy-create $SERVICE_ID --roles Editor --service-name "IAM Access Groups"
    

    Add the service ID to your access group.

    ibmcloud iam access-group-service-id-add test-secrets-admin-group $SERVICE_ID
    

    Create an IBM Cloud API key for your service ID.

    export IBM_CLOUD_API_KEY=`ibmcloud iam service-api-key-create test-storage-read-write $SERVICE_ID --description "An API key for testing Secrets Manager and Cloud Object Storage." --output json | jq -r ".apikey"`
    

    You use this API key later to enable the Secrets Manager IAM secrets engine.

Prepare your Cloud Object Storage instance

Next, create a bucket in your Cloud Object Storage instance and set up access.

  1. Create a test bucket in your instance.

    1. In the IBM Cloud console, go to your Resource List.
    2. From your list of storage services, select test-cos-instance-tutorial.
    3. In the Cloud Object Storage UI, click Create bucket.
    4. Create a bucket in the us-south region.
    5. Copy the ID of the bucket.
  2. Assign read and write access to your new Cloud Object Storage bucket.

    1. Go to Manage > Access (IAM) > Access groups.
    2. From your list of groups, select test-storage-admin-group.
    3. Click the Access tab.
    4. Click Assign access.
    5. From the list of services, select Cloud Object Storage and click Next.
    6. In the Resources section, select Specific resources.
    7. In the Attribute type field, select Service instance.
    8. From the list of service instances, select test-cos-instance-tutorial and click Next.
    9. Assign the Reader, Content Reader, Object Reader, and Object Writer service access roles.
    10. Review your selections and click Add.
    11. Click Assign.
  3. Upload an object to the storage bucket.

    You can drag and drop any file or folder from your local system. For example, you can create and upload a file that is named sample.txt with the following sample text.

    A quick brown fox jumped over the lazy dog.
    

Prepare your Secrets Manager instance

Finally, configure your Secrets Manager instance to start working with dynamic secrets.

  1. From the command line, verify that you can access the Secrets Manager CLI plug-in.

    ibmcloud secrets-manager --help
    

    Don't have the plug-in? To install the Secrets Manager CLI plug-in, run ibmcloud plugin install secrets-manager.

  2. Create a secret group for your instance.

    Secret groups are a way to organize and control who on your team has access to specific secrets in your instance. To create a secret group from the IBM Cloud CLI, run the ibmcloud secrets-manager secret-group-create command.

    export SECRET_GROUP_ID=`ibmcloud secrets-manager secret-group-create --name cloud-object-storage-writers --description "Read and write to Cloud Object storage buckets" --service-url https://${instance-id}.${region}.secrets-manager.appdomain.cloud --output json | jq -r '.id'`; echo $SECRET_GROUP_ID
    

    Be sure to update instance_id and region to yours.

    Using a Windows™ command prompt (cmd.exe) or PowerShell? If you encounter errors with passing JSON content on the command line, you might need to adjust the strings for quotation-escaping requirements that are specific to your operating system. For more information, see Using quotation marks with strings in the IBM Cloud CLI.

  3. Enable the IAM secrets engine for your instance.

    Secret engines are components in Secrets Manager that are used to process operations for secrets of different types. These engines serve as backends for those secrets. By enabling the IAM secrets engine, you can create an API key for a service ID dynamically, and then lease it to a user based on the lease duration that you specify.

    To configure the IAM secrets engine from the IBM Cloud CLI, run the ibmcloud secrets-manager config-update command.

    Be sure to first create an IAM Service ID and API Key with the required access policies. Use the generated API key in the command below.

    ibmcloud secrets-manager configuration-create --config-type iam_credentials_configuration --iam-credentials-apikey $IBM_CLOUD_API_KEY
    

    Success! Now you can create an IAM credential that you can use to dynamically generate service IDs and API keys. Continue to the next step.

Create an IAM credential

IAM credentials are dynamic secrets that you can use to access an IBM Cloud resource on-demand, such as a Cloud Object Storage bucket. A set of IAM credentials consists of a service ID and an API key that is generated each time that the protected resource is read or accessed. You can define a time-to-live (TTL) or a lease duration for your IAM credential at its creation so that you shorten the amount of time that the secret exists.

To create an IAM credential from the IBM Cloud CLI, run the ibmcloud secrets-manager secret-create command.

export SECRET_ID=`ibmcloud secrets-manager secret-create --secret-name test-iam-credentials --secret-description "Extended description for my secret.",--am-credentials-access_groups $ACCESS_GROUP_ID" --secret_group_id $SECRET_GROUP_ID --iam-credentials-ttl 2h --secret-labels "storage, us-south" --output json --service-url https://${instance-id}.${region}.secrets-manager.appdomain.cloud | jq -r '.id`

Be sure to update instance_id and region to yours.

You can also create an IAM credential by using the Secrets Manager UI. For more information, see Creating IAM credentials.

Generate an API key

After you create the IAM credential, a user with lesser privileges can retrieve the secret when it's time to access the storage bucket.

To retrieve an IAM credential from the IBM Cloud CLI, you can run the ibmcloud secrets-manager secret command.

  1. Retrieve the contents of your IAM credential secret.

    ibmcloud secrets-manager secret --id $SECRET_ID --output json
    

    The following JSON snippet shows the example response.

    {
        "metadata": {
        "collection_type": "application/vnd.ibm.secrets-manager.secret+json",
        "collection_total": 1
        },
        "resources": [
        {
          "access_groups": [
            "AccessGroupId-e7e1a364-c5b9-4027-b4fe-083454499a20"
          ],
          "api_key": "pVTL3W7o1uqIWSE8sSh8ebNhRN4-d1D0W9GcPVgwcLUr",
          "created_by": "iam-ServiceId-222b47ab-b08e-4619-b68f-8014a2c3acb8",
          "creation_date": "2020-12-15T14:31:42Z",
          "crn": "crn:v1:bluemix:public:secrets-manager:us-south:a/a5ebf2570dcaedf18d7ed78e216c263a:f1bc94a6-64aa-4c55-b00f-f6cd70e4b2ce:secret:cb7a2502-8ede-47d6-b5b6-1b7af6b6f563",
          "description": "Extended description for my secret.",
          "labels": [
              "storage",
              "us-south"
          ],
          "last_update_date": "2020-12-15T14:31:42Z",
          "name": "test-iam-credentials",
          "secret_group_id": "432b91f1-ff6d-4b47-9f06-82debc236d90",
          "secret_type": "IAM_CREDENTIALS",
          "service_id": "ServiceId-0576925a-9651-4f3e-b8c7-ce1332324aa4",
          "state": 1,
          "state_description": "Active",
          "ttl": 900
        }
        ]
    }
    

    The response body shows the single-use api_key and service_id values that are generated for your secret. The ttl value indicates how long your credentials are valid (in seconds).

  2. Export an environment variable with the newly generated API key.

    Copy the api_key value that you generated in the previous step, and set it as an environment variable.

    export API_key="<api_key>"
    

Use the API key to generate an access token

By using the API key that you retrieved in the previous step, you can generate an IAM access token. The token is valid only for the duration that you initially defined for the IAM credential.

You can exchange the API key for an IAM access token by calling IAM Identity Services API. To generate an IAM token, run the following cURL command from your command line.

export IAM_TOKEN=`curl -X POST \
    "https://iam.cloud.ibm.com/identity/token" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -H "Accept: application/json" \
    -d "grant_type=urn%3Aibm%3Aparams%3Aoauth%3Agrant-type%3Aapikey&apikey=$API_KEY" | jq -r ".access_token"`; echo $IAM_TOKEN

Success! You now have a valid IAM token that you can use to access your Cloud Object Storage bucket. Continue to the next step.

Access a storage bucket

As a user with lesser privileges, you can now read and write to your Cloud Object Storage bucket by using the access token that you generated in the previous step. After the lease duration of your IAM credentials expires, its associated API key is revoked automatically.

On the command line, run the following cURL request to list the buckets in your Cloud Object Storage instance.

curl -X GET \
    "https://s3.us-south.cloud-object-storage.appdomain.cloud" \
    -H "Authorization: Bearer $IAM_TOKEN" \
    -H "ibm-service-instance-id: $COS_INSTANCE_ID" \
    -H "Accept: application/json"

The following XML snippet shows an example response.

<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Owner>
        <ID>cccbd89c-6eb0-4aed-8ffb-899d39bc022e</ID>
        <DisplayName>cccbd89c-6eb0-4aed-8ffb-899d39bc022e</DisplayName>
    </Owner>
    <Buckets>
        <Bucket>
            <Name>test-secrets-tutorial</Name>
            <CreationDate>2020-12-14T21:40:55.739Z</CreationDate>
        </Bucket>
    </Buckets>
</ListAllMyBucketsResult>

To see the contents of your test-secrets-tutorial bucket, you can run the following command.

curl -X GET \
    "https://s3.us-south.cloud-object-storage.appdomain.cloud/test-secrets-tutorial" \
    -H "Authorization: Bearer $IAM_TOKEN" \
    -H "ibm-service-instance-id: $COS_INSTANCE_ID" \
    -H "Accept: application/json"

The following XML snippet shows an example response.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Name>test-secrets-tutorial</Name>
    <Prefix></Prefix>
    <Marker></Marker>
    <MaxKeys>1000</MaxKeys>
    <Delimiter></Delimiter>
    <IsTruncated>false</IsTruncated>
    <Contents>
        <Key>sample.txt</Key>
        <LastModified>2020-12-14T21:49:29.502Z</LastModified>
        <ETag>"df756a3769fcab0a261880957590c768"</ETag>
        <Size>42</Size>
        <Owner>
            <ID>cccbd89c-6eb0-4aed-8ffb-899d39bc022e</ID>
            <DisplayName>cccbd89c-6eb0-4aed-8ffb-899d39bc022e</DisplayName>
        </Owner>
        <StorageClass>STANDARD</StorageClass>
    </Contents>
</ListBucketResult>

Finally, to read the object in the bucket, you can run the following command.

curl -X GET \
    "https://s3.us-south.cloud-object-storage.appdomain.cloud/test-secrets-tutorial/sample.txt" \
    -H "Authorization: Bearer $IAM_TOKEN" \
    -H "ibm-service-instance-id: $COS_INSTANCE_ID" \
    -H "Accept: application/json"

Replace sample.txt with the name of the file that you uploaded to your bucket. The following screen shows an example response.

A quick brown fox jumps over the lazy dog.

(Optional) Clean up resources

If you no longer need the resources that you created in this tutorial, you can complete the following steps to remove them from your account.

  1. Delete your test Cloud Object Storage instance.

    ibmcloud resource service-instance-delete test-cos-instance-tutorial
    
  2. Delete your test Secrets Manager instance.

    ibmcloud resource service-instance-delete test-sm-instance-tutorial
    
  3. Delete your test access group.

    ibmcloud resource access-group-delete $ACCESS_GROUP
    
  4. Delete your test service ID.

    ibmcloud resource service-id-delete $SERVICE_ID
    

Next steps

Great job! In this tutorial, you learned how to set up Secrets Manager to access a Cloud Object Storage bucket with a dynamic secret. Check out more resources to help you get started with Secrets Manager.