Secure secrets for apps that run in your Kubernetes cluster

Create a service ID and API key

Start by creating the account credentials that you need to be able to run operations against Secrets Manager and Kubernetes Service.

1. From the command line, log in to IBM Cloud through the IBM Cloud CLI.

   ibmcloud login
   

   If the login fails, run the ibmcloud login --sso command to try again. The --sso parameter is required when you log in with a federated ID. If this option is used, go to the link listed in the CLI output to generate a one-time passcode.

2. Create a service ID and set it as an environment variable.

   export SERVICE_ID=`ibmcloud iam service-id-create kubernetes-secrets-tutorial --description "A service ID for testing ESO integration" --output json | jq -r ".id"`; echo $SERVICE_ID
   

3. Assign the service ID permissions to read secrets from Secrets Manager.

   ibmcloud iam service-policy-create $SERVICE_ID --roles "SecretsReader" --service-name secrets-manager
   

   By assigning SecretsReader service access, the External Secrets controller has the correct level of access to read secrets from Secrets Manager and populate them in a Kubernetes cluster.

4. Create an IBM Cloud API key for your service ID.

   export IBM_CLOUD_API_KEY=`ibmcloud iam service-api-key-create kubernetes-secrets-tutorial $SERVICE_ID --description "An API key for testing ESO integration." --output json | jq -r ".apikey"`
   

   You use this API key later to configure Secrets Manager for your cluster deployment.

Create a Kubernetes cluster and Secrets Manager instance

Create a Kubernetes cluster and an instance of Secrets Manager in your IBM Cloud account.

You can create one free Kubernetes cluster and Secrets Manager service instance per IBM Cloud account. If you already have both resources in your account, you can use your existing free cluster and Secrets Manager instance to complete the tutorial.

1. From the command line, select the account, region, and resource group where you want to create a Secrets Manager service instance.

   In this tutorial, you interact with the Dallas region. If you're logged in to a different region, be sure to set Dallas as your target region by running the following command.

   ibmcloud target -r us-south -g default
   

2. Create a Kubernetes cluster.

   ibmcloud ks cluster create classic --zone dal10 --flavor free --name my-test-cluster
   

3. Create a Secrets Manager instance.

   ibmcloud resource service-instance-create my-secrets-manager secrets-manager trial us-south
   

   Provisioning for both Secrets Manager and your Kubernetes cluster takes 5 - 15 minutes to complete.

4. Before you continue to the next step, verify that your cluster and Secrets Manager instance are provisioned successfully.

   1. Verify that the deployment of your worker node is complete.

      ibmcloud ks worker ls --cluster my-test-cluster
      

      When your worker node is finished provisioning, the status changes to Ready.

      ID                                                       Public IP       Private IP      Flavor   State          Status                Zone    Version
      kube-c39pf4ld0m87o3fv1utg-mytestclust-default-000000dd   169.xx.xx.xxx   10.xxx.xx.xxx   free     normal   Ready   mex01   1.20.7_1543
      

   2. Next, verify that your Secrets Manager instance provisioned successfully.

      ibmcloud resource service-instance my-secrets-manager
      

      When the instance is finished provisioning, the state changes to Active.

      Name:                  my-secrets-manager
      ID:                    crn:v1:bluemix:public:secrets-manager:us-south:a/f047b55a3362ac06afad8a3f2f5586ea:fe06948b-0c6b-4183-8d4b-e6c1d38ff65f::
      GUID:                  fe06948b-0c6b-4183-8d4b-e6c1d38ff65f
      Location:              us-south
      Service Name:          secrets-manager
      Service Plan Name:     trial
      Resource Group Name:   default
      State:                 active
      Type:                  service_instance
      Sub Type:
      Created at:            2021-01-06T17:11:32Z
      Created by:            zara@example.com
      Updated at:            2021-03-31T02:33:26Z
      

5. Set the context for your Kubernetes cluster in the CLI.

   ibmcloud ks cluster config --cluster my-test-cluster
   

6. Verify that kubectl commands run properly and that the Kubernetes context is set to your cluster.

   kubectl config current-context
   

   Example output:

   my-test-cluster/<your_cluster_ID>
   

Create a trusted profile

A trusted profile enables the External Secrets operator to read from Secrets Manager, without having to create a service ID or manage an API key.

1. Get the CRNs for your Secrets Manager instance and Kubernetes cluster.

   CLUSTER_CRN=$(ibmcloud ks cluster get --cluster my-test-cluster --output json | jq -r '.crn')
   SECRETS_MANAGER_CRN=$(ibmcloud resource service-instance my-secrets-manager --output JSON | jq -r '.[0].crn')
   

2. Create the profile.

   ibmcloud iam trusted-profile-create 'External Secrets'
   

3. Authorize the Kubernetes cluster to use the trusted profile.

   ibmcloud iam trusted-profile-rule-create 'External Secrets' --name kubernetes --type Profile-CR --conditions claim:namespace,operator:EQUALS,value:external-secrets --conditions claim:name,operator:EQUALS,value:external-secrets --conditions claim:crn,operator:EQUALS,value:$CLUSTER_CRN --cr-type IKS_SA
   

4. Create an access policy that allows the trusted profile to read secrets from your Secrets Manager instance.

   ibmcloud iam trusted-profile-policy-create 'External Secrets' --roles SecretsReader --service-instance $SECRETS_MANAGER_CRN --service-name secrets-manager
   

Prepare your Secrets Manager instance

Finally, configure your Secrets Manager instance to start working with secrets.

1. From the command line, verify that you can access the Secrets Manager CLI plug-in.

   ibmcloud secrets-manager --help
   

   Don't have the plug-in yet? To install the Secrets Manager CLI plug-in, run ibmcloud plugin install secrets-manager.

2. Export an environment variable with your unique Secrets Manager API endpoint URL.

   export SECRETS_MANAGER_URL=`ibmcloud resource service-instance my-secrets-manager --output json | jq -r '.[].dashboard_url | .[0:-3]'`; echo $SECRETS_MANAGER_URL
   

3. Create a secret group for your instance.

   Secret groups are a way to organize and control who on your team has access to specific secrets in your instance. To create a secret group from the IBM Cloud CLI, you use the ibmcloud secrets-manager secret-group-create command. Run the following command to create a secret group and store its ID as an environment variable.

   export SECRET_GROUP_ID=`ibmcloud secrets-manager secret-group-create --name my-test-secret-group --description "Read and write to my test app" --output json --service-url $SECRETS_MANAGER_URL | jq -r '.id'`; echo $SECRET_GROUP_ID
   

   Using a Windows™ command prompt (cmd.exe) or PowerShell? If you encounter errors with passing JSON content on the command line, you might need to adjust the strings for quotation-escaping requirements that are specific to your operating system. For more information, see Using quotation marks with strings in the IBM Cloud CLI.

   Success! Now you can store the secret in Secrets Manager that you want to populate in your Kubernetes cluster. Continue to the next step.

Configure External Secrets Operator for your cluster

Update your app configuration

After you install External Secrets Operator in your cluster, you can define Secrets Manager as the secrets backend for your application. Start by creating a configuration file that targets the secret in Secrets Manager that you want to use.

1. In the root directory of your application, create an external-secrets-example.yml file.

   touch external-secrets-example.yml
   

2. Modify the file to include information about the secret that you want to fetch from your Secrets Manager instance.

   apiVersion: external-secrets.io/v1beta1
   kind: SecretStore
   metadata:
     name: ibmcloud-secrets-manager-example
   spec:
     provider:
       ibm:
         serviceUrl: <endpoint_url>
         auth:
           secretRef:
             secretApiKeySecretRef:
               name: secret-api-key
               key: apikey
   ---
   apiVersion: external-secrets.io/v1beta1
   kind: ExternalSecret
   metadata:
     name: ibmcloud-secrets-manager-example
   spec:
     secretStoreRef:
       name: ibmcloud-secrets-manager-example
       kind: SecretStore
     target:
       name: ibmcloud-secrets-manager-example
     data:
     - secretKey: username
       remoteRef:
         property: username
         key: username_password/<SECRET_ID>
     - secretKey: password
       remoteRef:
         property: password
         key: username_password/<SECRET_ID>
   

   There are two ways to pull secrets - by Secret ID or Secret name. To pull by Secret ID use secret_type/secret_id. To pull by Secret name use secret_group/secret_type/secret_name.

   Replace <endpoint_url> with the Secrets Manager endpoint URL that you retrieved earlier. Replace <SECRET_ID> with the unique ID of the secret that you created in the previous step.

   If you're using a trusted profile to authenticate, replace the auth block with the following snippet.

         auth:
           containerAuth:
             profile: "External Secrets"
             iamEndpoint: https://iam.cloud.ibm.com
             tokenLocation: /var/run/secrets/tokens/sa-token
   

3. Apply the configuration to your cluster.

   kubectl apply -f external-secrets-example.yml
   

4. Verify that the External Secrets Operator is able to fetch the secret that is stored in your Secrets Manager instance.

   kubectl get secret ibmcloud-secrets-manager-example -o json | jq '.data | map_values(@base64d)'
   

   Example output:

   {
       "password": "cloudy-rainy-coffee-book",
       "username": "user123"
   }
   

   Success! You're now able to fetch the secret data that is stored in your Secrets Manager instance. Continue to the next step.