Secure secrets for apps with Vault Dedicated and Vault Secrets Operator

In this tutorial, you learn how to use IBM Cloud Vault Enterprise to manage secrets for applications that run in your IBM Cloud Kubernetes Service cluster by using the Vault Secrets Operator, HashiCorp's official Kubernetes operator.

You're a developer in an organization using Kubernetes Service to deploy containerized apps on IBM Cloud. Your team uses HashiCorp Vault for secrets management, and you want a native Vault integration for your Kubernetes workloads. The Vault Secrets Operator (VSO) provides deep integration with Vault, supporting advanced features like dynamic secrets, secret rotation, and Vault-native authentication methods.

With Vault Dedicated and Vault Secrets Operator, you can leverage the full power of HashiCorp Vault in your Kubernetes environment. Vault Secrets Operator provides a Kubernetes-native way to work with Vault secrets, supporting both static and dynamic secrets. For example, consider the following scenario:

The diagram shows the basic flow between Secrets Manager and your Kubernetes cluster.
External Secrets flow

  1. As a developer, you use Vault Dedicated to store secrets for an application that you want to deploy in a Kubernetes cluster.
  2. You configure the Vault Secrets Operator with VaultConnection and VaultAuth resources to connect to your Vault Dedicated instance.
  3. You create VaultStaticSecret or VaultDynamicSecret resources that define which secrets to sync.
  4. At application run time, VSO retrieves the secret data from Vault Dedicated and creates Kubernetes secrets for your cluster.
  5. VSO continuously monitors and syncs secrets, handling rotation and updates automatically.

Vault Secrets Operator is an official HashiCorp tool. For support and troubleshooting, refer to the official documentation.

Before you begin

Before you get started, be sure that you have Administrator platform access so that you can create account credentials and provision resources. You also need the following prerequisites:

Set up your environment

To work with Vault Dedicated and Kubernetes Service, you need to create a cluster and configure your Vault Dedicated instance with AppRole authentication.

Create a Kubernetes cluster

Create a Kubernetes cluster in your IBM Cloud account.

  1. From the command line, log in to IBM Cloud through the IBM Cloud CLI.

    ibmcloud login
    

    If the login fails, run the ibmcloud login --sso command to try again. The --sso parameter is required when you log in with a federated ID. If this option is used, go to the link listed in the CLI output to generate a one-time passcode.

  2. Select the account, region, and resource group where you want to create your cluster.

    ibmcloud target -r REGION -g RESOURCE_GROUP
    
  3. Create a Kubernetes cluster.

    ibmcloud ks cluster create vpc-gen2 --zone ZONE --flavor FLAVOR --workers 1 --name vso-test-cluster --vpc-id VPC_ID --subnet-id SUBNET_ID
    

    Provisioning takes 5 - 15 minutes.

  4. Verify that your cluster is provisioned successfully.

    ibmcloud ks worker ls --cluster vso-test-cluster
    

    Wait until the status changes to Ready.

  5. Set the context for your Kubernetes cluster.

    ibmcloud ks cluster config --cluster vso-test-cluster
    kubectl config current-context
    

Prepare your Vault Dedicated instance

Configure your Vault Dedicated instance with secrets and AppRole authentication for VSO.

  1. Export environment variables with your Vault Dedicated instance details.

    export VAULT_DEDICATED_ADDR="https://<your-vault_dedicated-instance-id>.vault.<region>.appdomain.cloud"
    export VAULT_DEDICATED_NAMESPACE="admin"
    export VAULT_TOKEN="<your-vault-token>"
    

    Replace <your-vault_dedicated-instance-id> with your Vault Dedicated instance ID, <region> with your Vault Dedicated region, and <your-vault-token> with your Vault token.

  2. Create a test secret in Vault Dedicated.

    curl -k -X POST \
      -H "X-Vault-Token: $VAULT_TOKEN" \
      -H "X-Vault-Namespace: $VAULT_DEDICATED_NAMESPACE" \
      -d '{"data":{"username":"vso-user","password":"vso-secure-pass-123"}}' \
      $VAULT_DEDICATED_ADDR/v1/kv/data/example_username_password
    

    Note that Vault Dedicated uses kv/ as the mount path for the KV secrets engine.

  3. Enable AppRole authentication for VSO.

    curl -k -X POST \
      -H "X-Vault-Token: $VAULT_TOKEN" \
      -H "X-Vault-Namespace: $VAULT_DEDICATED_NAMESPACE" \
      -d '{"type":"approle"}' \
      $VAULT_DEDICATED_ADDR/v1/sys/auth/approle
    

    VSO requires AppRole, Kubernetes, JWT, AWS, or GCP authentication. It does not support direct token authentication.

  4. Create a policy for VSO.

    curl -k -X PUT \
      -H "X-Vault-Token: $VAULT_TOKEN" \
      -H "X-Vault-Namespace: $VAULT_DEDICATED_NAMESPACE" \
      -d '{"policy":"path \"kv/data/*\" { capabilities = [\"read\", \"list\"] }\npath \"kv/metadata/*\" { capabilities = [\"read\", \"list\"] }"}' \
      $VAULT_DEDICATED_ADDR/v1/sys/policies/acl/kv-read
    
  5. Create an AppRole for VSO.

    curl -k -X POST \
      -H "X-Vault-Token: $VAULT_TOKEN" \
      -H "X-Vault-Namespace: $VAULT_DEDICATED_NAMESPACE" \
      -d '{"policies":["kv-read"],"token_ttl":"1h","token_max_ttl":"4h"}' \
      $VAULT_DEDICATED_ADDR/v1/auth/approle/role/vso-role
    
  6. Get the Role ID and Secret ID.

    export ROLE_ID=$(curl -k -X GET \
      -H "X-Vault-Token: $VAULT_TOKEN" \
      -H "X-Vault-Namespace: $VAULT_DEDICATED_NAMESPACE" \
      $VAULT_DEDICATED_ADDR/v1/auth/approle/role/vso-role/role-id | jq -r '.data.role_id')
    
    export SECRET_ID=$(curl -k -X POST \
      -H "X-Vault-Token: $VAULT_TOKEN" \
      -H "X-Vault-Namespace: $VAULT_DEDICATED_NAMESPACE" \
      $VAULT_DEDICATED_ADDR/v1/auth/approle/role/vso-role/secret-id | jq -r '.data.secret_id')
    
    echo "Role ID: $ROLE_ID"
    echo "Secret ID: $SECRET_ID"
    

Install Vault Secrets Operator

Install the Vault Secrets Operator using Helm.

  1. Add the HashiCorp Helm repository.

    helm repo add hashicorp https://helm.releases.hashicorp.com
    helm repo update
    
  2. Install Vault Secrets Operator.

    helm install vault-secrets-operator \
      hashicorp/vault-secrets-operator \
      --namespace vault-secrets-operator-system \
      --create-namespace \
      --version 0.9.0
    
  3. Verify the installation.

    kubectl get pods -n vault-secrets-operator-system
    

    Wait until all pods are in Running state.

  4. Verify the Custom Resource Definitions (CRDs) are installed.

    kubectl get crd | grep vault
    

    You should see CRDs like vaultauths, vaultconnections, vaultdynamicsecrets, and vaultstaticsecrets.

Configure VaultConnection and VaultAuth

Configure VSO to connect to your Vault Dedicated instance using VaultConnection and VaultAuth resources.

Create VaultConnection

  1. Create a Kubernetes secret with the AppRole SecretID.

    kubectl create secret generic approle-secret \
      --namespace default \
      --from-literal=id="$SECRET_ID"
    

    The key must be named id for VSO to recognize it.

  2. Create a vaultconnection.yaml file.

    touch vaultconnection.yaml
    
  3. Add the following configuration.

    apiVersion: secrets.hashicorp.com/v1beta1
    kind: VaultConnection
    metadata:
      name: vault-connection
      namespace: default
    spec:
      address: "<VAULT_DEDICATED_ADDR>"
      skipTLSVerify: true
    

    Replace <VAULT_DEDICATED_ADDR> with your Vault Dedicated instance address. For production, configure proper TLS instead of using skipTLSVerify.

  4. Apply the VaultConnection.

    kubectl apply -f vaultconnection.yaml
    

Create VaultAuth

  1. Create a vaultauth.yaml file.

    touch vaultauth.yaml
    
  2. Add the following configuration.

    apiVersion: secrets.hashicorp.com/v1beta1
    kind: VaultAuth
    metadata:
      name: vault-dedicates-auth
      namespace: default
    spec:
      vaultConnectionRef: vault-dedicated-connection
      method: appRole
      mount: approle
      namespace: admin
      appRole:
        roleId: vso-role
        secretRef: approle-secret
    
  3. Apply the VaultAuth.

    kubectl apply -f vaultauth.yaml
    
  4. Verify the VaultAuth status.

    kubectl get vaultauth vault-dedicated-auth -n default
    kubectl describe vaultauth vault-dedicated-auth -n default
    

Create VaultStaticSecret

Create a VaultStaticSecret resource to sync secrets from Vault Dedicated to Kubernetes.

  1. Create a vaultstaticsecret.yaml file.

    touch vaultstaticsecret.yaml
    
  2. Add the following configuration.

    apiVersion: secrets.hashicorp.com/v1beta1
    kind: VaultStaticSecret
    metadata:
      name: vault-dedicated-app-secret
      namespace: default
    spec:
      vaultAuthRef: vault-dedicated-auth
      mount: kv
      type: kv-v2
      path: example_username_password
      refreshAfter: 1h
      destination:
        name: my-k8s-secret-vso
        create: true
    

    This configuration fetches the secret from kv/data/example_username_password in Vault Dedicated and creates a Kubernetes secret named my-k8s-secret-vso. The secret refreshes every hour.

  3. Apply the VaultStaticSecret.

    kubectl apply -f vaultstaticsecret.yaml
    
  4. Verify the secret was synced.

    kubectl get vaultstaticsecret vault-dedicated-app-secret -n default
    kubectl get secret my-k8s-secret-vso -n default -o json | jq '.data | map_values(@base64d)'
    

    Example output:

    {
        "password": "vso-secure-pass-123",
        "username": "vso-user"
    }
    

Deploy an app to the cluster

Deploy an application that uses the synced secrets from Vault Dedicated.

  1. Create a test deployment.

    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: Pod
    metadata:
      name: test-app-vso
      namespace: default
    spec:
      containers:
      - name: app
        image: busybox
        command: ['sh', '-c', 'echo "Username: \$USERNAME"; echo "Password: \$PASSWORD"; sleep 3600']
        env:
        - name: USERNAME
          valueFrom:
            secretKeyRef:
              name: my-k8s-secret-vso
              key: username
        - name: PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-k8s-secret-vso
              key: password
    EOF
    
  2. Check the pod logs.

    kubectl logs test-app-vso -n default
    

    Expected output:

    Username: vso-user
    Password: vso-secure-pass-123
    

(Optional) Clean up resources

If you no longer need the resources, remove them from your account.

  1. Delete the test namespace and resources.

    kubectl delete pod test-app-vso -n default
    kubectl delete vaultstaticsecret vault-dedicated-app-secret -n default
    kubectl delete vaultauth vault-dedicated-auth -n default
    kubectl delete vaultconnection vault-dedicated-connection -n default
    kubectl delete secret approle-secret -n default
    
  2. Uninstall Vault Secrets Operator.

    helm uninstall vault-secrets-operator -n vault-secrets-operator-system
    kubectl delete namespace vault-secrets-operator-system
    
  3. Delete your test cluster.

    ibmcloud ks cluster rm --cluster vso-test-cluster
    
  4. Clean up Vault Dedicated test data.

    curl -k -X DELETE \
      -H "X-Vault-Token: $VAULT_TOKEN" \
      -H "X-Vault-Namespace: $VAULT_DEDICATED_NAMESPACE" \
      $VAULT_DEDICATED_ADDR/v1/kv/metadata/example_username_password
    

Notes of interest

Key considerations when using Vault Secrets Operator:

  1. Authentication methods: VSO does not support direct token authentication. You must use AppRole, Kubernetes, JWT, AWS, or GCP authentication methods.

  2. SecretID key name: When creating a Kubernetes secret for AppRole authentication, the key must be named id, not secret-id or secretId.

  3. Refresh interval: The refreshAfter field determines how often VSO checks for secret updates. Balance between freshness and API load.

  4. Automatic rollouts: Use rolloutRestartTargets in your VaultStaticSecret to automatically restart deployments when secrets change.

  5. Vault Dedicated mount path: Vault Dedicated uses kv/ as the default mount path for the KV secrets engine, not secret/.

  6. Vault Dedicated namespaces: Vault Dedicated uses Vault Enterprise namespaces. The default namespace is admin. Always specify the correct namespace in your VaultAuth configuration.

  7. TLS configuration: For production, configure proper TLS certificate validation instead of using skipTLSVerify.

Next steps

Great job! In this tutorial, you learned how to use Vault Secrets Operator to integrate Vault Dedicated with your Kubernetes cluster. Explore more VSO capabilities: