Secure secrets for apps with Vault Dedicated and Vault Secrets Operator
In this tutorial, you learn how to use IBM Cloud Vault Enterprise to manage secrets for applications that run in your IBM Cloud Kubernetes Service cluster by using the Vault Secrets Operator, HashiCorp's official Kubernetes operator.
You're a developer in an organization using Kubernetes Service to deploy containerized apps on IBM Cloud. Your team uses HashiCorp Vault for secrets management, and you want a native Vault integration for your Kubernetes workloads. The Vault Secrets Operator (VSO) provides deep integration with Vault, supporting advanced features like dynamic secrets, secret rotation, and Vault-native authentication methods.
With Vault Dedicated and Vault Secrets Operator, you can leverage the full power of HashiCorp Vault in your Kubernetes environment. Vault Secrets Operator provides a Kubernetes-native way to work with Vault secrets, supporting both static and dynamic secrets. For example, consider the following scenario:
- As a developer, you use Vault Dedicated to store secrets for an application that you want to deploy in a Kubernetes cluster.
- You configure the Vault Secrets Operator with VaultConnection and VaultAuth resources to connect to your Vault Dedicated instance.
- You create VaultStaticSecret or VaultDynamicSecret resources that define which secrets to sync.
- At application run time, VSO retrieves the secret data from Vault Dedicated and creates Kubernetes secrets for your cluster.
- VSO continuously monitors and syncs secrets, handling rotation and updates automatically.
Vault Secrets Operator is an official HashiCorp tool. For support and troubleshooting, refer to the official documentation.
Before you begin
Before you get started, be sure that you have Administrator platform access so that you can create account credentials and provision resources. You also need the following prerequisites:
-
jqhelps you slice and filter JSON data. You usejqin this tutorial to grab and use stored environment variables.
-
A Vault Dedicated instance provisioned in your IBM Cloud account. For more information, see Setting up your Vault Dedicated instance.
Set up your environment
To work with Vault Dedicated and Kubernetes Service, you need to create a cluster and configure your Vault Dedicated instance with AppRole authentication.
Create a Kubernetes cluster
Create a Kubernetes cluster in your IBM Cloud account.
-
From the command line, log in to IBM Cloud through the IBM Cloud CLI.
ibmcloud loginIf the login fails, run the
ibmcloud login --ssocommand to try again. The--ssoparameter is required when you log in with a federated ID. If this option is used, go to the link listed in the CLI output to generate a one-time passcode. -
Select the account, region, and resource group where you want to create your cluster.
ibmcloud target -r REGION -g RESOURCE_GROUP -
Create a Kubernetes cluster.
ibmcloud ks cluster create vpc-gen2 --zone ZONE --flavor FLAVOR --workers 1 --name vso-test-cluster --vpc-id VPC_ID --subnet-id SUBNET_IDProvisioning takes 5 - 15 minutes.
-
Verify that your cluster is provisioned successfully.
ibmcloud ks worker ls --cluster vso-test-clusterWait until the status changes to Ready.
-
Set the context for your Kubernetes cluster.
ibmcloud ks cluster config --cluster vso-test-cluster kubectl config current-context
Prepare your Vault Dedicated instance
Configure your Vault Dedicated instance with secrets and AppRole authentication for VSO.
-
Export environment variables with your Vault Dedicated instance details.
export VAULT_DEDICATED_ADDR="https://<your-vault_dedicated-instance-id>.vault.<region>.appdomain.cloud" export VAULT_DEDICATED_NAMESPACE="admin" export VAULT_TOKEN="<your-vault-token>"Replace
<your-vault_dedicated-instance-id>with your Vault Dedicated instance ID,<region>with your Vault Dedicated region, and<your-vault-token>with your Vault token. -
Create a test secret in Vault Dedicated.
curl -k -X POST \ -H "X-Vault-Token: $VAULT_TOKEN" \ -H "X-Vault-Namespace: $VAULT_DEDICATED_NAMESPACE" \ -d '{"data":{"username":"vso-user","password":"vso-secure-pass-123"}}' \ $VAULT_DEDICATED_ADDR/v1/kv/data/example_username_passwordNote that Vault Dedicated uses
kv/as the mount path for the KV secrets engine. -
Enable AppRole authentication for VSO.
curl -k -X POST \ -H "X-Vault-Token: $VAULT_TOKEN" \ -H "X-Vault-Namespace: $VAULT_DEDICATED_NAMESPACE" \ -d '{"type":"approle"}' \ $VAULT_DEDICATED_ADDR/v1/sys/auth/approleVSO requires AppRole, Kubernetes, JWT, AWS, or GCP authentication. It does not support direct token authentication.
-
Create a policy for VSO.
curl -k -X PUT \ -H "X-Vault-Token: $VAULT_TOKEN" \ -H "X-Vault-Namespace: $VAULT_DEDICATED_NAMESPACE" \ -d '{"policy":"path \"kv/data/*\" { capabilities = [\"read\", \"list\"] }\npath \"kv/metadata/*\" { capabilities = [\"read\", \"list\"] }"}' \ $VAULT_DEDICATED_ADDR/v1/sys/policies/acl/kv-read -
Create an AppRole for VSO.
curl -k -X POST \ -H "X-Vault-Token: $VAULT_TOKEN" \ -H "X-Vault-Namespace: $VAULT_DEDICATED_NAMESPACE" \ -d '{"policies":["kv-read"],"token_ttl":"1h","token_max_ttl":"4h"}' \ $VAULT_DEDICATED_ADDR/v1/auth/approle/role/vso-role -
Get the Role ID and Secret ID.
export ROLE_ID=$(curl -k -X GET \ -H "X-Vault-Token: $VAULT_TOKEN" \ -H "X-Vault-Namespace: $VAULT_DEDICATED_NAMESPACE" \ $VAULT_DEDICATED_ADDR/v1/auth/approle/role/vso-role/role-id | jq -r '.data.role_id') export SECRET_ID=$(curl -k -X POST \ -H "X-Vault-Token: $VAULT_TOKEN" \ -H "X-Vault-Namespace: $VAULT_DEDICATED_NAMESPACE" \ $VAULT_DEDICATED_ADDR/v1/auth/approle/role/vso-role/secret-id | jq -r '.data.secret_id') echo "Role ID: $ROLE_ID" echo "Secret ID: $SECRET_ID"
Install Vault Secrets Operator
Install the Vault Secrets Operator using Helm.
-
Add the HashiCorp Helm repository.
helm repo add hashicorp https://helm.releases.hashicorp.com helm repo update -
Install Vault Secrets Operator.
helm install vault-secrets-operator \ hashicorp/vault-secrets-operator \ --namespace vault-secrets-operator-system \ --create-namespace \ --version 0.9.0 -
Verify the installation.
kubectl get pods -n vault-secrets-operator-systemWait until all pods are in Running state.
-
Verify the Custom Resource Definitions (CRDs) are installed.
kubectl get crd | grep vaultYou should see CRDs like
vaultauths,vaultconnections,vaultdynamicsecrets, andvaultstaticsecrets.
Configure VaultConnection and VaultAuth
Configure VSO to connect to your Vault Dedicated instance using VaultConnection and VaultAuth resources.
Create VaultConnection
-
Create a Kubernetes secret with the AppRole SecretID.
kubectl create secret generic approle-secret \ --namespace default \ --from-literal=id="$SECRET_ID"The key must be named
idfor VSO to recognize it. -
Create a
vaultconnection.yamlfile.touch vaultconnection.yaml -
Add the following configuration.
apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultConnection metadata: name: vault-connection namespace: default spec: address: "<VAULT_DEDICATED_ADDR>" skipTLSVerify: trueReplace
<VAULT_DEDICATED_ADDR>with your Vault Dedicated instance address. For production, configure proper TLS instead of usingskipTLSVerify. -
Apply the VaultConnection.
kubectl apply -f vaultconnection.yaml
Create VaultAuth
-
Create a
vaultauth.yamlfile.touch vaultauth.yaml -
Add the following configuration.
apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultAuth metadata: name: vault-dedicates-auth namespace: default spec: vaultConnectionRef: vault-dedicated-connection method: appRole mount: approle namespace: admin appRole: roleId: vso-role secretRef: approle-secret -
Apply the VaultAuth.
kubectl apply -f vaultauth.yaml -
Verify the VaultAuth status.
kubectl get vaultauth vault-dedicated-auth -n default kubectl describe vaultauth vault-dedicated-auth -n default
Create VaultStaticSecret
Create a VaultStaticSecret resource to sync secrets from Vault Dedicated to Kubernetes.
-
Create a
vaultstaticsecret.yamlfile.touch vaultstaticsecret.yaml -
Add the following configuration.
apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultStaticSecret metadata: name: vault-dedicated-app-secret namespace: default spec: vaultAuthRef: vault-dedicated-auth mount: kv type: kv-v2 path: example_username_password refreshAfter: 1h destination: name: my-k8s-secret-vso create: trueThis configuration fetches the secret from
kv/data/example_username_passwordin Vault Dedicated and creates a Kubernetes secret namedmy-k8s-secret-vso. The secret refreshes every hour. -
Apply the VaultStaticSecret.
kubectl apply -f vaultstaticsecret.yaml -
Verify the secret was synced.
kubectl get vaultstaticsecret vault-dedicated-app-secret -n default kubectl get secret my-k8s-secret-vso -n default -o json | jq '.data | map_values(@base64d)'Example output:
{ "password": "vso-secure-pass-123", "username": "vso-user" }
Deploy an app to the cluster
Deploy an application that uses the synced secrets from Vault Dedicated.
-
Create a test deployment.
cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: test-app-vso namespace: default spec: containers: - name: app image: busybox command: ['sh', '-c', 'echo "Username: \$USERNAME"; echo "Password: \$PASSWORD"; sleep 3600'] env: - name: USERNAME valueFrom: secretKeyRef: name: my-k8s-secret-vso key: username - name: PASSWORD valueFrom: secretKeyRef: name: my-k8s-secret-vso key: password EOF -
Check the pod logs.
kubectl logs test-app-vso -n defaultExpected output:
Username: vso-user Password: vso-secure-pass-123
(Optional) Clean up resources
If you no longer need the resources, remove them from your account.
-
Delete the test namespace and resources.
kubectl delete pod test-app-vso -n default kubectl delete vaultstaticsecret vault-dedicated-app-secret -n default kubectl delete vaultauth vault-dedicated-auth -n default kubectl delete vaultconnection vault-dedicated-connection -n default kubectl delete secret approle-secret -n default -
Uninstall Vault Secrets Operator.
helm uninstall vault-secrets-operator -n vault-secrets-operator-system kubectl delete namespace vault-secrets-operator-system -
Delete your test cluster.
ibmcloud ks cluster rm --cluster vso-test-cluster -
Clean up Vault Dedicated test data.
curl -k -X DELETE \ -H "X-Vault-Token: $VAULT_TOKEN" \ -H "X-Vault-Namespace: $VAULT_DEDICATED_NAMESPACE" \ $VAULT_DEDICATED_ADDR/v1/kv/metadata/example_username_password
Notes of interest
Key considerations when using Vault Secrets Operator:
-
Authentication methods: VSO does not support direct token authentication. You must use AppRole, Kubernetes, JWT, AWS, or GCP authentication methods.
-
SecretID key name: When creating a Kubernetes secret for AppRole authentication, the key must be named
id, notsecret-idorsecretId. -
Refresh interval: The
refreshAfterfield determines how often VSO checks for secret updates. Balance between freshness and API load. -
Automatic rollouts: Use
rolloutRestartTargetsin your VaultStaticSecret to automatically restart deployments when secrets change. -
Vault Dedicated mount path: Vault Dedicated uses
kv/as the default mount path for the KV secrets engine, notsecret/. -
Vault Dedicated namespaces: Vault Dedicated uses Vault Enterprise namespaces. The default namespace is
admin. Always specify the correct namespace in your VaultAuth configuration. -
TLS configuration: For production, configure proper TLS certificate validation instead of using
skipTLSVerify.
Next steps
Great job! In this tutorial, you learned how to use Vault Secrets Operator to integrate Vault Dedicated with your Kubernetes cluster. Explore more VSO capabilities:
- Learn about VaultDynamicSecret for dynamic database credentials.
- Explore External Secrets Operator as an alternative multi-provider solution.
- Review the Vault Secrets Operator documentation.
- Learn more about Vault features and configurations.