Skip to content

Secret Management Guide

Securely manage secrets in USL applications across different environments and platforms.

Overview

USL supports multiple secret management backends: - EnvVars: Environment variables (development only) - Vault: HashiCorp Vault (recommended for production) - AWS Secrets Manager: AWS-native secret storage - Azure Key Vault: Azure-native secret storage - GCP Secret Manager: Google Cloud-native secret storage

Secret Backends

Environment Variables (Development)

Use for: Local development, testing

deployment {
  secrets: envvars
}

Generates standard Kubernetes Secret:

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
stringData:
  database-url: "postgresql://user:pass@localhost:5432/db"
  api-key: "dev-api-key"

⚠️ Warning: Never commit secrets to version control!

Use for: Production, multi-cloud, dynamic secrets

deployment {
  secrets: vault
}

Setup Vault

  1. Install Vault: 

    helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault \
  --namespace vault \
  --create-namespace

  2. Initialize and Unseal: 

    kubectl exec -it vault-0 -n vault -- vault operator init
kubectl exec -it vault-0 -n vault -- vault operator unseal

  3. Run Setup Script: 

    ./deployment/vault/vault-setup.sh

This script: - Enables KV secrets engine - Creates application secrets - Configures Kubernetes auth - Sets up database dynamic secrets - Creates access policies

External Secrets Operator

Install External Secrets Operator:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
  external-secrets/external-secrets \
  --namespace external-secrets-system \
  --create-namespace

Create SecretStore:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production
spec:
  provider:
    vault:
      server: "https://vault.example.com:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "my-app"
          serviceAccountRef:
            name: "my-app"

Generated ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: app-secrets
    creationPolicy: Owner
  data:
  - secretKey: database-url
    remoteRef:
      key: secret/data/app/database
      property: url

Vault Policies

Create policy (vault-policy.hcl):

# Read application secrets
path "secret/data/app/*" {
  capabilities = ["read", "list"]
}

# Get dynamic database credentials
path "database/creds/app" {
  capabilities = ["read"]
}

# Token operations
path "auth/token/renew-self" {
  capabilities = ["update"]
}

Apply policy:

vault policy write app-policy vault-policy.hcl

Kubernetes Auth

Configure Kubernetes authentication:

vault write auth/kubernetes/role/my-app \
  bound_service_account_names=my-app \
  bound_service_account_namespaces=production \
  policies=app-policy \
  ttl=24h

Dynamic Database Credentials

Configure database connection:

vault write database/config/mydb \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb" \
  username="vault_admin" \
  password="vault_password"

Create role:

vault write database/roles/my-app \
  db_name=mydb \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

AWS Secrets Manager

Use for: AWS ECS, EKS deployments

deployment {
  secrets: secretsmanager
}

Setup AWS Secrets Manager

  1. Create secrets: 

    aws secretsmanager create-secret \
  --name app/database-url \
  --secret-string "postgresql://user:pass@rds.amazonaws.com:5432/db" \
  --region us-east-1

aws secretsmanager create-secret \
  --name app/api-key \
  --secret-string "your-api-key" \
  --region us-east-1

  2. IAM Policy: 

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-east-1:ACCOUNT_ID:secret:app/*"
      ]
    }
  ]
}

  3. Attach to ECS Task Role: 

    aws iam attach-role-policy \
  --role-name app-task-role \
  --policy-arn arn:aws:iam::ACCOUNT_ID:policy/app-secrets-policy

ECS Task Definition

{
  "containerDefinitions": [{
    "name": "app",
    "secrets": [
      {
        "name": "DATABASE_URL",
        "valueFrom": "arn:aws:secretsmanager:us-east-1:ACCOUNT_ID:secret:app/database-url"
      },
      {
        "name": "API_KEY",
        "valueFrom": "arn:aws:secretsmanager:us-east-1:ACCOUNT_ID:secret:app/api-key"
      }
    ]
  }]
}

Azure Key Vault

Use for: Azure Container Apps, AKS deployments

deployment {
  secrets: keyvault
}

Setup Azure Key Vault

  1. Create Key Vault: 

    az keyvault create \
  --name app-keyvault \
  --resource-group app-rg \
  --location eastus

  2. Store secrets: 

    az keyvault secret set \
  --vault-name app-keyvault \
  --name database-url \
  --value "postgresql://user:pass@postgres.azure.com:5432/db"

az keyvault secret set \
  --vault-name app-keyvault \
  --name api-key \
  --value "your-api-key"

  3. Enable Managed Identity: 

    az containerapp identity assign \
  --name my-app \
  --resource-group app-rg \
  --system-assigned

  4. Grant Access: 

    IDENTITY_ID=$(az containerapp identity show \
  --name my-app \
  --resource-group app-rg \
  --query principalId -o tsv)

az keyvault set-policy \
  --name app-keyvault \
  --object-id $IDENTITY_ID \
  --secret-permissions get list

Container App Configuration

properties:
  configuration:
    secrets:
    - name: database-url
      keyVaultUrl: https://app-keyvault.vault.azure.net/secrets/database-url
    - name: api-key
      keyVaultUrl: https://app-keyvault.vault.azure.net/secrets/api-key
  template:
    containers:
    - name: app
      env:
      - name: DATABASE_URL
        secretRef: database-url
      - name: API_KEY
        secretRef: api-key

GCP Secret Manager

Use for: Cloud Run, GKE deployments

deployment {
  secrets: secretmanager
}

Setup GCP Secret Manager

  1. Enable API: 

    gcloud services enable secretmanager.googleapis.com

  2. Create secrets: 

    echo -n "postgresql://user:pass@postgres:5432/db" | \
  gcloud secrets create database-url --data-file=-

echo -n "your-api-key" | \
  gcloud secrets create api-key --data-file=-

  3. Grant Access: 

    gcloud secrets add-iam-policy-binding database-url \
  --member="serviceAccount:my-app@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

Cloud Run Configuration

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
      - name: my-app
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: database-url
              key: latest
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: api-key
              key: latest

Secret Rotation

Automatic Rotation with External Secrets

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 1h  # Sync every hour
  secretStoreRef:
    name: vault-backend
  target:
    name: app-secrets
    creationPolicy: Owner

Manual Rotation

# Update secret in Vault
vault kv put secret/app/database url="new-connection-string"

# Force sync (if needed)
kubectl annotate externalsecret app-secrets \
  force-sync=$(date +%s) \
  --overwrite

Rolling Restart on Secret Change

Use Reloader to automatically restart pods:

helm repo add stakater https://stakater.github.io/stakater-charts
helm install reloader stakater/reloader

Add annotation to Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    reloader.stakater.com/auto: "true"

Best Practices

1. Never Commit Secrets

✅ Good: Use secret management systems ❌ Bad: Hardcode in code or config files

# .gitignore
*.env
secrets.yaml
*.pem
*.key

2. Use Separate Secrets per Environment

vault kv put secret/app/dev database_url="..."
vault kv put secret/app/staging database_url="..."
vault kv put secret/app/prod database_url="..."

3. Principle of Least Privilege

Grant minimal permissions:

# Only read, not write
path "secret/data/app/*" {
  capabilities = ["read"]
}

4. Short-Lived Credentials

Use dynamic secrets:

# 1-hour TTL
vault write database/roles/app \
  default_ttl="1h" \
  max_ttl="24h"

5. Audit Secret Access

Enable Vault audit logging:

vault audit enable file file_path=/vault/logs/audit.log

6. Encrypt at Rest

Ensure secrets are encrypted: - Kubernetes: Enable encryption at rest - Cloud providers: Use KMS encryption

Security Checklist

  • Secrets not in version control
  • Separate secrets per environment
  • Minimal IAM permissions
  • Encryption at rest enabled
  • Encryption in transit (TLS)
  • Secret rotation enabled
  • Audit logging enabled
  • Access reviews regular
  • Backup secrets securely
  • Incident response plan

Troubleshooting

ExternalSecret Not Syncing

kubectl describe externalsecret app-secrets -n production
kubectl logs -l app=external-secrets -n external-secrets-system

Vault Connection Issues

# Test connection
kubectl run vault-test --rm -it --image=vault:latest -- \
  vault kv get -address=http://vault:8200 secret/app/database

# Check auth
kubectl exec -it pod-name -- \
  cat /var/run/secrets/kubernetes.io/serviceaccount/token

AWS Secrets Manager Access Denied

# Check IAM role
aws sts get-caller-identity

# Test secret access
aws secretsmanager get-secret-value \
  --secret-id app/database-url \
  --region us-east-1

Next Steps

References