How-To Guide

Bootstrap Application Secrets with ESO

Type: How-To (Task-oriented)

Related: Tutorial: Cross-Namespace Secrets | Application Secrets Architecture

Goal: Set up application secrets using the External Secrets Operator (ESO) namespace isolation pattern for secure, centralized secret management.

Time: ~30 minutes

When to use this pattern:

• Application has multiple secrets (>3)

• Need centralized secret rotation

• Want to avoid committing secrets to Git

• Require namespace isolation for security

• Multiple environments (dev/staging/prod)

Prerequisites

• External Secrets Operator installed (included in infrastructure tier)

• kubectl access to the cluster

• Application namespace exists or will be created

• Basic understanding of Kubernetes Secrets and RBAC

Architecture Overview

This pattern uses a three-tier namespace architecture:

┌──────────────────────────────────────────────┐
│ application-secrets (source namespace)       │
│ - Source secrets (bootstrap once)            │
│ - ServiceAccount for ESO                     │
│ - RBAC (Role + RoleBinding)                  │
└──────────────────────────────────────────────┘
                    ↓
        (ClusterSecretStore with namespace restrictions)
                    ↓
┌──────────────────────────────────────────────┐
│ <your-app> (target namespace)                │
│ - ExternalSecrets (declarative config)       │
│ - Synced secrets (auto-managed by ESO)       │
│ - Application pods (consume secrets)         │
└──────────────────────────────────────────────┘

Key benefits:

• Source secrets stored in dedicated namespace (not Git)

• ClusterSecretStore restricted to specific namespaces

• Automatic synchronization via ESO

• Centralized rotation (update once, sync everywhere)

Step 1: Create the application-secrets Namespace

If it doesn’t exist yet:

kubectl create namespace application-secrets

Verify:

kubectl get namespace application-secrets

Step 2: Create Source Secrets

Create secrets in the application-secrets namespace. These are the source of truth.

Option A: Manual creation for testing

# Example: Database credentials
kubectl create secret generic <app-name>-db-secrets \
  --from-literal=username=myapp \
  --from-literal=password=$(openssl rand -base64 32) \
  --from-literal=host=postgres.myapp.svc.cluster.local \
  --from-literal=database=myapp_production \
  -n application-secrets

# Example: API credentials
kubectl create secret generic <app-name>-api-secrets \
  --from-literal=api-key=$(openssl rand -hex 32) \
  --from-literal=api-secret=$(openssl rand -base64 64) \
  -n application-secrets

Option B: Script-based generation (recommended for production)

Create a bootstrap script for repeatable secret generation. See How-To: Create Bootstrap Script for a template.

Example script structure:

#!/bin/bash
# Generate cryptographically secure random values
DB_PASSWORD=$(openssl rand -base64 32)
API_KEY=$(openssl rand -hex 32)

# Create secrets
kubectl create secret generic myapp-db-secrets \
  --from-literal=password="$DB_PASSWORD" \
  ... \
  -n application-secrets --dry-run=client -o yaml | kubectl apply -f -

Verify source secrets

kubectl get secrets -n application-secrets | grep <app-name>

Expected output:

<app-name>-db-secrets     Opaque   4      10s
<app-name>-api-secrets    Opaque   2      10s

Step 3: Configure ESO RBAC

Create a ServiceAccount with read-only access to secrets in application-secrets.

kubectl apply -f - <<EOF
---
# ServiceAccount for ESO to read secrets
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eso-<app-name>-reader
  namespace: application-secrets
---
# Role with read-only permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: eso-secret-reader
  namespace: application-secrets
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]
---
# RoleBinding to grant ServiceAccount permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: eso-<app-name>-reader-binding
  namespace: application-secrets
subjects:
  - kind: ServiceAccount
    name: eso-<app-name>-reader
    namespace: application-secrets
roleRef:
  kind: Role
  name: eso-secret-reader
  apiGroup: rbac.authorization.k8s.io
EOF

Note

The Role grants read-only access (get, list, watch) following the principle of least privilege. ESO never needs to create, update, or delete source secrets.

Verify RBAC:

kubectl get serviceaccount eso-<app-name>-reader -n application-secrets
kubectl get role eso-secret-reader -n application-secrets
kubectl get rolebinding eso-<app-name>-reader-binding -n application-secrets

Step 4: Create ClusterSecretStore with Namespace Restriction

Create a ClusterSecretStore that uses the ServiceAccount and restricts access to your application namespace only.

kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: <app-name>-secrets-store
spec:
  # SECURITY: Restrict to specific namespace(s)
  conditions:
    - namespaces:
        - <your-app-namespace>

  provider:
    kubernetes:
      # Source namespace where secrets live
      remoteNamespace: application-secrets

      # Authentication using ServiceAccount
      auth:
        serviceAccount:
          name: eso-<app-name>-reader
          namespace: application-secrets
EOF

Important

Security Feature: The conditions.namespaces field restricts this ClusterSecretStore to ONLY the specified namespaces. Even though it’s cluster-scoped, other namespaces cannot use it.

Verify ClusterSecretStore is ready:

kubectl get clustersecretstore <app-name>-secrets-store

Expected output:

NAME                       AGE   STATUS   CAPABILITIES   READY
<app-name>-secrets-store   5s    Valid    ReadWrite      True

If READY is False, check the status:

kubectl describe clustersecretstore <app-name>-secrets-store

Step 5: Create ExternalSecrets in Application Namespace

Create your application namespace if it doesn’t exist:

kubectl create namespace <your-app-namespace>

Create ExternalSecrets to replicate secrets from application-secrets to your app namespace.

Basic Replication (Copy All Keys)

kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: <your-app-namespace>
spec:
  # Refresh every hour
  refreshInterval: 1h

  # Reference the ClusterSecretStore
  secretStoreRef:
    name: <app-name>-secrets-store
    kind: ClusterSecretStore

  # Target Kubernetes secret
  target:
    name: db-credentials
    creationPolicy: Owner

  # Fetch all keys from source secret
  dataFrom:
    - extract:
        key: <app-name>-db-secrets
EOF

Advanced: Template Secrets (Combine Multiple Sources)

Use ESO templates to combine configuration (from Git) with credentials (not in Git):

kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: database-connection
  namespace: <your-app-namespace>
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: <app-name>-secrets-store
    kind: ClusterSecretStore

  target:
    name: database-connection
    template:
      engineVersion: v2
      data:
        # Static config (safe in Git)
        host: "postgres.myapp.svc.cluster.local"
        port: "5432"
        database: "myapp_production"

        # Credentials from secret (not in Git)
        username: "{{ .username }}"
        password: "{{ .password }}"

        # Generated connection string
        connection-string: "postgresql://{{ .username }}:{{ .password }}@postgres.myapp.svc.cluster.local:5432/myapp_production"

  dataFrom:
    - extract:
        key: <app-name>-db-secrets
EOF

Step 6: Verify Secret Synchronization

Check ExternalSecret status:

kubectl get externalsecret -n <your-app-namespace>

Expected output:

NAME              STORE                      REFRESH INTERVAL   STATUS         READY
db-credentials    <app-name>-secrets-store   1h                 SecretSynced   True

Verify target secrets were created:

kubectl get secret -n <your-app-namespace>

Check secret contents match source:

# Source secret
kubectl get secret <app-name>-db-secrets -n application-secrets \
  -o jsonpath='{.data.username}' | base64 -d
echo

# Target secret
kubectl get secret db-credentials -n <your-app-namespace> \
  -o jsonpath='{.data.username}' | base64 -d
echo

Both should output the same value.

Step 7: Use Secrets in Application

Mount the synced secrets in your application pods:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: <your-app-namespace>
spec:
  template:
    spec:
      containers:
        - name: myapp
          image: myapp:latest
          env:
            # Option A: Individual environment variables
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: password

            # Option B: Connection string
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: database-connection
                  key: connection-string

Or mount as files:

          volumeMounts:
            - name: db-creds
              mountPath: /etc/secrets
              readOnly: true
      volumes:
        - name: db-creds
          secret:
            secretName: db-credentials

Secret Rotation

To rotate secrets:

1. Update source secret in application-secrets namespace:

   kubectl patch secret <app-name>-db-secrets -n application-secrets \
     --type='json' \
     -p='[{"op": "replace", "path": "/data/password", "value": "'"$(echo -n 'new-password' | base64)"'"}]'
   

2. Wait for ESO sync (based on refreshInterval, default 1h)

3. Force immediate sync (if needed):

   # Delete target secret - ESO will recreate it
   kubectl delete secret db-credentials -n <your-app-namespace>
   

4. Restart application pods (if they don’t auto-reload secrets):

   kubectl rollout restart deployment/myapp -n <your-app-namespace>
   

Troubleshooting

ExternalSecret shows “SecretSyncedError”

Check ExternalSecret status:

kubectl describe externalsecret db-credentials -n <your-app-namespace>

Common issues:

• ServiceAccount doesn’t exist: Verify Step 3

• RBAC permissions missing: Check RoleBinding

• Source secret doesn’t exist: Verify Step 2

• Wrong namespace restriction: Check ClusterSecretStore conditions.namespaces

ClusterSecretStore shows READY=False

kubectl describe clustersecretstore <app-name>-secrets-store

Common issues:

• ServiceAccount token not mounted

• Network connectivity issues

• RBAC permissions not granted

Secret not updating after source change

Check refresh interval:

kubectl get externalsecret db-credentials -n <your-app-namespace> \
  -o jsonpath='{.spec.refreshInterval}'

Force immediate update by deleting ExternalSecret (ESO will recreate it):

kubectl delete externalsecret db-credentials -n <your-app-namespace>
# Re-apply the ExternalSecret manifest

Namespace restriction not working

Verify ClusterSecretStore conditions:

kubectl get clustersecretstore <app-name>-secrets-store -o yaml | grep -A 5 conditions

Should show:

conditions:
  - namespaces:
      - <your-app-namespace>

Security Best Practices

1. Never commit secrets to Git: Always store in application-secrets namespace

2. Use namespace restrictions: Always set conditions.namespaces in ClusterSecretStore

3. Least privilege ServiceAccounts: Only grant get, list, watch permissions

4. Save bootstrap credentials securely: Use password manager, delete temp files

5. Regular rotation: Update source secrets periodically

6. Audit logs: Monitor ExternalSecret events for unauthorized access

Example Implementation

GitLab BDA uses this exact pattern to manage 15+ secrets across 3 ExternalSecrets:

• Application secrets (7 keys): root password, Redis, secret keys

• SMTP credentials (2 keys): username, password

• Harbor registry (4 keys): admin password, OAuth secrets

See Reference: GitLab BDA Secrets for the complete working example.

Next Steps

• Create a bootstrap script for repeatable secret generation

• Configure External Secrets for advanced ESO patterns

• Security Model to understand overall cluster security

• Application Secrets Architecture for design rationale

Related Documentation

• Tutorial: Cross-Namespace Secrets - Step-by-step learning

• External Secrets Operator Docs - Official ESO documentation

• Kubernetes RBAC - RBAC concepts