Tutorial

Sharing Secrets Across Namespaces with ClusterSecretStore

Type: Tutorial (Learning-oriented)

Time: ~20 minutes | Level: Intermediate

Prerequisites: ESO installed, kubectl access, basic Kubernetes knowledge

Related: Application Secrets Architecture

What You’ll Learn

By the end of this tutorial, you’ll understand:

• The difference between SecretStore and ClusterSecretStore

• How to configure RBAC for cross-namespace secret access

• How to use ServiceAccounts with External Secrets Operator

• Security best practices for shared secrets

Scenario

You have a shared PostgreSQL database that multiple applications need to access. Instead of duplicating credentials in each namespace, you’ll store them once in a central location and sync them to application namespaces using ClusterSecretStore.

Architecture:

shared-secrets (namespace)
    └── postgres-credentials (Secret)
            ↓ (ClusterSecretStore syncs)
    ├── app1 (namespace) → postgres-credentials (synced Secret)
    └── app2 (namespace) → postgres-credentials (synced Secret)

Step 1: Create the Source Namespace and Credentials

First, create a namespace to hold your shared secrets:

kubectl create namespace shared-secrets

Create the source secret with PostgreSQL credentials:

kubectl create secret generic postgres-credentials \
  --from-literal=username=postgres \
  --from-literal=password=super-secret-password \
  --from-literal=host=postgres.database.svc.cluster.local \
  --from-literal=database=myapp \
  -n shared-secrets

Verify the secret was created:

kubectl get secret postgres-credentials -n shared-secrets

Expected output:

NAME                   TYPE     DATA   AGE
postgres-credentials   Opaque   4      5s

Step 2: Create a ServiceAccount for ESO

External Secrets Operator needs a ServiceAccount with permissions to read secrets from the source namespace.

Create the ServiceAccount:

# serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eso-shared-secrets-reader
  namespace: shared-secrets

Apply it:

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eso-shared-secrets-reader
  namespace: shared-secrets
EOF

Step 3: Configure RBAC Permissions

Create a Role that allows reading secrets in the shared-secrets namespace:

# role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: eso-secret-reader
  namespace: shared-secrets
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]

Create a RoleBinding to grant the ServiceAccount these permissions:

# rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: eso-secret-reader-binding
  namespace: shared-secrets
subjects:
  - kind: ServiceAccount
    name: eso-shared-secrets-reader
    namespace: shared-secrets
roleRef:
  kind: Role
  name: eso-secret-reader
  apiGroup: rbac.authorization.k8s.io

Apply both:

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: eso-secret-reader
  namespace: shared-secrets
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: eso-secret-reader-binding
  namespace: shared-secrets
subjects:
  - kind: ServiceAccount
    name: eso-shared-secrets-reader
    namespace: shared-secrets
roleRef:
  kind: Role
  name: eso-secret-reader
  apiGroup: rbac.authorization.k8s.io
EOF

Step 4: Create the ClusterSecretStore

Now create a ClusterSecretStore that uses the ServiceAccount to access secrets:

# cluster-secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: shared-k8s-secrets
spec:
  provider:
    kubernetes:
      # Source namespace where secrets live
      remoteNamespace: shared-secrets

      # Use in-cluster server (same cluster)
      server:
        caProvider:
          type: ConfigMap
          name: kube-root-ca.crt
          key: ca.crt
          namespace: kube-system

      # Authentication using ServiceAccount
      auth:
        serviceAccount:
          name: eso-shared-secrets-reader
          namespace: shared-secrets

Apply it:

kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: shared-k8s-secrets
spec:
  provider:
    kubernetes:
      remoteNamespace: shared-secrets
      server:
        caProvider:
          type: ConfigMap
          name: kube-root-ca.crt
          key: ca.crt
          namespace: kube-system
      auth:
        serviceAccount:
          name: eso-shared-secrets-reader
          namespace: shared-secrets
EOF

Verify the ClusterSecretStore is ready:

kubectl get clustersecretstore shared-k8s-secrets

Expected output:

NAME                  AGE   STATUS   CAPABILITIES   READY
shared-k8s-secrets    10s   Valid    ReadWrite      True

Important: If READY is False, check the status:

kubectl describe clustersecretstore shared-k8s-secrets

Step 5: Create Consumer Namespaces

Create the application namespaces that will consume the shared secrets:

kubectl create namespace app1
kubectl create namespace app2

Step 6: Create ExternalSecrets in Consumer Namespaces

Create an ExternalSecret in the first application namespace:

# app1-external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: postgres-credentials
  namespace: app1
spec:
  # Refresh every hour
  refreshInterval: 1h

  # Reference the ClusterSecretStore
  secretStoreRef:
    name: shared-k8s-secrets
    kind: ClusterSecretStore

  # Target Kubernetes secret to create
  target:
    name: postgres-credentials
    creationPolicy: Owner

  # Fetch all keys from the source secret
  dataFrom:
    - extract:
        key: postgres-credentials

Apply it:

kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: postgres-credentials
  namespace: app1
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: shared-k8s-secrets
    kind: ClusterSecretStore
  target:
    name: postgres-credentials
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: postgres-credentials
EOF

Create the same ExternalSecret for the second application:

kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: postgres-credentials
  namespace: app2
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: shared-k8s-secrets
    kind: ClusterSecretStore
  target:
    name: postgres-credentials
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: postgres-credentials
EOF

Step 7: Verify Secret Synchronization

Check that ExternalSecrets are syncing:

kubectl get externalsecret -n app1
kubectl get externalsecret -n app2

Expected output for each:

NAME                   STORE                REFRESH INTERVAL   STATUS         READY
postgres-credentials   shared-k8s-secrets   1h                 SecretSynced   True

Verify the secrets were created in consumer namespaces:

kubectl get secret postgres-credentials -n app1
kubectl get secret postgres-credentials -n app2

Check the secret contents match the source:

# Compare source secret
kubectl get secret postgres-credentials -n shared-secrets -o jsonpath='{.data.username}' | base64 -d
echo

# Compare app1 secret
kubectl get secret postgres-credentials -n app1 -o jsonpath='{.data.username}' | base64 -d
echo

# Compare app2 secret
kubectl get secret postgres-credentials -n app2 -o jsonpath='{.data.username}' | base64 -d
echo

All should output: postgres

Step 8: Test Secret Updates Propagate

Update the source secret to see synchronization in action:

kubectl patch secret postgres-credentials -n shared-secrets \
  --type='json' \
  -p='[{"op": "replace", "path": "/data/password", "value": "'"$(echo -n 'new-super-secret-password' | base64)"'"}]'

Wait for the refresh interval (or delete and recreate ExternalSecrets for immediate sync):

# Force immediate sync by deleting ExternalSecrets (secrets will be recreated)
kubectl delete externalsecret postgres-credentials -n app1
kubectl delete externalsecret postgres-credentials -n app2

# Recreate them
kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: postgres-credentials
  namespace: app1
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: shared-k8s-secrets
    kind: ClusterSecretStore
  target:
    name: postgres-credentials
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: postgres-credentials
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: postgres-credentials
  namespace: app2
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: shared-k8s-secrets
    kind: ClusterSecretStore
  target:
    name: postgres-credentials
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: postgres-credentials
EOF

Verify the updated password propagated:

kubectl get secret postgres-credentials -n app1 -o jsonpath='{.data.password}' | base64 -d
echo
# Should output: new-super-secret-password

Security Best Practices

1. Principle of Least Privilege

The ServiceAccount only has get, list, watch permissions on secrets in the source namespace - no create, update, or delete permissions.

Good:

rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]  # Read-only

Bad (too permissive):

rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["*"]  # Full access - AVOID!

2. When to Use ClusterSecretStore vs SecretStore

Use ClusterSecretStore when:

• Multiple namespaces need the same secrets (shared database, API keys)

• Central secrets management team controls access

• Reduces secret duplication

Use SecretStore (namespaced) when:

• Secrets are specific to one application

• Namespace-level isolation is required

• Different teams manage different namespaces

3. ServiceAccount Token Security

• ServiceAccount tokens are automatically mounted in ESO pods

• Tokens are scoped to the specific ServiceAccount’s permissions

• Rotate ServiceAccounts if compromised (delete and recreate)

4. Audit and Monitoring

Monitor ExternalSecret events for unauthorized access:

kubectl get events -n app1 --field-selector involvedObject.kind=ExternalSecret

Enable audit logging for ESO operations in kube-apiserver if compliance is required.

5. Secret Rotation Strategy

• Update source secret in shared-secrets namespace

• ESO automatically syncs to all consumer namespaces (based on refreshInterval)

• Applications should reload secrets on change (use reloader or similar tools)

Troubleshooting

ExternalSecret shows “SecretSyncedError”

Check ClusterSecretStore status:

kubectl describe clustersecretstore shared-k8s-secrets

Common issues:

• ServiceAccount doesn’t exist

• RBAC permissions not granted

• Source secret doesn’t exist in remoteNamespace

RBAC Permission Denied

Error: secrets "postgres-credentials" is forbidden: User "system:serviceaccount:shared-secrets:eso-shared-secrets-reader" cannot get resource "secrets"

Solution: Verify RoleBinding is correct:

kubectl get rolebinding eso-secret-reader-binding -n shared-secrets -o yaml

Ensure the subjects reference the correct ServiceAccount and namespace.

Secret Not Updating After Source Change

Check refresh interval:

kubectl get externalsecret postgres-credentials -n app1 -o jsonpath='{.spec.refreshInterval}'

To force immediate update, delete and recreate the ExternalSecret.

Cleanup

Remove all resources created in this tutorial:

# Delete ExternalSecrets
kubectl delete externalsecret postgres-credentials -n app1
kubectl delete externalsecret postgres-credentials -n app2

# Delete ClusterSecretStore
kubectl delete clustersecretstore shared-k8s-secrets

# Delete RBAC
kubectl delete rolebinding eso-secret-reader-binding -n shared-secrets
kubectl delete role eso-secret-reader -n shared-secrets
kubectl delete serviceaccount eso-shared-secrets-reader -n shared-secrets

# Delete namespaces (optional)
kubectl delete namespace app1
kubectl delete namespace app2
kubectl delete namespace shared-secrets

What You Learned

In this tutorial, you:

• ✅ Created a ClusterSecretStore for cross-namespace secret access

• ✅ Configured a ServiceAccount with minimal RBAC permissions

• ✅ Synchronized secrets from a central namespace to multiple consumer namespaces

• ✅ Verified automatic secret updates

• ✅ Learned security best practices for shared secrets

Next Steps

• Integrate with External Backends: See Configure External Secrets to use HashiCorp Vault or cloud secret managers instead of Kubernetes secrets

• Secret Templating: Learn how to combine multiple secrets into connection strings

• PushSecrets: Explore pushing secrets FROM Kubernetes TO external backends

• Monitoring: Set up alerts for ExternalSecret sync failures

Further Reading

• External Secrets Operator Documentation

• Kubernetes RBAC Documentation

• How-To: Configure External Secrets