Explanation

Application Secrets Architecture

Type: Explanation (Understanding-oriented)

Related Concepts: Security Model | Bootstrap Application Secrets

Understanding the design philosophy, security model, and architectural decisions behind the application-secrets namespace pattern for secret management in the KUP6S cluster.

The Problem

Secrets in Git (Anti-Pattern)

Traditional application deployments often suffer from insecure secret management:

# ❌ BAD: Hardcoded in Git
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
stringData:
  password: "CHANGE_ME"  # Placeholder never changed
  api-key: "hardcoded-value"  # Committed to Git

Problems with this approach:

  • Secrets visible in version control history

  • Placeholder values (CHANGE_ME) forgotten in production

  • No secret rotation mechanism

  • Audit trail exposes credentials

  • Multiple copies across environments

Manual Secret Management

Another common approach is manual secret creation:

# ❌ TEDIOUS: Manual kubectl commands
kubectl create secret generic db-creds \
  --from-literal=password="paste-from-password-manager" \
  -n app1

# Repeat for every namespace...
kubectl create secret generic db-creds \
  --from-literal=password="paste-again" \
  -n app2

Problems with manual management:

  • Doesn’t scale (many apps × many namespaces)

  • No automation or GitOps integration

  • Secret duplication across namespaces

  • Rotation requires manual updates everywhere

  • No single source of truth

The Need for a Better Solution

What we need:

  • ✅ No secrets in Git - Version control only stores configuration, not credentials

  • ✅ Single source of truth - One place to manage each secret

  • ✅ Automatic replication - Secrets sync to multiple namespaces

  • ✅ GitOps-friendly - Declarative ExternalSecret resources in Git

  • ✅ Rotation support - Update once, propagate everywhere

  • ✅ Namespace isolation - RBAC prevents unauthorized access

  • ✅ Audit trail - Kubernetes events track secret access

The Solution: Application-Secrets Namespace Pattern

The application-secrets pattern uses External Secrets Operator (ESO) with a dedicated namespace architecture to solve these problems.

Architecture Overview

┌─────────────────────────────────────────────────────────┐
│ SOURCE LAYER: application-secrets namespace             │
│                                                           │
│  [Source Secrets]                                        │
│  ├─ myapp-db-secrets (bootstrap once, update for rot.)  │
│  ├─ myapp-api-secrets                                    │
│  └─ myapp-smtp-secrets                                   │
│                                                           │
│  [ESO Infrastructure]                                    │
│  ├─ ServiceAccount (eso-myapp-reader)                   │
│  ├─ Role (read-only secrets)                            │
│  └─ RoleBinding                                          │
└─────────────────────────────────────────────────────────┘
                          ↓
         [ClusterSecretStore with namespace restrictions]
                          ↓
┌─────────────────────────────────────────────────────────┐
│ REPLICATION LAYER: ClusterSecretStore                   │
│                                                           │
│  apiVersion: external-secrets.io/v1                     │
│  kind: ClusterSecretStore                               │
│  spec:                                                   │
│    conditions:                                           │
│      namespaces: [myapp]  ← SECURITY: Namespace scope  │
│    provider:                                             │
│      kubernetes:                                         │
│        remoteNamespace: application-secrets             │
│        auth:                                             │
│          serviceAccount: eso-myapp-reader               │
└─────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────┐
│ TARGET LAYER: Application namespaces (myapp)            │
│                                                           │
│  [ExternalSecrets] (declarative, stored in Git)         │
│  ├─ db-credentials-es → creates → db-credentials        │
│  ├─ api-credentials-es → creates → api-credentials      │
│  └─ smtp-credentials-es → creates → smtp-credentials    │
│                                                           │
│  [Synced Secrets] (auto-managed by ESO)                 │
│  ├─ db-credentials (used by application)                │
│  ├─ api-credentials                                      │
│  └─ smtp-credentials                                     │
│                                                           │
│  [Application Pods]                                      │
│  └─ Read secrets as env vars or mounted files           │
└─────────────────────────────────────────────────────────┘

What Goes Where

In Git (safe to commit):

  • Infrastructure manifests creating application-secrets namespace

  • ServiceAccount, Role, RoleBinding resources

  • ClusterSecretStore definitions

  • ExternalSecret resources (declarative replication config)

  • Application deployment manifests

NOT in Git (bootstrap manually):

  • Source secrets in application-secrets namespace

  • Actual credential values

  • Bootstrap scripts (generate random values)

Design Decisions

Decision 1: Dedicated Namespace for Source Secrets

Why not store source secrets in application namespaces?

We use a separate application-secrets namespace instead of co-locating secrets with applications.

Rationale:

  • Separation of concerns: Secret management vs. application runtime

  • RBAC isolation: Different teams manage secrets vs. deploy apps

  • Single source of truth: One namespace to audit for source secrets

  • Prevents accidental deletion: Application namespace deletion doesn’t lose secrets

  • Centralized security: Apply strict RBAC policies to one namespace

Alternative considered: Store source secrets in each application namespace

  • ❌ Rejected: Duplication, no centralization, harder to audit

Decision 2: ClusterSecretStore vs. SecretStore

Why ClusterSecretStore instead of namespace-scoped SecretStore?

We use ClusterSecretStore (cluster-scoped) with namespace restrictions.

Rationale:

  • Reusability: One ClusterSecretStore can serve multiple applications

  • Centralized config: Single place to update authentication

  • Namespace isolation via conditions: conditions.namespaces provides security

  • Operational simplicity: Fewer resources to manage

Key security feature:

spec:
  conditions:
    namespaces:
      - myapp  # Only myapp namespace can use this store

Even though it’s cluster-scoped, other namespaces cannot access it.

Alternative considered: SecretStore in each application namespace

  • ❌ Rejected: Duplication, harder to maintain, more RBAC complexity

Decision 3: ESO Templates for Configuration + Credentials

Why use ESO templates to combine static config with dynamic credentials?

Some secrets need both:

  • Configuration (safe to store in Git): hostnames, ports, domains

  • Credentials (NOT in Git): usernames, passwords, tokens

Example: SMTP configuration

# ESO Template combines both
target:
  template:
    engineVersion: v2
    data:
      # Static config (from Git)
      host: "smtp.example.com"
      port: "587"

      # Credentials (from source secret)
      username: "{{ .username }}"
      password: "{{ .password }}"

Rationale:

  • Separation: Config changes tracked in Git, credentials stay secret

  • Single secret: Application reads one secret, not multiple

  • Connection strings: Generate DATABASE_URL from parts

  • Flexibility: Mix and match as needed

Alternative considered: Store all config in source secrets

  • ❌ Rejected: No Git tracking, harder to review config changes

Decision 4: Read-Only ServiceAccount

Why only grant get, list, watch permissions to ESO ServiceAccount?

We follow the principle of least privilege:

rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]  # Read-only

Rationale:

  • Security: ESO never needs to create/update/delete source secrets

  • Least privilege: Minimize impact if ServiceAccount is compromised

  • Audit trail: Clear separation - humans bootstrap, ESO only reads

  • Immutability: Source secrets changed deliberately, not by automation

Alternative considered: Grant full access (verbs: ["*"])

  • ❌ Rejected: Violates least privilege, unnecessary permissions

Security Model

Threat Model and Mitigations

Threat

Mitigation

Secrets in Git history

Source secrets never committed, only ExternalSecret configs

Unauthorized namespace access

ClusterSecretStore conditions.namespaces restricts access

Lateral movement

RBAC prevents one app’s ServiceAccount reading another app’s secrets

Secret exfiltration

Kubernetes audit logs track secret access attempts

Compromised ServiceAccount

Read-only permissions limit damage, no create/update/delete

Source secret deletion

Infrastructure namespace protected, application deletion doesn’t affect sources

Security Properties

  1. Namespace Isolation

    • Each application gets a dedicated ClusterSecretStore

    • ClusterSecretStore restricted to specific namespace(s)

    • ServiceAccount in application-secrets can only read, not write

  2. No Secrets in Git

    • ExternalSecret resources (declarative config) in Git

    • Source secrets (actual values) created via bootstrap script

    • Git history never contains credentials

  3. Audit Trail

    • Kubernetes events track ExternalSecret sync operations

    • RBAC denials logged in kube-apiserver audit logs

    • Source secret changes tracked in application-secrets namespace

  4. Least Privilege

    • ESO ServiceAccount: read-only access to source secrets

    • Application pods: read access to synced secrets only

    • No pod can access application-secrets directly

Secret Lifecycle

1. Bootstrap (One-Time Setup)

# Run bootstrap script to generate random values
./scripts/bootstrap-myapp-secrets.sh

# Creates source secrets in application-secrets namespace
# Saves credentials to secure location (password manager)

2. GitOps Declaration (Stored in Git)

# ExternalSecret resource in Git
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: myapp
spec:
  secretStoreRef:
    name: myapp-secrets-store
    kind: ClusterSecretStore
  target:
    name: db-credentials
  dataFrom:
    - extract:
        key: myapp-db-secrets

3. Automatic Synchronization (ESO)

  • ESO reads ExternalSecret resource

  • Uses ClusterSecretStore to authenticate

  • Fetches source secret from application-secrets

  • Creates/updates target secret in application namespace

  • Refreshes based on refreshInterval (default: 1h)

4. Application Consumption

# Application pod reads synced secret
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: app
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: password

5. Rotation (Update and Propagate)

# Update source secret in application-secrets
kubectl patch secret myapp-db-secrets -n application-secrets \
  --type='json' \
  -p='[{"op": "replace", "path": "/data/password", "value": "<new-base64-value>"}]'

# ESO automatically syncs to all target namespaces
# within refreshInterval (or force with kubectl delete)

# Restart application pods to pick up new secret
kubectl rollout restart deployment/myapp -n myapp

Comparison with Alternatives

vs. Direct Kubernetes Secrets

Direct Secrets:

# Committed to Git
apiVersion: v1
kind: Secret
stringData:
  password: "hardcoded"

Problems:

  • ❌ Secrets in Git history

  • ❌ No rotation mechanism

  • ❌ Manual management

ESO Pattern:

  • ✅ No secrets in Git (only ExternalSecret config)

  • ✅ Automatic rotation via source secret updates

  • ✅ GitOps-friendly declarative resources

vs. Sealed Secrets

Sealed Secrets:

  • Encrypts secrets with public key

  • Stores encrypted secrets in Git

  • Controller decrypts at runtime

Problems:

  • ❌ Key management complexity

  • ❌ Encrypted secrets still in Git (harder to audit)

  • ❌ Rotation requires re-encryption

ESO Pattern:

  • ✅ No secrets in Git (encrypted or not)

  • ✅ Simpler key management (ServiceAccount tokens)

  • ✅ Rotation is simple source secret update

vs. HashiCorp Vault

Vault:

  • Dedicated secret management system

  • External dependency

  • Rich policy engine

Trade-offs:

  • ✅ Enterprise-grade secret management

  • ❌ Operational complexity (Vault cluster, backups, HA)

  • ❌ Additional cost and maintenance

  • ❌ External dependency (cluster can’t bootstrap without Vault)

ESO Pattern:

  • ✅ No external dependencies

  • ✅ Simpler operations (Kubernetes-native)

  • ✅ Lower cost (no additional infrastructure)

  • ❌ Less sophisticated than Vault

When to use Vault: Large organizations, compliance requirements, multi-cluster secret sharing

When to use ESO pattern: Small/medium clusters, Kubernetes-native preferred, simplicity valued

When to Use This Pattern

Use Application-Secrets Pattern When:

✅ Application has multiple secrets (>3)

  • Database credentials, API keys, SMTP config, etc.

  • Pattern scales well with secret count

✅ Need centralized secret rotation

  • Update source secret once, propagate everywhere

  • Multiple environments need same credentials

✅ Want to avoid Git commits of secrets

  • Git history stays clean

  • No encrypted blobs or placeholders

✅ Require namespace isolation

  • Multi-tenant cluster

  • Different teams manage different apps

✅ Multiple environments (dev/staging/prod)

  • Same pattern, different source secrets

  • Clear separation of concerns

Don’t Use This Pattern When:

❌ Single-application cluster

  • Overhead not justified

  • Direct secrets or Sealed Secrets simpler

❌ Secrets already in external system

  • Already have Vault or AWS Secrets Manager

  • Use ESO with that provider instead

❌ Very simple application

  • 1-2 secrets total

  • Direct secret management acceptable

❌ Need external secret sharing

  • Multi-cluster deployments

  • Consider centralized Vault instead

Real-World Examples

GitLab BDA (Production)

The GitLab BDA deployment on KUP6S uses this pattern to manage 15+ secrets across 3 ExternalSecrets:

Source Secrets (application-secrets namespace):

  • gitlabbda-app-secrets: 7 keys (root password, Redis, secret keys, tokens)

  • gitlabbda-smtp-secrets: 2 keys (username, password)

  • gitlabbda-harbor-secrets: 4 keys (admin password, OAuth client ID/secret)

Target Secrets (gitlabbda namespace):

  • gitlab-secrets: Replicated from app-secrets

  • gitlab-smtp-credentials: ESO template combines config (host, port) + credentials

  • harbor-secrets: Replicated from harbor-secrets

Special Cases:

  • CNPG-managed database password: gitlab-postgres-app secret managed by CNPG operator, not ESO

  • S3 credentials: Managed via separate crossplane-system namespace pattern

ClusterSecretStore:

  • Name: gitlabbda-app-secrets-store

  • Namespace restriction: conditions.namespaces: [gitlabbda]

  • Authentication: ServiceAccount eso-app-secrets-reader in application-secrets

Outcomes:

  • ✅ Zero secrets committed to Git

  • ✅ Centralized rotation (update 3 source secrets, 15+ keys sync)

  • ✅ Secure namespace isolation

  • ✅ GitOps-friendly (ExternalSecret manifests in Git)

See Reference: GitLab BDA Secrets for complete implementation details.

Future Applications

This pattern is designed for reuse:

  • Nextcloud: User credentials, admin password, database, S3 storage

  • GitLab Runner: Registration tokens, cache credentials

  • Monitoring Stack: Alert manager credentials, webhook URLs

  • CI/CD Tools: Jenkins, Argo Workflows authentication

Summary

The application-secrets namespace pattern provides:

  • ✅ Security: No secrets in Git, namespace isolation, least privilege RBAC

  • ✅ Automation: ESO handles replication, rotation propagates automatically

  • ✅ GitOps: Declarative ExternalSecret resources in version control

  • ✅ Simplicity: Kubernetes-native, no external dependencies

  • ✅ Scalability: Supports multiple applications and environments

It’s a pragmatic middle ground between manual secret management and enterprise solutions like Vault, well-suited for small to medium Kubernetes clusters that value operational simplicity without sacrificing security.

Next Steps