Reference

GitLab BDA Secrets Configuration

Type: Reference (Information-oriented)

See Also: Application Secrets Architecture | Bootstrap Application Secrets

Technical reference for GitLab BDA’s secret management using the application-secrets namespace pattern with External Secrets Operator (ESO).

Overview

GitLab BDA (Blue Dynamics Austria) is a complete GitLab deployment with integrated Harbor registry, managing 15+ secrets across 3 source secrets using the application-secrets pattern.

Deployment namespace: gitlabbda Source namespace: application-secrets Pattern: ClusterSecretStore with namespace restrictions

Architecture Diagram

application-secrets namespace
├─ gitlabbda-app-secrets-source (7 keys)
├─ gitlabbda-smtp-secrets-source (2 keys)
└─ gitlabbda-harbor-secrets-source (4 keys)
            ↓
    ClusterSecretStore: gitlabbda-app-secrets-store
    (restricted to gitlabbda namespace only)
            ↓
gitlabbda namespace
├─ gitlab-secrets (ESO-managed, 7 keys)
├─ gitlab-smtp-credentials (ESO template, 7 keys)
├─ harbor-secrets (ESO-managed, 4 keys)
└─ gitlab-postgres-app (CNPG-managed, 11 keys)

Source Secrets

1. gitlabbda-app-secrets-source

Location: application-secrets namespace Purpose: GitLab application credentials

Keys (7):

Key	Purpose	Format	Example
initial-root-password	GitLab root user initial password	base64, 32 bytes	Login to GitLab UI
redis-password	Redis cache authentication	base64, 32 bytes	GitLab ↔ Redis
secret-key-base	Rails secret key	base64, 64 bytes	Session encryption
otp-key-base	One-time password secret	base64, 64 bytes	2FA tokens
db-key-base	Database encryption key	base64, 64 bytes	At-rest encryption
shell-secret	GitLab Shell shared secret	hex, 32 bytes	SSH operations
runner-registration-token	GitLab Runner auto-registration	hex, 32 bytes	CI/CD runners

Creation:

kubectl get secret gitlabbda-app-secrets-source -n application-secrets

2. gitlabbda-smtp-secrets-source

Location: application-secrets namespace Purpose: SMTP server authentication for email notifications

Keys (2):

Key	Purpose	Source
username	SMTP username	From environment or user input
password	SMTP password	From environment or user input

Note: These are user-provided credentials, not generated.

3. gitlabbda-harbor-secrets-source

Location: application-secrets namespace Purpose: Harbor container registry credentials

Keys (4):

Key	Purpose	Format
admin-password	Harbor admin user password	base64, 32 bytes
gitlab-oauth-client-id	GitLab OAuth app client ID	hex, 16 bytes
gitlab-oauth-client-secret	GitLab OAuth app secret	base64, 32 bytes
registry-secret	Harbor internal registry secret	base64, 32 bytes

Bootstrap Script

Location: kube-hetzner/scripts/bootstrap-gitlabbda-secrets.sh

What it does:

1. Generates 13 cryptographically secure random values

2. Prompts for SMTP credentials (from $TF_VAR_smtp_username and $TF_VAR_smtp_password or user input)

3. Creates or updates 3 source secrets in application-secrets namespace

4. Saves all credentials to ~/.kup6s/gitlabbda-credentials-TIMESTAMP.txt

5. Displays verification commands

Usage:

# Option 1: Use environment variables for SMTP
cd kube-hetzner
source .env
bash scripts/bootstrap-gitlabbda-secrets.sh

# Option 2: Interactive prompts
cd kube-hetzner
bash scripts/bootstrap-gitlabbda-secrets.sh
# Script will prompt for SMTP username and password

Output:

🔐 Bootstrapping GitLab BDA application secrets...
🎲 Generating random secrets...
📧 SMTP Configuration
📦 Creating source secrets in application-secrets namespace...
secret/gitlabbda-app-secrets-source configured
secret/gitlabbda-smtp-secrets-source configured
secret/gitlabbda-harbor-secrets-source configured
✅ Bootstrap complete!
🔐 Credentials saved to: /home/user/.kup6s/gitlabbda-credentials-20251027-123456.txt

Security note: Delete the credentials file after saving to password manager:

rm ~/.kup6s/gitlabbda-credentials-*.txt

ESO Infrastructure

ServiceAccount

Name: eso-app-secrets-reader Namespace: application-secrets

Purpose: ESO uses this ServiceAccount to read secrets from application-secrets namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: eso-app-secrets-reader
  namespace: application-secrets

RBAC

Role: eso-secret-reader (in application-secrets)

Permissions: Read-only access to secrets

rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]

RoleBinding: eso-secret-reader-binding

• Binds eso-secret-reader Role to eso-app-secrets-reader ServiceAccount

ClusterSecretStore

Name: gitlabbda-app-secrets-store Type: ClusterSecretStore (cluster-scoped)

Namespace restriction:

spec:
  conditions:
    namespaces:
      - gitlabbda  # SECURITY: Only gitlabbda can use this store

Authentication:

spec:
  provider:
    kubernetes:
      remoteNamespace: application-secrets
      auth:
        serviceAccount:
          name: eso-app-secrets-reader
          namespace: application-secrets

Verification:

kubectl get clustersecretstore gitlabbda-app-secrets-store

Expected output:

NAME                          AGE   STATUS   CAPABILITIES   READY
gitlabbda-app-secrets-store   1d    Valid    ReadWrite      True

Target Secrets (gitlabbda namespace)

1. gitlab-secrets

Purpose: GitLab application secrets Managed by: ExternalSecret gitlab-app-secrets-es Refresh interval: 1h

Keys (7):

• initial-root-password

• redis-password

• secret-key-base

• otp-key-base

• db-key-base

• shell-secret

• runner-registration-token

Note: db-password field is intentionally excluded. GitLab reads database credentials directly from gitlab-postgres-app secret (CNPG-managed).

ExternalSecret manifest:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: gitlab-app-secrets-es
  namespace: gitlabbda
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gitlabbda-app-secrets-store
    kind: ClusterSecretStore
  target:
    name: gitlab-secrets
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: gitlabbda-app-secrets-source

2. gitlab-smtp-credentials

Purpose: SMTP server configuration and credentials Managed by: ExternalSecret smtp-secrets-es Refresh interval: 1h

Keys (7):

• host - SMTP server hostname (from Helm chart config)

• port - SMTP server port (from Helm chart config)

• domain - Email domain (from Helm chart config)

• from - From email address (from Helm chart config)

• reply-to - Reply-to email address (from Helm chart config)

• username - SMTP authentication username (from source secret)

• password - SMTP authentication password (from source secret)

ESO Template: Uses template engine v2 to combine static config with dynamic credentials.

ExternalSecret manifest:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: smtp-secrets-es
  namespace: gitlabbda
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gitlabbda-app-secrets-store
    kind: ClusterSecretStore
  target:
    name: gitlab-smtp-credentials
    template:
      engineVersion: v2
      data:
        # Static config (safe in Git)
        host: "smtp.example.com"
        port: "587"
        domain: "example.com"
        from: "gitlab@example.com"
        reply-to: "noreply@example.com"

        # Credentials (from source secret, NOT in Git)
        username: "{{ .username }}"
        password: "{{ .password }}"
  dataFrom:
    - extract:
        key: gitlabbda-smtp-secrets-source

3. harbor-secrets

Purpose: Harbor container registry credentials Managed by: ExternalSecret harbor-secrets-es Refresh interval: 1h

Keys (4):

• admin-password

• gitlab-oauth-client-id

• gitlab-oauth-client-secret

• registry-secret

ExternalSecret manifest:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: harbor-secrets-es
  namespace: gitlabbda
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gitlabbda-app-secrets-store
    kind: ClusterSecretStore
  target:
    name: harbor-secrets
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: gitlabbda-harbor-secrets-source

Special Cases

Database Password (NOT managed by ESO)

Secret: gitlab-postgres-app Namespace: gitlabbda Managed by: CloudNativePG (CNPG) operator

Why not ESO: CNPG automatically generates and manages the PostgreSQL cluster password. All GitLab components (Helm chart, PgBouncer) correctly read from this secret.

Keys include:

• username - Database username (gitlab)

• password - Database password (CNPG-generated)

• dbname - Database name

• host - Database host

• port - Database port

• And more…

GitLab Helm configuration:

global:
  psql:
    password:
      secret: gitlab-postgres-app
      key: password

No synchronization needed - CNPG and GitLab Helm work together correctly.

S3 Credentials (Separate Pattern)

Secret: gitlab-s3-credentials, harbor-s3-credentials Namespace: gitlabbda Managed by: Different ClusterSecretStore (hetzner-s3-cluster-store)

Source namespace: crossplane-system (not application-secrets)

Rationale: S3 credentials are infrastructure-level, shared across multiple applications. Managed via separate pattern for reusability.

Verification Commands

Check Source Secrets

# List source secrets
kubectl get secrets -n application-secrets | grep gitlabbda

# Check key count
kubectl get secret gitlabbda-app-secrets-source -n application-secrets \
  -o jsonpath='{.data}' | jq 'keys | length'
# Expected: 7

kubectl get secret gitlabbda-smtp-secrets-source -n application-secrets \
  -o jsonpath='{.data}' | jq 'keys | length'
# Expected: 2

kubectl get secret gitlabbda-harbor-secrets-source -n application-secrets \
  -o jsonpath='{.data}' | jq 'keys | length'
# Expected: 4

Check ExternalSecrets

# Check ExternalSecret status
kubectl get externalsecret -n gitlabbda

# Expected output:
# NAME                       STORE                         REFRESH INTERVAL   STATUS         READY
# gitlab-app-secrets-es      gitlabbda-app-secrets-store   1h                 SecretSynced   True
# harbor-secrets-es          gitlabbda-app-secrets-store   1h                 SecretSynced   True
# smtp-secrets-es            gitlabbda-app-secrets-store   1h                 SecretSynced   True

Check Target Secrets

# List target secrets
kubectl get secrets -n gitlabbda | grep -E "gitlab-secrets|smtp|harbor"

# Verify key counts
kubectl get secret gitlab-secrets -n gitlabbda -o jsonpath='{.data}' | jq 'keys | length'
# Expected: 7

kubectl get secret gitlab-smtp-credentials -n gitlabbda -o jsonpath='{.data}' | jq 'keys | length'
# Expected: 7

kubectl get secret harbor-secrets -n gitlabbda -o jsonpath='{.data}' | jq 'keys | length'
# Expected: 4

Verify Secret Values Match

# Compare source and target (should be identical)
SOURCE=$(kubectl get secret gitlabbda-app-secrets-source -n application-secrets \
  -o jsonpath='{.data.redis-password}')

TARGET=$(kubectl get secret gitlab-secrets -n gitlabbda \
  -o jsonpath='{.data.redis-password}')

if [ "$SOURCE" == "$TARGET" ]; then
  echo "✅ Values match"
else
  echo "❌ Values differ - ESO sync issue"
fi

Secret Rotation

Rotating Application Secrets

1. Update source secret:

   # Generate new password
   NEW_PASSWORD=$(openssl rand -base64 32)
   
   # Update source secret
   kubectl patch secret gitlabbda-app-secrets-source -n application-secrets \
     --type='json' \
     -p='[{"op": "replace", "path": "/data/redis-password", "value": "'"$(echo -n "$NEW_PASSWORD" | base64)"'"}]'
   

2. Force immediate ESO sync (or wait for refresh interval):

   # Delete target secret - ESO will recreate it
   kubectl delete secret gitlab-secrets -n gitlabbda
   

3. Verify sync:

   kubectl get externalsecret gitlab-app-secrets-es -n gitlabbda
   # Should show: STATUS=SecretSynced, READY=True
   

4. Restart affected pods:

   # Restart GitLab components
   kubectl rollout restart deployment/gitlab-webservice-default -n gitlabbda
   kubectl rollout restart deployment/gitlab-sidekiq-all-in-1-v2 -n gitlabbda
   
   # Restart Redis
   kubectl rollout restart statefulset/redis -n gitlabbda
   

Rotating All Secrets

Run the bootstrap script again - it’s idempotent:

cd kube-hetzner
bash scripts/bootstrap-gitlabbda-secrets.sh

ESO will automatically sync updated values within 1 hour (or force sync by deleting ExternalSecrets).

Troubleshooting

ExternalSecret shows “SecretSyncedError”

kubectl describe externalsecret gitlab-app-secrets-es -n gitlabbda

Common causes:

• Source secret doesn’t exist in application-secrets

• ServiceAccount permissions issue

• ClusterSecretStore not ready

Target secret not updating after source change

Check refresh interval:

kubectl get externalsecret gitlab-app-secrets-es -n gitlabbda \
  -o jsonpath='{.spec.refreshInterval}'

Force immediate sync:

kubectl delete secret gitlab-secrets -n gitlabbda
# ESO will recreate within seconds

GitLab pods crashing after secret update

Check if pods picked up new secret:

# Check pod environment
kubectl exec -n gitlabbda deployment/gitlab-webservice-default -- \
  env | grep REDIS

# Restart pods to reload secrets
kubectl rollout restart deployment/gitlab-webservice-default -n gitlabbda

SMTP emails not sending

Verify SMTP credentials are correct:

# Check username
kubectl get secret gitlab-smtp-credentials -n gitlabbda \
  -o jsonpath='{.data.username}' | base64 -d
echo

# Test SMTP connection (from GitLab toolbox)
kubectl exec -it deployment/gitlab-toolbox -n gitlabbda -- \
  gitlab-rake gitlab:smtp:check

Maintenance

Regular Tasks

Monthly:

• Verify all ExternalSecrets are SecretSynced and READY=True

• Check bootstrap credential files are deleted (~/.kup6s/gitlabbda-credentials-*.txt)

Quarterly:

• Consider rotating application secrets (redis-password, secret-key-base, etc.)

• Update SMTP credentials if changed

Annually:

• Full secret rotation (run bootstrap script)

• Review and update Harbor OAuth credentials

Backup Procedures

Source secrets backup:

# Export source secrets to encrypted file
kubectl get secret -n application-secrets \
  gitlabbda-app-secrets-source \
  gitlabbda-smtp-secrets-source \
  gitlabbda-harbor-secrets-source \
  -o yaml > gitlabbda-secrets-backup.yaml

# Encrypt with GPG
gpg --encrypt --recipient admin@example.com gitlabbda-secrets-backup.yaml

# Store encrypted file securely
# Delete plaintext: rm gitlabbda-secrets-backup.yaml

Restore from backup:

# Decrypt
gpg --decrypt gitlabbda-secrets-backup.yaml.gpg > gitlabbda-secrets-backup.yaml

# Apply
kubectl apply -f gitlabbda-secrets-backup.yaml

# Clean up
rm gitlabbda-secrets-backup.yaml

Related Documentation

• How-To: Bootstrap Application Secrets - Generic pattern guide

• How-To: Create Bootstrap Script - Script template

• Explanation: Application Secrets Architecture - Design rationale

• Security Model - Cluster security overview