Setup: Grafana Admin Password with ESO¶

This guide shows how to set up stable Grafana admin credentials using External Secrets Operator (ESO), following the centralized secret management pattern from the GitLab BDA security model.

Overview¶

Architecture:

application-secrets namespace (source)
  └─ Secret: grafana-admin
       ├─ admin-user: admin
       └─ admin-password: <your-password>

     ↓ ClusterSecretStore: monitoring-app-secrets-store

monitoring namespace (target)
  └─ ExternalSecret: grafana-admin-credentials-es
       ↓ (replicates from application-secrets)
     Secret: grafana-admin-credentials
       ├─ admin-user: admin
       └─ admin-password: <your-password>

     ↓ Grafana Helm chart

Grafana pods use grafana-admin-credentials secret

Prerequisites¶

External Secrets Operator installed (already deployed)
application-secrets namespace exists (already exists)
ServiceAccount eso-app-secrets-reader exists in application-secrets namespace

Step 1: Create Source Secret¶

Create the Grafana admin password in the centralized application-secrets namespace:

# Generate a secure password (or use your own)
PASSWORD=$(openssl rand -base64 24)
echo "Generated password: $PASSWORD"
echo "SAVE THIS PASSWORD - you'll need it to login!"

# Create secret in application-secrets namespace
kubectl create secret generic grafana-admin -n application-secrets \
  --from-literal=admin-user=admin \
  --from-literal=admin-password="$PASSWORD"

# Verify secret created
kubectl get secret grafana-admin -n application-secrets

Alternative: Use a specific password instead of generating:

kubectl create secret generic grafana-admin -n application-secrets \
  --from-literal=admin-user=admin \
  --from-literal=admin-password='YourSecurePassword123!'

Step 2: Create ClusterSecretStore¶

Create a ClusterSecretStore that allows the monitoring namespace to read secrets from application-secrets:

kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: monitoring-app-secrets-store
spec:
  # Only allow monitoring namespace to use this store
  conditions:
  - namespaces:
    - monitoring

  # Kubernetes provider (read from same cluster)
  provider:
    kubernetes:
      # Remote namespace containing source secrets
      remoteNamespace: application-secrets

      # Use in-cluster Kubernetes API
      server:
        url: kubernetes.default
        caProvider:
          type: ConfigMap
          name: kube-root-ca.crt
          namespace: kube-system
          key: ca.crt

      # ServiceAccount with RBAC to read secrets
      auth:
        serviceAccount:
          name: eso-app-secrets-reader
          namespace: application-secrets
EOF

Verify ClusterSecretStore is ready:

kubectl get clustersecretstore monitoring-app-secrets-store
# STATUS should be "Valid", READY should be "True"

Step 3: Deploy Monitoring Stack¶

The monitoring stack CDK8S code already includes the ExternalSecret configuration. Simply deploy:

cd /home/jensens/ws/pro/kup6s/dp-infra/monitoring

# Build manifests (if needed)
npm run build

# Apply manifests
kubectl apply -f manifests/monitoring.k8s.yaml

Step 4: Verify Secret Replication¶

Check that ESO successfully replicated the secret:

# Check ExternalSecret status
kubectl get externalsecret grafana-admin-credentials-es -n monitoring
# STATUS should be "SecretSynced"

# Check target secret was created
kubectl get secret grafana-admin-credentials -n monitoring
# Should show secret with 2 data keys: admin-user, admin-password

# Verify secret contents (base64 encoded)
kubectl get secret grafana-admin-credentials -n monitoring -o jsonpath='{.data.admin-user}' | base64 -d
# Should output: admin

Step 5: Login to Grafana¶

Access Grafana and login with the credentials:

# Get password from source secret
PASSWORD=$(kubectl get secret grafana-admin -n application-secrets -o jsonpath='{.data.admin-password}' | base64 -d)
echo "Grafana password: $PASSWORD"

# Grafana URL
echo "Grafana URL: https://grafana.ops.kup6s.net"
echo "Username: admin"

Password Rotation¶

To rotate the Grafana password:

# 1. Update source secret in application-secrets
kubectl create secret generic grafana-admin -n application-secrets \
  --from-literal=admin-user=admin \
  --from-literal=admin-password='NewSecurePassword456!' \
  --dry-run=client -o yaml | kubectl apply -f -

# 2. ESO will automatically sync (within 1 hour, or force immediate sync)
kubectl annotate externalsecret grafana-admin-credentials-es -n monitoring \
  force-sync=$(date +%s) --overwrite

# 3. Restart Grafana to pick up new password
kubectl rollout restart deployment/kube-prometheus-stack-grafana -n monitoring

# 4. Wait for Grafana to be ready
kubectl rollout status deployment/kube-prometheus-stack-grafana -n monitoring

Troubleshooting¶

ExternalSecret shows “SecretSyncedError”¶

Check ExternalSecret status:

kubectl describe externalsecret grafana-admin-credentials-es -n monitoring

Common causes:

ClusterSecretStore not ready
Source secret doesn’t exist in application-secrets
ServiceAccount lacks RBAC permissions

ClusterSecretStore not ready¶

Check ClusterSecretStore status:

kubectl describe clustersecretstore monitoring-app-secrets-store

Verify ServiceAccount exists:

kubectl get serviceaccount eso-app-secrets-reader -n application-secrets

Verify RBAC:

kubectl get role,rolebinding -n application-secrets | grep eso

Grafana still auto-generates password¶

Check Helm values are applied:

kubectl get helmchart kube-prometheus-stack -n monitoring -o yaml | grep -A 5 "admin:"

Should show:

admin:
  existingSecret: "grafana-admin-credentials"
  userKey: "admin-user"
  passwordKey: "admin-password"

If not, HelmChart needs to be recreated (ArgoCD sync issue).

Security Considerations¶

Why this approach?

Centralized: All application secrets in application-secrets namespace
GitOps-safe: ExternalSecret CRD in git, actual password NOT in git
Rotation-ready: Update source secret → auto-syncs to all consumers
Follows best practices: Same pattern as GitLab BDA deployment

Least privilege:

ClusterSecretStore only allows monitoring namespace
ServiceAccount only has read access to secrets
Source secret isolated in application-secrets namespace

Audit trail:

Secret creation logged in Kubernetes audit logs
ExternalSecret sync events visible in ESO logs
Password rotation tracked via Kubernetes events