Explanation

ArgoCD GitOps Workflow for GitLab BDA

Type: Explanation (Understanding-oriented)

Related Concepts: ArgoCD Configuration | CDK8S Approach | ArgoCD GitOps

This document explains GitLab BDA’s specific ArgoCD deployment workflow and sync wave strategy. For general GitOps concepts and ArgoCD fundamentals, see ArgoCD GitOps.

Overview

GitLab BDA uses a 3-wave deployment strategy to ensure proper dependency ordering:

• Wave 1: Infrastructure (namespace, RBAC, S3 buckets)

• Wave 2: Secrets (ExternalSecrets with credentials)

• Wave 3: Applications (PostgreSQL, Redis, GitLab, Harbor)

GitLab BDA Deployment Pipeline

        graph TD
    subgraph "Development (Local)"
        A[Edit TypeScript] -->|charts/*.ts| B[Source .env]
        B --> C[npm run build]
        C --> D{Manifests<br/>Generated}
    end

    subgraph "Version Control"
        D -->|git commit| E[Local Git]
        E -->|git push| F[Git Repository<br/>gitlab/harbor]
    end

    subgraph "ArgoCD (In Cluster)"
        F -->|Poll every 3 min| G[ArgoCD Detects Change]
        G --> H{Compare States}
        H -->|Diff found| I[Calculate Changes]
        I --> J[Apply Sync Waves]
    end

    subgraph "Kubernetes Cluster"
        J -->|Wave 1| K[Infrastructure<br/>Namespace, RBAC, S3]
        K -->|Wave 2| L[Secrets<br/>ExternalSecrets]
        L -->|Wave 3| M[Applications<br/>GitLab, Harbor]
        M --> N[Running Pods]
    end

    N -.->|Monitor| G
    style G fill:#f96
    style F fill:#fc6
    style N fill:#6f6
    

Step-by-Step: Making a Change

1. Developer edits configuration:

cd dp-infra/gitlabbda
vim charts/constructs/database.ts
# Change PostgreSQL storage from 10Gi to 20Gi

2. Build manifests:

# CRITICAL: Source .env to load credentials
bash -c 'source .env && npm run build'

This generates updated manifests/gitlab.k8s.yaml with changes.

3. Review changes:

git diff manifests/gitlab.k8s.yaml

# Shows PostgreSQL PVC size change:
-    storage: 10Gi
+    storage: 20Gi

4. Commit and push:

git add manifests/ charts/
git commit -m "Increase PostgreSQL storage to 20Gi for growth"
git push origin main

5. ArgoCD automatically syncs:

• ArgoCD polls git every 3 minutes (configurable)

• Detects manifest changes

• Applies changes to cluster

• Reports sync status

6. Verify deployment:

# Check ArgoCD Application status
kubectl get application gitlab-bda -n argocd

# Should show:
# SYNC STATUS: Synced
# HEALTH STATUS: Healthy

GitLab BDA Sync Waves

ArgoCD applies resources in waves (numbered order) to respect dependencies:

Wave 1: Infrastructure

Resources:

• Namespace (gitlabbda)

• ServiceAccounts, Roles, RoleBindings

• Crossplane ProviderConfig (S3)

• S3 Buckets (8 buckets: artifacts, uploads, LFS, pages, registry, backups, postgresbackups, cache)

Why wave 1:

• Must exist before secrets and applications

• S3 buckets must be created before PostgreSQL backups configured

• RBAC must exist before applications create pods

Example:

// charts/constructs/namespace.ts
new kplus.Namespace(this, 'namespace', {
  metadata: {
    name: 'gitlabbda',
    annotations: {
      'argocd.argoproj.io/sync-wave': '1',
    },
  },
});

Wave 2: Secrets

Resources:

• ClusterSecretStore (ESO)

• ExternalSecrets (GitLab secrets, PostgreSQL credentials, S3 credentials)

Why wave 2:

• Must exist after infrastructure (wave 1)

• Must exist before applications (wave 3) that reference them

• PostgreSQL needs credentials before starting

• GitLab needs secrets before pods start

Example:

// charts/constructs/app-secrets.ts
new ExternalSecret(this, 'gitlab-secrets', {
  metadata: {
    namespace: 'gitlabbda',
    annotations: {
      'argocd.argoproj.io/sync-wave': '2',
    },
  },
  spec: {
    secretStoreRef: {
      name: 'gitlabbda-app-secrets-store',
      kind: 'ClusterSecretStore',
    },
    // ...
  },
});

Wave 3: Applications

Resources:

• PostgreSQL Cluster (CloudNativePG)

• Redis StatefulSet

• GitLab Helm release

• Harbor Helm release

Why wave 3:

• Depends on secrets (wave 2)

• Depends on S3 buckets (wave 1)

• Can start safely after all dependencies ready

Example:

// charts/constructs/database.ts
new cnpg.Cluster(this, 'postgres', {
  metadata: {
    namespace: 'gitlabbda',
    annotations: {
      'argocd.argoproj.io/sync-wave': '3',
    },
  },
  spec: {
    instances: 2,
    storage: {
      storageClass: 'longhorn-redundant-app',
      size: '10Gi',
    },
    backup: {
      barmanObjectStore: {
        destinationPath: 's3://gitlab-postgresbackups-gitlabbda-kup6s/',
        // S3 credentials from ExternalSecret (wave 2)
      },
    },
  },
});

GitLab BDA Sync Wave Summary

Wave	Component	Resources	Depends On
1	Infrastructure	Namespace, RBAC, ProviderConfig, S3 Buckets	None
2	Secrets	ClusterSecretStore, ExternalSecrets	Wave 1 (namespace, S3)
3	Applications	PostgreSQL, Redis, GitLab, Harbor	Wave 2 (secrets), Wave 1 (S3)

ArgoCD Application Manifest

The ArgoCD Application for GitLab BDA:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gitlab-bda
  namespace: argocd
spec:
  project: default

  source:
    repoURL: https://git.bluedynamics.eu/kup6s/dp/gitlab.git
    targetRevision: main
    path: manifests  # ArgoCD reads from this directory

  destination:
    server: https://kubernetes.default.svc  # This cluster
    namespace: gitlabbda

  syncPolicy:
    automated:
      prune: true      # Delete resources removed from git
      selfHeal: true   # Auto-revert manual kubectl changes
    syncOptions:
      - CreateNamespace=true        # Create namespace if missing
      - ApplyOutOfSyncOnly=true     # Only update changed resources

Key Settings:

• automated.prune: true - Removes resources deleted from git (keeps cluster clean)

• automated.selfHeal: true - Reverts manual kubectl changes (enforces GitOps)

• source.path: manifests - ArgoCD reads generated manifests (not CDK8S charts/)

See ArgoCD Configuration Reference for complete manifest.

GitLab BDA Specific Patterns

1. Multi-Bucket Coordination

Challenge: GitLab needs 8 S3 buckets, all must exist before GitLab starts

Solution: All buckets in wave 1, GitLab in wave 3

// All 8 buckets created in wave 1
new Bucket(this, 'artifacts', {
  metadata: { annotations: { 'argocd.argoproj.io/sync-wave': '1' } },
  spec: { forProvider: { bucket: 'gitlab-artifacts-gitlabbda-kup6s' } },
});

// GitLab in wave 3, buckets guaranteed to exist
new GitlabHelmConstruct(this, 'gitlab', {
  metadata: { annotations: { 'argocd.argoproj.io/sync-wave': '3' } },
  // ...
});

2. PostgreSQL with Barman Backup

Challenge: PostgreSQL backup configuration references S3 bucket and credentials

Solution:

• S3 bucket in wave 1

• S3 credentials in wave 2 (ExternalSecret)

• PostgreSQL in wave 3

// Wave 2: ExternalSecret with S3 credentials
new ExternalSecret(this, 's3-creds', {
  metadata: { annotations: { 'argocd.argoproj.io/sync-wave': '2' } },
  // ...
});

// Wave 3: PostgreSQL references credentials
new cnpg.Cluster(this, 'postgres', {
  metadata: { annotations: { 'argocd.argoproj.io/sync-wave': '3' } },
  spec: {
    backup: {
      barmanObjectStore: {
        destinationPath: 's3://gitlab-postgresbackups-gitlabbda-kup6s/',
        s3Credentials: {
          accessKeyId: { name: 's3-creds', key: 'access-key' },
          secretAccessKey: { name: 's3-creds', key: 'secret-key' },
        },
      },
    },
  },
});

3. ExternalSecrets with ClusterSecretStore

Challenge: ExternalSecret needs ClusterSecretStore to exist first

Solution: Both in wave 2, but use sub-ordering (ArgoCD sorts alphabetically within wave)

// Created first (alphabetically: "a-...")
new ClusterSecretStore(this, 'a-secret-store', {
  metadata: { annotations: { 'argocd.argoproj.io/sync-wave': '2' } },
  // ...
});

// Created second (alphabetically: "z-...")
new ExternalSecret(this, 'z-app-secrets', {
  metadata: { annotations: { 'argocd.argoproj.io/sync-wave': '2' } },
  spec: { secretStoreRef: { name: 'gitlabbda-app-secrets-store' } },
});

Monitoring GitLab BDA Deployment

# Check overall status
kubectl get application gitlab-bda -n argocd

# Check which wave is currently applying
argocd app get gitlab-bda --show-operation

# View sync history
argocd app history gitlab-bda

# Check for drift
argocd app diff gitlab-bda

Troubleshooting

Application Stuck on Wave 2

Symptoms:

argocd app get gitlab-bda --show-operation
# Shows: Waiting for Wave 2 resources

Solution: Check ExternalSecrets synced

# Check ExternalSecret status
kubectl get externalsecret -n gitlabbda

# Check if secrets created
kubectl get secret -n gitlabbda | grep gitlab

# If not synced, check ClusterSecretStore
kubectl get clustersecretstore gitlabbda-app-secrets-store

PostgreSQL Pods CrashLoopBackOff

Symptoms:

kubectl get pods -n gitlabbda | grep postgres
# postgres-1 CrashLoopBackOff

Common causes:

1. S3 credentials missing (wave 2 not complete)

2. S3 bucket doesn’t exist (wave 1 failed)

3. Barman backup configuration invalid

Solution:

# Check secrets exist
kubectl get secret -n gitlabbda | grep s3

# Check S3 buckets created
kubectl get bucket -A | grep gitlab

# Check PostgreSQL logs
kubectl logs -n gitlabbda postgres-1

GitLab Pods Pending

Symptoms:

kubectl get pods -n gitlabbda | grep gitlab
# gitlab-webservice-xxx Pending

Common causes:

1. PostgreSQL not ready (wave 3 dependency)

2. Redis not ready (wave 3 dependency)

3. Secrets missing (wave 2 not complete)

Solution:

# Check PostgreSQL ready
kubectl get cluster -n gitlabbda gitlab-postgres

# Check Redis ready
kubectl get statefulset -n gitlabbda redis

# Check secrets exist
kubectl get secret -n gitlabbda | grep gitlab-secrets

Best Practices

1. Always Build Before Commit

Good:

bash -c 'source .env && npm run build'
git add charts/ manifests/
git commit -m "Update PostgreSQL storage"

Bad:

# Forgot to rebuild manifests
git add charts/
git commit -m "Update PostgreSQL storage"
# manifests/ out of sync!

2. Descriptive Commit Messages

Good:

feat: Increase PostgreSQL storage to 20Gi

Anticipating growth from 5 to 10 users over next quarter.
Storage usage currently at 7Gi, will hit limit in ~2 months.

Affects:
- PostgreSQL PVC size
- Barman backup configuration

See: #123

Bad:

Update config

3. Verify Sync Waves in Manifests

# After build, check sync waves are correct
grep "sync-wave" manifests/gitlab.k8s.yaml | sort -u

# Should show:
#   sync-wave: "1"  # Infrastructure
#   sync-wave: "2"  # Secrets
#   sync-wave: "3"  # Applications

4. Monitor Deployment Progress

# Watch deployment progress
watch kubectl get application gitlab-bda -n argocd

# Check pod status during deployment
watch kubectl get pods -n gitlabbda

Related Documentation

General GitOps Concepts

• ArgoCD GitOps - Universal GitOps concepts, sync policies, troubleshooting

GitLab BDA Specific

• Architecture Overview - GitLab BDA system design

• CDK8S Approach - How CDK8S generates manifests

• ArgoCD Configuration Reference - Complete Application manifest

• Sync Waves Reference - All sync wave assignments

External Resources

• ArgoCD Official Documentation

• GitOps Principles