Migrate AlpineCity from Docker Swarm to Kubernetes¶

This document describes migrating the AlpineCity Plone site (alpinecity.tirol) from the old Docker Swarm hosting to the kup6s K3S cluster.

Overview¶

Component	Swarm	Kubernetes
PostgreSQL	eeacms/postgres:9.6 (bind-mount)	StatefulSet with Longhorn PVC
Memcached	memcached:1	Deployment
Plone	2 replicas, bind-mount blobs	1 replica, Longhorn PVC blobs
Ingress	Traefik (Swarm labels)	Traefik (K8s Ingress + Middleware)
Secrets	env vars in compose	ESO ExternalSecrets
Domains	alpinecity.tirol, alpinecity.web.kup.tirol	alpinecity.tirol, alpinecity.sites.kup6s.com

Estimated downtime: 30–60 minutes (phases 2–5).

Prerequisites¶

One-time secrets setup¶

Create the source secrets in the application-secrets namespace and the ClusterSecretStore (if not already done):

# Create source secret with DB credentials and registry credentials
kubectl create secret generic alpinecity-secrets-source \
  --namespace application-secrets \
  --from-literal=POSTGRES_USER=postgres \
  --from-literal=POSTGRES_PASSWORD=postgres \
  --from-literal=POSTGRES_DBNAME=plone \
  --from-literal=POSTGRES_DBUSER=plone \
  --from-literal=POSTGRES_DBPASS=plone

# Create separate source for DB credentials ExternalSecret
kubectl create secret generic alpinecity-db-credentials-source \
  --namespace application-secrets \
  --from-literal=POSTGRES_USER=postgres \
  --from-literal=POSTGRES_PASSWORD=postgres \
  --from-literal=POSTGRES_DBNAME=plone \
  --from-literal=POSTGRES_DBUSER=plone \
  --from-literal=POSTGRES_DBPASS=plone

# Create source for registry credentials
kubectl create secret generic alpinecity-registry-source \
  --namespace application-secrets \
  --from-literal=docker_server=registry.bluedynamics.eu \
  --from-literal=docker_username=<REGISTRY_USER> \
  --from-literal=docker_password=<REGISTRY_PASS>

# Create ClusterSecretStore (if not exists)
kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: alpinecity-app-secrets-store
spec:
  conditions:
    - namespaces:
        - alpinecity
  provider:
    kubernetes:
      remoteNamespace: application-secrets
      auth:
        serviceAccount:
          name: eso-app-secrets-reader
          namespace: application-secrets
      server:
        url: kubernetes.default
        caProvider:
          type: ConfigMap
          name: kube-root-ca.crt
          namespace: kube-system
          key: ca.crt
EOF

Verify K8s infrastructure¶

# ArgoCD should have synced the alpinecity app
kubectl get ns alpinecity
kubectl get pods -n alpinecity
kubectl get pvc -n alpinecity

# Expected: namespace exists, db-0 pod running, memcached running,
# plone deployment at 1 replica (will fail until data is migrated),
# PVCs bound

Phase 1: Pre-migration verification (no downtime)¶

Verify the K8s deployment is operational (minus data):

# Check all pods
kubectl get pods -n alpinecity

# Check PVCs are bound
kubectl get pvc -n alpinecity

# Check ExternalSecrets are synced
kubectl get externalsecrets -n alpinecity

# Check DB is reachable
kubectl exec -n alpinecity db-0 -- pg_isready -U postgres

Phase 2: Enable maintenance mode on Swarm (downtime starts)¶

# On the Swarm host — scale Plone to 0
ssh <SWARM_HOST>
cd /path/to/alpinecity
docker service scale alpinecity_plone=0

# Record the timestamp for rollback reference
date -u +%Y-%m-%dT%H:%M:%SZ

Phase 3: Database migration¶

3.1 Dump on Swarm host¶

# On Swarm host
docker exec $(docker ps -q -f name=alpinecity_db) \
  pg_dump -U plone -Fc plone > /tmp/plone.dump

# Verify dump
ls -lh /tmp/plone.dump

3.2 Transfer to local machine¶

scp <SWARM_HOST>:/tmp/plone.dump /tmp/plone.dump

3.3 Restore into K8s¶

# Copy dump into DB pod
kubectl cp /tmp/plone.dump alpinecity/db-0:/tmp/plone.dump

# Drop and recreate the database to ensure clean state
kubectl exec -n alpinecity db-0 -- bash -c \
  'dropdb -U postgres --if-exists plone && createdb -U postgres -O plone plone'

# Restore
kubectl exec -n alpinecity db-0 -- \
  pg_restore -U plone -d plone --no-owner --no-privileges /tmp/plone.dump

# Verify
kubectl exec -n alpinecity db-0 -- \
  psql -U plone -d plone -c "SELECT count(*) FROM pg_tables WHERE schemaname = 'public'"

# Clean up dump
kubectl exec -n alpinecity db-0 -- rm /tmp/plone.dump

Phase 4: Blobstorage migration¶

4.1 Archive on Swarm host¶

# On Swarm host
tar czf /tmp/blobs.tar.gz -C /data/alpinecity/blobs .

# Verify
ls -lh /tmp/blobs.tar.gz

4.2 Transfer and restore¶

# Transfer to local
scp <SWARM_HOST>:/tmp/blobs.tar.gz /tmp/blobs.tar.gz

# Scale Plone to 0 temporarily if needed (to get PVC mounted by a helper)
kubectl scale deployment plone -n alpinecity --replicas=0

# Start a temporary pod to copy blobs into the PVC
kubectl run blob-copier -n alpinecity \
  --image=alpine:latest \
  --restart=Never \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "blob-copier",
        "image": "alpine:latest",
        "command": ["sleep", "3600"],
        "volumeMounts": [{"name": "blobs", "mountPath": "/blobstorage"}]
      }],
      "volumes": [{
        "name": "blobs",
        "persistentVolumeClaim": {"claimName": "alpinecity-blobstorage"}
      }]
    }
  }'

# Wait for pod to be ready
kubectl wait --for=condition=Ready pod/blob-copier -n alpinecity --timeout=60s

# Copy and extract
kubectl cp /tmp/blobs.tar.gz alpinecity/blob-copier:/tmp/blobs.tar.gz
kubectl exec -n alpinecity blob-copier -- tar xzf /tmp/blobs.tar.gz -C /blobstorage

# Verify
kubectl exec -n alpinecity blob-copier -- ls -la /blobstorage/

# Clean up helper pod
kubectl delete pod blob-copier -n alpinecity

# Scale Plone back up
kubectl scale deployment plone -n alpinecity --replicas=1

Phase 5: Verify Plone on K8s (downtime ends)¶

# Wait for Plone pod to be ready
kubectl wait --for=condition=Ready pod -l app.kubernetes.io/component=web \
  -n alpinecity --timeout=120s

# Check pod logs
kubectl logs -n alpinecity -l app.kubernetes.io/component=web --tail=50

# Test via ZMI domain (already pointed to K8s)
curl -sI https://alpinecity.sites.kup6s.com/
# Expected: HTTP/2 200

Phase 6: DNS cutover¶

Update DNS for alpinecity.tirol to point to the K8s cluster ingress IP
Wait for DNS propagation (check with dig alpinecity.tirol)
Verify TLS certificate is issued:

kubectl get certificate -n alpinecity
# Expected: alpinecity-production-tls should be Ready=True

Test production domain access:

curl -sI https://alpinecity.tirol/
# Expected: HTTP/2 200

Phase 7: Cleanup¶

Keep Swarm running for 7 days as a rollback option
After verification period:
- Shut down Swarm services: docker service rm alpinecity_plone alpinecity_db alpinecity_memcached
- Remove swarm/alpinecity/ from the kup6s repository
- Clean up /tmp/plone.dump and /tmp/blobs.tar.gz from all hosts

Rollback¶

If issues are found before DNS cutover:

Scale K8s Plone to 0: kubectl scale deployment plone -n alpinecity --replicas=0
Restart Swarm Plone: docker service scale alpinecity_plone=2
Investigate and retry

If issues are found after DNS cutover:

Revert DNS for alpinecity.tirol to Swarm host IP
Restart Swarm Plone: docker service scale alpinecity_plone=2
Investigate the K8s deployment

Troubleshooting¶

Plone pod CrashLoopBackOff¶

Check logs: kubectl logs -n alpinecity -l app.kubernetes.io/component=web --previous

Common causes:

DB not reachable: verify db service exists and DB pod is running
Blobstorage permissions: verify files in PVC have correct ownership
Missing DB data: verify pg_restore completed successfully

ExternalSecret not syncing¶

kubectl describe externalsecret -n alpinecity
# Check Events section for errors
# Common: ClusterSecretStore not found or source secret missing

TLS certificate not issued¶

kubectl describe certificate alpinecity-production-tls -n alpinecity
kubectl describe certificaterequest -n alpinecity
# Check cert-manager logs if needed