How-To Guide

Restore etcd from an S3 Snapshot (Disaster Recovery)

Type: How-To (Task-oriented)

Related: Back Up K3S Encryption Keys | Infrastructure as Code

Goal: Restore a K3S control plane from an etcd snapshot stored in S3, either after losing a single control plane node or after losing the whole control-plane quorum (full disaster recovery).

Time: ~20–40 minutes (single-server reset + rejoin of the remaining CPs)

Critical: An etcd snapshot restores namespaces, workloads, and CRs — but Secrets are encrypted at rest and their decryption key is NOT in the snapshot. You MUST have the encryption-config.json backup (see Back Up K3S Encryption Keys) or every Secret in the cluster is unrecoverable. This was verified on 2026-07-02 (see Verification).

What the etcd snapshot contains — and what it does not

Recoverable from the etcd S3 snapshot	NOT in the snapshot — back up separately
Namespaces, Deployments, all workload state	Secret values (AES-CBC encrypted at rest)
CRs (ArgoCD Applications, CNPG Clusters, …)	The encryption key → /var/lib/rancher/k3s/server/cred/encryption-config.json
Node objects, RBAC, ConfigMaps	The cluster join token → /var/lib/rancher/k3s/server/token
ArgoCD then re-installs operators/CRDs	.env credentials + kube.tf config (in git)

For a survivable DR you need all four of: the S3 etcd snapshot, the encryption-config.json, the cluster token, and the git-tracked tofu config.

Where the backups live

• etcd S3 snapshots — bucket per cluster, region = the endpoint region. For helix-production: backup-etcd-prod-helixpass-anau on nbg1.your-objectstorage.com. Uploaded by every control plane every 12h (etcd-snapshot-schedule-cron = "0 */12 * * *"), retention 10 (configured in kube-hetzner/module/main.tf, etcd_s3_backup).

• Snapshot name pattern: <name>-<cp-hostname>-<unixtime> (one object per CP per run).

• The S3 credentials + endpoint are in the cluster’s .env (TF_VAR_hetzner_s3_access_key / _secret_key, TF_VAR_etcdbackup_s3_endpoint).

Prerequisites

• The encryption-config.json backup for this cluster (KeePass entry <cluster>-k3s-encryption-config) — see Back Up K3S Encryption Keys.

• The cluster join token (from any surviving CP, or your bootstrap backup).

• SSH access to at least one (target) control plane node.

• The cluster’s .env sourced for S3 credentials.

Step 1: Pick a snapshot

List from any surviving CP:

ssh root@<cp-ip> "k3s etcd-snapshot ls"

Or list the S3 bucket directly (no CP needed — useful in a full-region loss). There is no mc/aws-cli on the nodes and /usr/bin/mc is Midnight Commander — use boto3:

cd kube-hetzner/clusters/<cluster> && source .env
uv run --with boto3 python3 - <<'PY'
import os, boto3
ep = "https://" + os.environ["TF_VAR_etcdbackup_s3_endpoint"]
region = os.environ["TF_VAR_etcdbackup_s3_endpoint"].split(".")[0]
s3 = boto3.client("s3", endpoint_url=ep, region_name=region,
                  aws_access_key_id=os.environ["TF_VAR_hetzner_s3_access_key"],
                  aws_secret_access_key=os.environ["TF_VAR_hetzner_s3_secret_key"])
b = os.environ["TF_VAR_etcdbackup_s3_bucket"]
for k in sorted(s3.list_objects_v2(Bucket=b).get("Contents", []), key=lambda x: x["LastModified"]):
    print(k["LastModified"].strftime("%Y-%m-%d %H:%M"), f'{k["Size"]/1e6:.1f}MB', k["Key"])
PY

Step 2: Stop k3s on ALL other control plane nodes

The reset must happen on exactly one node; the others must be down so they don’t fight the new single-member etcd.

# On EVERY control plane EXCEPT the chosen restore target:
ssh root@<other-cp-ip> "systemctl stop k3s"

Step 3: Reset + restore on ONE control plane node

Two options — the file-path method (proven in the drill below) is the robust default.

Option A — download the snapshot, restore from a local file:

# On your workstation: download the snapshot (boto3 as in Step 1, s3.download_file(...))
# then copy it to the target CP:
scp <snapshot> root@<target-cp-ip>:/tmp/

ssh root@<target-cp-ip>
k3s server \
  --cluster-reset \
  --cluster-reset-restore-path=/tmp/<snapshot> \
  --token="<cluster-token>"

Option B — restore directly from S3 (the CP already carries the etcd-s3 config in /etc/rancher/k3s/config.yaml, so only the path is needed):

ssh root@<target-cp-ip>
k3s server \
  --cluster-reset \
  --cluster-reset-restore-path=s3://<bucket>/<snapshot> \
  --token="<cluster-token>"

--token is mandatory — without it k3s aborts with server/token does not exist, please pass --token to complete the restoration.

This is a one-shot: k3s restores etcd, resets membership to a single node, backs up the old TLS dir, and exits with: Managed etcd cluster membership has been reset, restart without --cluster-reset flag now.

Step 4: Restore the encryption key BEFORE the first normal start

If this node’s disk is intact (single-node loss), the key is still in place — skip. If restoring onto a fresh node (full DR), copy the backed-up encryption-config.json into place now, otherwise the API server comes up but cannot decrypt any Secret:

# On the target CP, before starting k3s normally:
install -m 600 -o root -g root /path/to/encryption-config.json \
  /var/lib/rancher/k3s/server/cred/encryption-config.json

Step 5: Start the restore target normally

# On the target CP:
systemctl start k3s        # (installed nodes) — starts without --cluster-reset
# verify:
k3s kubectl get ns
k3s secrets-encrypt status # -> Enabled, hashes match
k3s kubectl get secret -A  # -> Secrets are LISTED AND DECRYPTABLE (proves Step 4)

Step 6: Rejoin the other control plane nodes

On each other CP, wipe its stale etcd member and rejoin the restored node:

ssh root@<other-cp-ip>
systemctl stop k3s
rm -rf /var/lib/rancher/k3s/server/db          # remove stale etcd member data
systemctl start k3s                            # rejoins; --server already points at the cluster

Rejoin one at a time, waiting for each to become Ready and for etcd to report the new member healthy before doing the next, to preserve quorum. Verify:

k3s kubectl get nodes
curl -sk https://127.0.0.1:2381/metrics | grep etcd_server_id   # on each CP
ssh root@<cp> "journalctl -u k3s | grep 'added member'"

Verification (restore drill)

Tested: 2026-07-02 · Cluster: helix-production · k3s: v1.34.9+k3s1 Snapshot: a live k3s etcd-snapshot save from control-fsn1, pulled from backup-etcd-prod-helixpass-anau (nbg1).

Method (non-destructive, does NOT touch live prod): restore-to-scratch in a throwaway single-node k3s in Docker — the definitive proof that the S3 snapshot is valid and restorable end-to-end. Repeatable drill:

SNAP=/tmp/<downloaded-snapshot>; TOKEN=<prod cluster token>
docker volume rm k3s-dr-data 2>/dev/null

# Phase 1 — cluster-reset-restore (one-shot, exits):
docker run --rm --privileged -e K3S_TOKEN="$TOKEN" \
  -v k3s-dr-data:/var/lib/rancher/k3s -v "$(dirname $SNAP)":/snap:ro \
  rancher/k3s:v1.34.9-k3s1 \
  server --cluster-reset --cluster-reset-restore-path=/snap/$(basename $SNAP) --token="$TOKEN"

# Phase 2 — start normally, then inspect:
docker run -d --name k3s-dr-test --privileged \
  -v k3s-dr-data:/var/lib/rancher/k3s \
  rancher/k3s:v1.34.9-k3s1 server --disable=traefik,servicelb,metrics-server
docker exec k3s-dr-test kubectl get ns
docker exec k3s-dr-test kubectl get nodes

# teardown:
docker rm -f k3s-dr-test; docker volume rm k3s-dr-data

Outcome: ✅ etcd restored (current-rev: 16485638); the scratch API server served the real helix-production state — all 8 node objects (control-fsn1/hel1/nbg1, witness-nbg1, fsn1-1/2, hel1-1/2), all 17 namespaces (argocd, cnpg-system, crossplane-system, longhorn-system, strimzi, ingress-system, traefik, …), and 7 ArgoCD Applications.

Key finding: in the scratch restore, kubectl get secret -A returned nothing and CNPG Clusters showed 0 — because (a) the throwaway node generated a fresh encryption-config.json so the prod-encrypted Secrets were undecryptable, and (b) operator CRDs aren’t installed on a bare node. Both are expected. The (a) case is the whole reason Step 4 exists: a full-DR restore onto fresh nodes MUST restore the original encryption-config.json, or all Secrets are lost. Snapshot integrity itself was proven regardless.

Troubleshooting

• server/token does not exist — you omitted --token in Step 3. It is mandatory.

• API up but kubectl get secret empty / decryption errors — the encryption-config.json doesn’t match the snapshot’s key (Step 4 was skipped or used the wrong key). Restore the correct key from the KeePass backup.

• Scheduled S3 snapshots missing while on-demand works — the k3s server cached a stale S3 client (e.g. bucket created after k3s started, so it cached “bucket does not exist”). A control-plane restart drops the cache. See k3s upgrade failures for the restart-via-tofu path.

Related Information

• Back Up K3S Encryption Keys — the other half of DR; without it this procedure cannot recover Secrets.

• Apply Infrastructure Changes