How-To Guide

Bootstrap a greenfield multi-region cluster

Type: How-To (Task-oriented)

Related: Add a Dedicated Robot Worker | Apply Changes | Create S3 Buckets

Goal: Provision a brand-new kube-hetzner cluster stretched across two Hetzner regions from an empty cloud project — control planes plus a witness via OpenTofu, dedicated Robot workers per region via Combustion, region-local ingress, and the post-bootstrap manifests the cluster needs.

This guide is written from the helix-production greenfield bring-up (fsn1 + hel1 workers, nbg1 witness) on 2026-06-18/19. It uses those concrete values as the worked example; substitute your own cluster name, regions, and IDs. For a single-region cluster, drop the second region and the indirection in the DNS step.

Important

This procedure has several traps that are easy to hit and hard to diagnose after the fact. Read the whole guide once before you start — especially Cilium encryption, the load balancer quota, and stale pod IPs.

Prerequisites

  • An empty Hetzner Cloud project (its own token), and a Hetzner Robot account in the same billing context for the dedicated workers.

  • Robot webservice credentials and a vSwitch (VLAN 4000) whose servers are assigned, as described in Add a Dedicated Robot Worker.

  • The fleet SSH key for the cluster (for example ~/.ssh/helix-production[.pub]), registered in the Hetzner project.

  • A GitOps repository for the cluster’s ArgoCD applications, and a charts repository for the workloads.

  • Working knowledge of Apply Changes — every OpenTofu run goes through scripts/apply-and-configure-longhorn.sh.

Step 1 — Prepare the cluster directory and .env

Create the cluster directory from the template and fill its .env (git-crypt protects .env, terraform.tfvars, and terraform.tfstate):

cp -r kube-hetzner/clusters/_template kube-hetzner/clusters/<cluster>
cp kube-hetzner/.env.example kube-hetzner/clusters/<cluster>/.env

Set in .env: hcloud_token, cluster_name, the base/api/ops domains, the SSH key paths, the S3 keys and endpoints, letsencrypt_email, the ArgoCD argoapps_git_* values, and the Robot robot_user / robot_password. Leave the Storage Box (cifs) variables empty unless you deliberately use one.

Important

Choose the control-plane architecture now. ARM cax is frequently sold out across fsn1/nbg1/hel1 — verify orderable capacity with hcloud datacenter list -o json (check server_types.available) before committing to cax. helix-production uses x86 cx33 control planes because ARM was sold out fleet-wide.

Step 2 — Plan the node pools

Edit terraform.tfvars. Three control planes (one per region), one cloud witness in the third region, and one temporary untainted worker for the bootstrap:

Important

A greenfield bootstrap needs at least one schedulable worker. The control planes and the KRaft witness are tainted, and the dedicated workers join only later, so the bootstrap add-ons (Traefik, Longhorn, cert-manager, ArgoCD) stay Pending and the apply hangs at “Waiting for load balancer to get an IP”. Add a temporary untainted cloud agent (for example agent-bootstrap-temp, cx33) to the node pools, and retire it in Step 7.

Do not taint the witness yet — that also belongs in Step 7. Respect list-index immutability: never reorder or insert pool entries; retire with count = 0 and append new entries.

Wire the Robot and vSwitch passthroughs in terraform.tfvars:

hetzner_ccm_use_helm = true
robot_ccm_enabled    = true
vswitch_id           = 82916   # your vSwitch ID
vswitch_subnet_index = 6       # 10.6.0.0/16, gateway .1, dedicated nodes .2+

Step 3 — First apply: control planes, witness, and the temp worker

Run the apply through the script (from the repository root):

bash scripts/apply-and-configure-longhorn.sh <cluster>

This creates the network, three etcd control planes, the witness, the temporary worker, and the bootstrap tier (Cilium, Longhorn, Traefik, cert-manager, ArgoCD, External Secrets, the Helm-based Robot CCM).

Verify before continuing:

kubectl get nodes
kubectl get --raw '/healthz/etcd' ; echo
kubectl get pods -A | grep -v -E 'Running|Completed'

Note

Until DNS exists (later steps), the generated kubeconfig points at an unresolvable api.<domain>. Reach the cluster directly with --server https://<control-plane-public-ip>:6443 --insecure-skip-tls-verify=true.

Step 4 — Cilium: native routing, WireGuard, and nodeEncryption

The shared module configures Cilium with native routing + WireGuard + nodeEncryption: true, which encrypts both pod-to-pod and node-to-node traffic across the cross-region vSwitch. Keep this default.

Warning

cilium_values is a full replace of the module-computed Cilium values, not a merge. Setting cilium_values to a small snippet (for example only encryption.nodeEncryption: false) silently discards native routing, WireGuard, and kube-proxy replacement, and Cilium falls back to an unencrypted VXLAN overlay. For a surgical per-key override, use cilium_merge_values (deep-merge) instead, which the module passes through alongside cilium_values. For a standard cluster, set neither — the module default is correct and matches the kup6s production cluster.

nodeEncryption: false on this topology (dedicated workers on the vSwitch subnet, control planes on the cloud subnet) breaks pod-to-host traffic asymmetrically: pod-to-pod works, but pod-to-apiserver and pod-to-node time out. nodeEncryption: true is the working configuration.

Note

Immediately after a Cilium rollout the WireGuard peer mesh needs a few minutes to converge. Add-ons may CrashLoop with dial tcp 10.43.0.1:443: i/o timeout during that window, then recover on their own. A sustained TCP-connect test from a worker pod to the apiserver ClusterIP (20 attempts, all succeeding) confirms the mesh is stable.

Step 5 — Join the dedicated Robot workers

Join each dedicated server as a worker by following Add a Dedicated Robot Worker (Steps 1–7: Rescue, MicroOS, partition both NVMe disks, Combustion, grow root, open-iscsi, K3S join with a Robot provider-id, and the Longhorn disks).

Use the per-host values from your plan: hostname <cluster>-<region>-<n>, private IP 10.6.0.<n>/24, and zone label topology.kubernetes.io/zone=<region>.

Important

Set topology.kubernetes.io/zone to the region (fsn1, hel1), not a per-node value. Region-level zone labels are what the region-local ingress and rack-aware workloads depend on.

Step 6 — Configure Longhorn disks

Configure both NVMe disks on each worker per Add a Dedicated Robot Worker (Step 7): mount by UUID, replace the default root-disk entry with the two xfs disks, and reserve 7 percent per disk.

Warning

If you mounted the xfs disks over /var/lib/longhorn after Longhorn was already running, the engine binary copied to the old path is now hidden under the empty mount, and the Longhorn BackupTarget later fails with fork/exec .../longhorn: no such file or directory. Restart the engine-image pods to repopulate the binary: kubectl -n longhorn-system delete pods -l longhorn.io/component=engine-image.

Step 7 — Retire the temporary worker and taint the witness

With the dedicated workers Ready and serving Longhorn, the bootstrap add-ons can move off the temporary worker.

Confirm nothing irreplaceable lives on it first:

kubectl get pods -A --field-selector spec.nodeName=<temp-worker> -o wide
kubectl -n longhorn-system get replicas.longhorn.io -o jsonpath='{range .items[*]}{.spec.nodeID}{"\n"}{end}' | sort | uniq -c

In terraform.tfvars, set the temporary worker pool to count = 0 and add the witness taint (for example dedicated=kafka-controller:NoSchedule). Apply through the script.

Important

The script aborts on any server destroy (“DESTRUCTIVE INFRASTRUCTURE CHANGES DETECTED”). Retiring the temporary worker is exactly such a destroy — re-run with --force once you have confirmed the plan removes only the temporary worker.

The add-ons reschedule onto the dedicated workers. Delete the stale Longhorn node object for the retired worker if it lingers (disable scheduling first, since the webhook refuses to delete a schedulable node):

kubectl -n longhorn-system patch nodes.longhorn.io <temp-worker> --type merge -p '{"spec":{"allowScheduling":false}}'
kubectl -n longhorn-system delete nodes.longhorn.io <temp-worker>

Step 8 — Apply the post-bootstrap manifests

These resources are not part of the shared bootstrap and are applied manually (or via the cluster’s GitOps once it is wired). Keep them under kube-hetzner/clusters/<cluster>/manifests/.

  • local-data StorageClass — Longhorn numberOfReplicas: "1", dataLocality: "strict-local" for data that gets its durability from the application layer (CloudNativePG synchronous replication, Kafka ISR) rather than from Longhorn.

  • nbg1 cold-backup S3 — a hetzner-s3-nbg1 ProviderConfig plus two buckets, following Create S3 Buckets with Crossplane. Always set managementPolicies: [Observe, Create, Delete] so Crossplane skips the tagging call that Hetzner Object Storage rejects with 501. Never use server-side encryption.

  • Longhorn BackupTarget — point it at the nbg1 bucket with a credential secret created out of band (S3 keys from .env), no SSE.

Important

Secrets are created out of band, never committed. Create the Longhorn backup-target secret — and the ArgoCD repository credential in Step 11 — with kubectl from values held in the cluster .env or your password manager, and record the step here rather than the secret itself.

Step 9 — Ingress: two region-local load balancers

kube-hetzner deploys a single Traefik with a single load balancer, which cannot give you region-local entry. Deploy your own ingress instead — for helix-production, the ingress chart in helix-infra-charts renders one zone-pinned Traefik per region (DaemonSet, nodeSelector on the region zone) plus a per-region LoadBalancer service with externalTrafficPolicy: Local. Both Traefik instances share one IngressClass, so a single Ingress is served by both regions — region-local by DNS, and by the other region on failover.

Important

Use the standard traefik IngressClass for the replacement, not a custom name. The kube-hetzner bootstrap Ingresses (argocd, longhorn, the Traefik dashboard) and the cert-manager HTTP-01 solver all reference class traefik. A custom class leaves all of those 404 and ACME challenges unserved, and forces you to patch every Ingress and parameterize the solver — for no benefit, since there is only one ingress controller. Make the replacement a transparent drop-in: same class, no per-Ingress patching.

Turn off the built-in Traefik so it does not waste a load-balancer slot:

ingress_controller = "none"

Warning

A Hetzner project has a load-balancer quota (3 on a fresh project). The control-plane load balancer plus two region ingress load balancers already fill it; the built-in Traefik load balancer is the fourth and blocks one region with load balancer limit exceeded. The module also sets enable_delete_protection.load_balancer = true, so the CCM cannot delete the redundant load balancer on its own — disable protection first: hcloud load-balancer disable-protection <lb> delete then hcloud load-balancer delete <lb>.

Verify each region load balancer has only its own region’s workers as healthy targets, and that both answer (a 404 from Traefik confirms the path end to end):

hcloud load-balancer list
curl -s -o /dev/null -w '%{http_code}\n' http://<region-lb-ipv4>/

Step 10 — DNS with a load-balancer indirection

Point an indirection record at both ingress load balancers, then CNAME everything onto it. Future load-balancer changes become a one-record edit:

lb.<domain>.            A      <fsn1-lb-ipv4>
lb.<domain>.            A      <hel1-lb-ipv4>
lb.<domain>.            AAAA   <fsn1-lb-ipv6>
lb.<domain>.            AAAA   <hel1-lb-ipv6>
*.cloud.<app-domain>.   CNAME  lb.<domain>.
*.ops.<domain>.         CNAME  lb.<domain>.
api.<domain>.           A      <control-plane-lb-ipv4>
api.<domain>.           AAAA   <control-plane-lb-ipv6>

Important

api.<domain> points at the control-plane load balancer (the kube-API endpoint), not at the ingress load balancers. A DNS wildcard matches only one label, so *.<domain> does not cover x.ops.<domain> — each sub-level needs its own record. Do not blanket-CNAME an existing wildcard (for example *.<app-domain>) onto the cluster if that name already serves a live site; use a dedicated sub-level such as *.cloud.<app-domain>.

Step 11 — Wire the ingress into ArgoCD

Add an ArgoCD Application for the ingress chart to the cluster’s GitOps repository, pointing at the charts repository path and branch.

If the charts repository is internal (not public), ArgoCD needs a credential for it. A per-repository deploy token works; a group token covers all the group’s repositories with one credential template. Create the repository secret out of band:

kubectl -n argocd create secret generic <repo>-repo --from-literal=type=git \
  --from-literal=url=<charts-repo-url> --from-literal=username=oauth2 --from-literal=password=<token>
kubectl -n argocd label secret <repo>-repo argocd.argoproj.io/secret-type=repository

The application should reach Synced and Healthy.

Troubleshooting

Stale pod IPs after a Cilium routing-mode change

Pods that kept running through a Cilium routing-mode change (for example VXLAN to native) keep their old pod IPs from the previous IPAM pool (for example 10.0.x instead of the cluster pool 10.42.x). Those IPs are not routable in the new dataplane, so inbound traffic to them fails — conversion webhooks time out, and the ArgoCD cluster cache fails with failed to load initial state ... conversion webhook ... request canceled, which leaves every application Unknown.

Find and restart the affected pods so their controllers recreate them with correct IPs:

kubectl get pods -A -o wide --no-headers | awk '$7 !~ /^10\.42\./ && $7 ~ /^10\./ {print "-n", $1, $2}'

Important

After any Cilium routing-mode change, roll all workload pods, not just the Cilium DaemonSet. Only restarting Cilium leaves long-lived workload pods on orphaned IPs.

ArgoCD application stuck Unknown

Unknown is a cache or repository problem, not a workload problem. Read the application conditions to tell them apart:

kubectl -n argocd get application <app> -o jsonpath='{range .status.conditions[*]}{.type}: {.message}{"\n"}{end}'

A conversion-webhook timeout points at stale pod IPs; an authentication required or Access denied points at a missing repository credential (see Step 11).

TLS certificate stuck Ready=False

If an ops dashboard serves a TRAEFIK DEFAULT CERT (self-signed) and its Certificate stays Ready=False with Issuing certificate as Secret does not exist, the ClusterIssuer it references probably does not exist:

kubectl get clusterissuers

The letsencrypt-cluster-issuer definitions ship in the 20-A-cert-manager manifest, but they fail to apply at first bootstrap — the cert-manager ClusterIssuer CRD is not yet registered when the manifest applies (the same CRD-ordering race as the Crossplane provider). Re-apply the manifest once the CRDs exist (from a control plane, where the rendered copy lives):

kubectl apply -f /var/user_kustomize/20-A-cert-manager.yaml

cert-manager then processes the pending CertificateRequest, the HTTP-01 solver (class traefik) is served by the replacement Traefik on port 80, and a real Let’s Encrypt certificate is issued.