How-To Guide

Add a Hetzner Dedicated (Robot) Server as a Worker Node

Type: How-To (Task-oriented)

Related: Apply Changes | Access kubectl

Goal: Join a Hetzner dedicated server (Robot, e.g. EX63/AX) to the cloud K3S cluster as a worker node — running openSUSE MicroOS, connected via vSwitch, recognised by the Cloud Controller Manager (CCM) so it is not deleted, and serving Longhorn storage across both NVMe disks.

First validated 2026-06-02 on kup6s-ex-hel-1 (EX63 #3003378, HEL). The cluster control plane stays on cloud nodes; dedicated servers are workers only.

Why this is non-trivial

A cloud + dedicated hybrid cluster has several traps. Each is solved below:

  • The CCM deletes any node “not in the cloud provider” → must enable Robot CCM + set a Robot providerID, and the Robot server NAME must equal the k8s node name (the CCM matches by name; a blank name → PossibleNodeDeletion).

  • MicroOS isn’t a Hetzner installimage → install via Self-Install image + Combustion.

  • growfs won’t expand root when the Longhorn partition is pre-created after it → root stays ~2 GB and fills up → manual btrfs filesystem resize max /var (Step 4).

  • Longhorn needs open-iscsi/nfs-client/kernel modules or it crashloops — and on MicroOS install it as its own transactional-update step + reboot (snapshots don’t stack).

  • NVMe device names swap across reboots → always work by serial / mount by UUID.

Prerequisites

  • Robot Webservice credentials (Robot → Settings → Webservice and app settings).

  • The server ordered; note its server number (from the order mail) and the MAC of its NIC.

  • Admin SSH: your personal key (Hetzner-registered, used for Rescue) + the fleet ~/.ssh/kup6s key (ops).

  • Template: kube-hetzner/baremetal/combustion-template.sh.

Step 1 — vSwitch + Robot server name (Robot UI)

Create a vSwitch (VLAN 4000), assign the server. Note the vSwitch ID. kube-hetzner manages the cloud-side subnet (do not create it by hand).

🔴 Set the Robot server NAME to the intended k8s node name (e.g. kup6s-ex-hel-2) — in the Robot UI or via the webservice API. With Robot enabled, the CCM matches dedicated nodes by name; a blank/mismatched name → PossibleNodeDeletion: Node name differs from Robot name "" and the node-lifecycle-controller may delete the node.

curl -u "$ROBOT_USER:$ROBOT_PASSWORD" -X POST \
  https://robot-ws.your-server.de/server/<SERVER_NUMBER> \
  --data-urlencode "server_name=<k8s-node-name>"

Step 2 — Enable Robot CCM via tofu (one-time per cluster)

In clusters/kup6s/.env (secrets):

export TF_VAR_robot_user="<robot-webservice-user>"
export TF_VAR_robot_password="<robot-webservice-password>"

In clusters/kup6s/terraform.tfvars:

hetzner_ccm_use_helm = true     # Helm CCM → robot.enabled + routes off (Robot LB-target mgmt)
robot_ccm_enabled    = true
vswitch_id           = 82213    # your vSwitch ID
vswitch_subnet_index = 6        # network_ipv4_subnets[6] = 10.6.0.0/16

These flow through module/ → the community module. hetzner_ccm_use_helm = true is required for Robot to actually work: the static-manifest CCM cannot run Robot (using Routes with Robot is not supported); only the Helm path renders robot.enabled and HCLOUD_NETWORK_ROUTES_ENABLED=false. Disabling CCM routes is safe here because cross-node pod traffic rides Cilium WireGuard (tunnels to node IPs, enable_wireguard=true), not CCM routes — and podCIDRs are allocated by k3s’ kube-controller-manager, not the CCM. Apply via the script:

bash scripts/apply-and-configure-longhorn.sh kup6s

With Robot enabled, the CCM auto-adds robot nodes as cloud-LB IP targets (no manual IP-targets). It does not reliably set the provided-by=robot node label, though — set that by hand at join time (Step 6).

Two one-time gotchas when first switching hetzner_ccm_use_helm false→true (already done on kup6s 2026-06-03; relevant only if re-bootstrapping):

  • Orphaned RBAC blocks the Helm install. The old static CCM left non-Helm ClusterRole/ClusterRoleBinding system:hcloud-cloud-controller-manager (+ :restricted); Helm refuses to import them (missing app.kubernetes.io/managed-by). Delete them (Helm recreates with proper ownership): kubectl delete clusterrole,clusterrolebinding -l '!app.kubernetes.io/managed-by' ... or by name, then let the helm-install-… job retry.

  • Serialize the apply. If a pending k3s-config change forces a node restart, run the script with TOFU_PARALLELISM=1 so control-planes restart one at a time (etcd quorum holds). Use SKIP_LONGHORN_CONFIG=1 to leave node storageReserved untouched.

Step 3 — Write MicroOS + partition BOTH disks (in Rescue, before first boot)

Activate Rescue (personal key) + reset. Get the NIC MAC (ip -br link, the iface with the public IP) — you need it for combustion. Then, in rescue:

cd /tmp
ASSET=openSUSE-MicroOS.x86_64-SelfInstall.raw.xz
wget https://download.opensuse.org/tumbleweed/appliances/$ASSET{,.sha256}
sha256sum -c $ASSET.sha256
# OS disk = nvme0n1, spare = nvme1n1 (verify with lsblk; names can differ).
xz -dc $ASSET | dd of=/dev/nvme0n1 bs=16M; sync; partprobe /dev/nvme0n1; sleep 2

The image is tiny (~2 GB) and leaves the rest of the disk unused. Partition now, before first boot — MicroOS growfs will NOT expand into pre-created space, so we size root explicitly and add the Longhorn partition:

# OS disk: fix GPT to use the full disk, grow root (p3) to 100 GiB, add p4 (xfs lh-os)
sgdisk -e /dev/nvme0n1                          # <-- ESSENTIAL: relocate backup GPT to end, else parted sees only ~2 GB
partprobe /dev/nvme0n1; sleep 1
parted -s /dev/nvme0n1 resizepart 3 100GiB      # GROW (no shrink warning) → -s works non-interactively
parted -s /dev/nvme0n1 mkpart longhorn 100GiB 100%
partprobe /dev/nvme0n1; sleep 2
mkfs.xfs -f -L lh-os /dev/nvme0n1p4

# Spare disk: a small "combustion" partition (for first-boot config) + the rest as lh-spare
parted -s /dev/nvme1n1 mklabel gpt
parted -s /dev/nvme1n1 mkpart combustion 1MiB 100MiB
parted -s /dev/nvme1n1 mkpart longhorn  100MiB 100%
partprobe /dev/nvme1n1; sleep 2
mkfs.ext4 -F -L combustion /dev/nvme1n1p1
mkfs.xfs  -f -L lh-spare   /dev/nvme1n1p2

Combustion config — fill kube-hetzner/baremetal/combustion-template.sh per host (HOSTNAME, MAC, PRIV_IP=10.6.0.<n>/24 with n ≥ 2, .1 is the gateway), then place it as /combustion/script on the combustion partition:

mount /dev/nvme1n1p1 /mnt && mkdir -p /mnt/combustion
cat > /mnt/combustion/script   # paste the per-host script (or scp it in)
chmod 755 /mnt/combustion/script && sync && umount /mnt

Template highlights (don’t change without reason): set -ux (NOT -e); sshd hardening via drop-in /etc/ssh/sshd_config.d/99-kup6s.conf; NIC matched by MAC (MicroOS names it enp130s0); public = DHCP, private VLAN = static, route 10.0.0.0/8 via 10.6.0.1, MTU 1400; echo "-F" > /etc/selinux/.autorelabel; installs open-iscsi+nfs-client; both admin SSH keys.

Reboot out of rescue (systemctl reboot; rescue is one-shot → next boot is from disk). First boot runs combustion + installs + SELinux-relabels + reboots into MicroOS (~1–8 min). Clear the stale host key locally (ssh-keygen -R <pub-ip>).

Step 4 — First boot verify + GROW the root filesystem 🔴

ssh root@<pub-ip> 'hostnamectl --static; getenforce; df -h / ; ip -br addr | grep 10.6.0'

Expect hostname set, Enforcing, VLAN 10.6.0.<n> up. But df -h / shows root at ~2 GB, 100 % full — growfs did not expand the btrfs (because p4 already occupies the space after p3). Writes will fail (k3s download etc.) until you grow it. Root is read-only, so resize via the rw /var mount of the same filesystem:

btrfs filesystem resize max /var      # NOT `/` (ro) — same fs, /var is rw → grows btrfs to the 100 GiB partition
df -h /                                # now ~100 G

Step 5 — Longhorn host prereqs (open-iscsi) → reboot

Combustion’s transactional-update install often fails (no network that early), so install now. Do this as its own step + reboot BEFORE the k3s install — two independent transactional-update runs don’t stack, so the k3s-selinux snapshot would otherwise drop these packages:

ssh root@<pub-ip> 'transactional-update --non-interactive pkg install -y open-iscsi nfs-client && systemctl reboot'
# wait for it back, confirm: command -v iscsiadm  → present

Step 6 — Join K3S with a Robot providerID

Transfer the node token without exposing it (relayed through your machine, only a byte count printed):

ssh root@<control-plane-pub-ip> 'cat /var/lib/rancher/k3s/server/node-token' \
 | ssh root@<server-pub-ip> 'umask 077; mkdir -p /etc/rancher; cat > /etc/rancher/agent-token; wc -c < /etc/rancher/agent-token'

Install the agent. Do NOT skip k3s-selinux (MicroOS is enforcing — the fleet runs k3s-selinux): the installer installs it via transactional-update (new snapshot) and enables the service; it starts after a reboot.

ssh root@<server-pub-ip> 'curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.34.6+k3s1 sh -s - agent \
    --server https://api.kup6s.net:6443 --token-file /etc/rancher/agent-token \
    --node-ip 10.6.0.<n> --node-external-ip <pub-ip> \
    --node-label topology.kubernetes.io/zone=ex-hel-<x> --node-label role=heavy \
    --kubelet-arg=provider-id=hrobot://<server-number> \
    --kubelet-arg=max-pods=250'
ssh root@<server-pub-ip> 'systemctl reboot'      # activates k3s-selinux snapshot → k3s-agent starts on boot
# after it's back:
ssh root@<server-pub-ip> 'systemctl enable --now iscsid; modprobe iscsi_tcp dm_crypt'

(If the k3s binary download fails with curl: (23) Failure writing output, root wasn’t grown — go back to Step 4.)

The --kubelet-arg=max-pods=250 raises kubelet’s default cap of 110, which is far too low for these 20-core / 64 GB nodes. With only two schedulable workers the 110 cap once let a cert-manager HTTP-01 solver storm exhaust the cluster-wide pod budget and freeze all scheduling. The /24 per-node pod CIDR allows up to ~250 addresses; the scheduler still gates real placement by CPU and memory requests. On an already-joined node the same effect is achieved by writing kubelet-arg: ["max-pods=250"] into /etc/rancher/k3s/config.yaml and restarting k3s-agent (running pods are not evicted).

Verify the node is Ready with providerID: hrobot://<number> and no PossibleNodeDeletion events (the Robot server name from Step 1 prevents them).

LB targeting is automatic: the Robot-enabled hcloud CCM adds the node’s private IP 10.6.0.<n> to kup6s-traefik on its own — verify with hcloud load-balancer describe kup6s-traefik | grep 10.6.0.

Step 7 — Longhorn disks

Mount both xfs disks by UUID (names swap across reboots):

LHOS=$(ssh root@<pub-ip> blkid -s UUID -o value /dev/nvme0n1p4)
LHSP=$(ssh root@<pub-ip> blkid -s UUID -o value /dev/nvme1n1p2)
ssh root@<pub-ip> "mkdir -p /var/lib/longhorn /var/lib/longhorn-disk2
  echo 'UUID=$LHOS /var/lib/longhorn       xfs defaults,noatime 0 2' >> /etc/fstab
  echo 'UUID=$LHSP /var/lib/longhorn-disk2 xfs defaults,noatime 0 2' >> /etc/fstab
  mount -a; df -hT /var/lib/longhorn /var/lib/longhorn-disk2"

Gotcha: longhorn-manager already created a default disk at /var/lib/longhorn on the root btrfs before you mounted the xfs over it → that disk now shows Ready=False, max=0 (UUID mismatch). On a fresh node (no replicas) replace it cleanly — first disable it (the webhook refuses to delete an enabled disk), then remove it and define both xfs disks:

N=<node>
kubectl -n longhorn-system patch nodes.longhorn.io $N --type merge \
  -p '{"spec":{"disks":{"<default-disk-id>":{"allowScheduling":false,"evictionRequested":true}}}}'
kubectl -n longhorn-system patch nodes.longhorn.io $N --type merge -p '{"spec":{"disks":{
  "<default-disk-id>": null,
  "lh-os":       {"path":"/var/lib/longhorn","allowScheduling":true,"evictionRequested":false,"diskType":"filesystem","storageReserved":0,"tags":[]},
  "spare-nvme2": {"path":"/var/lib/longhorn-disk2","allowScheduling":true,"evictionRequested":false,"diskType":"filesystem","storageReserved":0,"tags":[]}}}}'

Both should go Ready=True, Schedulable=True (~854 G + ~954 G). The zone comes from the node’s topology.kubernetes.io/zone label. Reservation is set to 7 % per disk — by scripts/apply-and-configure-longhorn.sh (default STORAGE_RESERVED_PERCENT=7) on the next run, or now:

for d in lh-os spare-nvme2; do
  max=$(kubectl -n longhorn-system get nodes.longhorn.io $N -o jsonpath="{.status.diskStatus.$d.storageMaximum}")
  kubectl -n longhorn-system patch nodes.longhorn.io $N --type merge \
    -p "{\"spec\":{\"disks\":{\"$d\":{\"storageReserved\":$(( max * 7 / 100 ))}}}}"
done

Verification

kubectl get node <node> -o wide                  # Ready, providerID hrobot://...
kubectl -n longhorn-system get nodes.longhorn.io <node>   # Ready, Schedulable, 2 disks
hcloud load-balancer describe kup6s-traefik | grep 10.6.0   # EX private IP auto-added as target (CCM)
# pod-to-pod both directions: cloud-node → a pod on the new node should ping (WireGuard, ~25 ms cross-DC)

Storage: Longhorn only

The Hetzner Cloud CSI driver is disabled cluster-wide (disable_hetzner_csi = true), so Robot workers run no hcloud-csi-node pod. All persistent storage on these nodes is Longhorn on the local NVMe disks (Step 7). open-iscsi (Step 5) is required for Longhorn, not for hcloud CSI.