How-To Guide

Recover an EX worker that pings but is dead¶

Type: How-To (Task-oriented)

This guide shows you how to recover a dedicated EX worker (kup6s-ex-hel-1, kup6s-ex-hel-2) that has hung during an ungraceful reboot.

Symptom¶

The node is NotReady, but it still answers ICMP ping. TCP port 22 is refused, and the kubelet and longhorn-manager are down. Longhorn then rejects every volume attach cluster-wide with HTTP 500 node ... is not ready, so stateful workloads across many namespaces go Pending or FailedAttach.

The cause is an unattended MicroOS reboot that ran without draining first. During shutdown the Longhorn iSCSI block devices were torn away while their XFS filesystems were still mounted, which produces a kernel storm (XFS ... Block device removal ... Shutting down filesystem, XFS log I/O error). The systemd shutdown then wedges on the unmount, the kernel stays up (so it still pings), but every service is gone and the final reboot never fires.

Diagnose¶

Confirm the signature before acting.

ping -c3 157.180.107.250

A reply with low packet loss means the kernel is alive.

nc -vz -w3 157.180.107.250 22

A refused connection on port 22 with a live ping confirms a shutdown hang rather than a hard-down server.

Recover with a hardware reset¶

The only lever is a Hetzner Robot hardware reset, because sshd is dead.

Find the server number from the node providerID.

kubectl get node kup6s-ex-hel-1 -o jsonpath='{.spec.providerID}'

The value has the form hrobot://<server-number>.

Trigger the reset with the Robot webservice credentials.

source kube-hetzner/clusters/kup6s/.env
curl -s -u "$TF_VAR_robot_user:$TF_VAR_robot_password" \
  https://robot-ws.your-server.de/reset/<server-number>
curl -s -u "$TF_VAR_robot_user:$TF_VAR_robot_password" -d type=hw \
  https://robot-ws.your-server.de/reset/<server-number>

The node returns in about two minutes, and Longhorn reattaches its volumes automatically.

Warning

Do not force-detach Longhorn volumes or delete PVCs while the node is in this state. Volumes with a single replica keep their only copy on the hung node’s disk, so a force operation destroys data. Wait for the node to return and let Longhorn reattach.

Read the post-mortem logs¶

journald is persistent on these nodes (/var/log/journal), so the crashed boot survives the reset.

ssh -i ~/.ssh/kup6s -o IdentitiesOnly=yes root@157.180.107.250 \
  'journalctl -b -1 -p err --no-pager | grep -iE "XFS|Block device removal|log I/O error"'

The -o IdentitiesOnly=yes flag is required, otherwise the agent offers too many keys and the connection fails. ex-hel-2 has no public address, so reach it over the vSwitch with a jump host.

ssh -i ~/.ssh/kup6s -o IdentitiesOnly=yes -o ProxyJump=root@157.180.107.250 root@10.6.0.3 \
  'journalctl -b -1 --no-pager -n 40'

Prevent a recurrence¶

The fix is to route reboots through kured instead of letting MicroOS reboot directly.

Set REBOOT_METHOD=kured in /etc/transactional-update.conf on every EX node, which the kube-hetzner/baremetal/combustion-template.sh template now does for new nodes. kured then cordons and drains the node first, so the Longhorn volumes detach cleanly before the reboot.

The ExWorkerNotReady alert fires after 15 minutes of NotReady, because a hung node also holds the cluster-wide kured lock and blocks all further graceful reboots until it is recovered.