How-To Guide

Troubleshoot Nodes Stuck After K3S Upgrade¶

Type: How-To (Task-oriented)

Learn how to diagnose and resolve nodes that remain cordoned (SchedulingDisabled) after K3S upgrades complete successfully.

Problem Description¶

Symptom: Node shows status Ready,SchedulingDisabled days or weeks after K3S upgrade completed successfully.

Example:

kubectl get nodes
NAME                         STATUS                     ROLES    AGE   VERSION
kup6s-agent-cax31-fsn1-yim   Ready,SchedulingDisabled   <none>   11d   v1.31.13+k3s1
#                            ^^^^^ Node is healthy but cannot schedule pods

Key differences from upgrade failures:

❌ Upgrade failures (covered in K3S Upgrade Failures): Upgrade job times out due to unhealthy pods blocking drain
✅ This issue: Upgrade completed successfully, but node was never uncordoned

Root Cause: System-Upgrade-Controller Bug¶

The K3S system-upgrade-controller has a known race condition:

Upgrade process cordons node → drains pods → upgrades K3S → should uncordon
Controller pod restarts during or after upgrade (normal operation)
Controller loses state and forgets which nodes need uncordoning
Node stuck in SchedulingDisabled state indefinitely

This is not a configuration error - it’s a timing-sensitive bug in the upstream controller.

Root Cause #2: kured Reboot-Lock Deadlock¶

A node can also get stuck cordoned without any SUC upgrade in progress — when kured (the MicroOS reboot daemon) tries to reboot it after a kernel/OS snapshot update:

kured acquires a single cluster-wide lock (concurrency 1), cordons the node, and drains it
The drain uses eviction, which respects PodDisruptionBudgets — and deadlocks on Longhorn instance-manager PDBs (disruptionsAllowed=0, longhorn #5688) and on CNPG *-postgres-* PDBs
kured retries the eviction forever → never reboots → never releases the lock
With the default --lock-ttl=0 the lock never expires, so one stuck node blocks the reboot of the entire fleet indefinitely (incident 2026-06-29: 3 nodes, 32h; orphan-lock Nov 2025–Jun 2026)

Symptom signature (distinguishes from the SUC race):

# kured holds a lock that never clears:
kubectl get ds kured -n kube-system -o jsonpath='{.metadata.annotations.weave\.works/kured-node-lock}'
# -> {"nodeID":"<node>", ... "TTL":0}  held for many hours

# the holder's kured log is stuck evicting PDB-protected pods:
kubectl logs -n kube-system <kured-pod-on-holder> | grep -i "violate the pod's disruption budget"

Recovery (kured deadlock)¶

# 1. Release the stale lock
kubectl annotate ds kured -n kube-system weave.works/kured-node-lock-

# 2. Restart the stuck kured pod on the holder node (drops the deadlocked drain loop)
kubectl delete pod -n kube-system <kured-pod-on-holder>

# 3. Uncordon the affected node(s)
kubectl uncordon <node>

This unjams the fleet. The node keeps its pending reboot; outside the reboot window (Sun 02:00–05:00 Europe/Berlin) kured will not re-attempt, giving time to land the durable fix.

Durable fix¶

kured_options in kube-hetzner/module/main.tf sets lock-ttl=30m (lock auto-expires → a stuck or dead holder can never jam the fleet) and drain-timeout=20m (drain gives up instead of retrying forever). --force-reboot is intentionally NOT set — rebooting before a graceful drain triggers the Longhorn iSCSI/XFS unmount hang on EX workers (see EX Worker Pings But Dead).

Why the drain deadlocks (and the standing fix)¶

longhorn-manager deletes the instance-manager PDB only once all volumes on the node are detached (canDeleteInstanceManagerPDB, verified against v1.12.0). The node-drain-policy: always-allow setting only skips the replica-safety checks, not this precondition. CNPG primaries are protected by their own PDB, so they cannot be evicted, their volumes stay attached, and the instance-manager PDB never goes away — kured’s drain runs into --drain-timeout every time. The in-window switchover CronJob (pre-drain-cnpg-switchover) could not help for two reasons: each drain retry re-evicts the CNPG replicas before they become ready (its “healthy replica” guard never passes), and until 2026-07-06 its unfiltered kubectl get pods --all-namespaces -o json query was OOM-killed by the 64Mi container limit before reaching the main path.

The standing fix is the sunday-reboot-prep CronJob (kube-system, Sun 01:30 Europe/Berlin, extra-manifests/89-sunday-reboot-prep.yaml.tpl). It queries each kured pod’s kured_reboot_required metric, picks the first reboot-pending node that hosts CNPG primaries, and switches those primaries to ready replicas on other nodes before the kured window opens. If both EX workers are pending, only one is prepped per Sunday; the other converges the following week (the “full” node’s drain fails after 20 minutes, then the prepped node drains clean). The ExRebootPendingTooLong alert (kured-alerts PrometheusRule, backed by the kured PodMonitor) fires when a node’s sentinel survives a full window.

Manual check after a Sunday window:

kubectl get jobs -n kube-system | grep sunday-reboot-prep
kubectl logs -n kube-system -l app.kubernetes.io/name=sunday-reboot-prep --tail=50
kubectl get nodes -L kured

Node stays cordoned after a successful reboot (window closed)¶

kured (≤1.22) does not run its post-reboot uncordon when the node comes back after the reboot window closed (--end-time=5am); the node then stays Ready,SchedulingDisabled with label kured=rebooting until the NEXT Sunday. Incident 2026-07-12: kup6s-ex-hel-1 finished rebooting at 05:02, stayed cordoned 30 h, half the workloads went Pending, and the capsule nodes.projectcapsule.dev webhook (failurePolicy Fail, controller itself Pending) blocked even a manual kubectl uncordon — a full admission deadlock.

Signature: node Ready,SchedulingDisabled, label kured=rebooting, kured lock annotation empty, fresh kernel/boot time within the last window.

Recovery (if the auto-uncordon safety net has not caught it yet):

# If capsule blocks node updates (no endpoints for capsule-webhook-service):
# temporarily set the nodes webhook to Ignore, revert to Fail afterwards.
kubectl patch validatingwebhookconfiguration capsule-validating-webhook-configuration \
  --type=json -p '[{"op":"replace","path":"/webhooks/<idx-of-nodes-webhook>/failurePolicy","value":"Ignore"}]'

kubectl uncordon <node>
kubectl label node <node> kured=done --overwrite

Prevention: the auto-uncordon CronJob (below) contains a kured-recovery block since 2026-07-13 — it uncordons nodes that are Ready, labeled kured=rebooting, and whose kured metric reports kured_reboot_required==0 (reboot done, only the uncordon was missed). It runs before the SUC version guards, so it also covers EX workers.

Automatic Solution: Auto-Uncordon CronJob¶

Since 2025-11-09, the cluster includes an automated solution that runs every 5 minutes.

How It Works¶

The auto-uncordon CronJob automatically detects and uncordons stuck nodes with these safety checks:

Safety checks (ALL must pass before uncordoning):

✅ Node is cordoned (spec.unschedulable == true)
✅ Node status is Ready
✅ Node version matches target version from upgrade plan
✅ No active upgrade jobs exist for the node
✅ Upgrade plan shows Complete status

Result: Stuck nodes are automatically uncordoned within 5 minutes with no manual intervention.

Verify Auto-Uncordon is Working¶

Check recent CronJob runs:

# List recent jobs
kubectl get jobs -n kube-system -l app.kubernetes.io/name=auto-uncordon \
  --sort-by=.metadata.creationTimestamp | tail -5

# View logs from most recent job
kubectl logs -n kube-system -l app.kubernetes.io/name=auto-uncordon --tail=100

Expected log output (when no stuck nodes):

[2025-11-09 12:37:00 UTC] Starting stuck node detection...
[2025-11-09 12:37:04 UTC] Target versions - Agent: v1.31.13-k3s1, Server: v1.31.13-k3s1
[2025-11-09 12:37:07 UTC] Upgrade plans Complete status - Agent: True, Server: True
[2025-11-09 12:37:08 UTC] INFO: No cordoned nodes found. Nothing to do.

When stuck node detected and uncordoned:

[2025-11-09 12:40:03 UTC] Found cordoned nodes: kup6s-agent-cax31-fsn1-yim
[2025-11-09 12:40:03 UTC] Checking node: kup6s-agent-cax31-fsn1-yim
[2025-11-09 12:40:05 UTC]   ✅ Node is Ready
[2025-11-09 12:40:06 UTC]   Node type: agent, target version: v1.31.13-k3s1
[2025-11-09 12:40:08 UTC]   ✅ Version matches target: v1.31.13+k3s1
[2025-11-09 12:40:10 UTC]   ✅ No active upgrade jobs
[2025-11-09 12:40:10 UTC]   ✅ ALL CHECKS PASSED - Ready to uncordon
[2025-11-09 12:40:10 UTC]   🔧 Uncordoning node kup6s-agent-cax31-fsn1-yim
node/kup6s-agent-cax31-fsn1-yim uncordoned
[2025-11-09 12:40:11 UTC]   ✅ Successfully uncordoned kup6s-agent-cax31-fsn1-yim
[2025-11-09 12:40:11 UTC] Complete. Uncordoned 1 node(s).

Check CronJob Status¶

# Verify CronJob is running
kubectl get cronjob auto-uncordon-stuck-nodes -n kube-system

# Expected output
NAME                        SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE
auto-uncordon-stuck-nodes   */5 * * * *   False     0        2m

Manual Diagnosis (If Auto-Uncordon Fails)¶

If a node remains stuck after 10+ minutes, manually diagnose:

1. Identify Cordoned Nodes¶

kubectl get nodes | grep SchedulingDisabled

2. Check Node Details¶

# Replace with your stuck node name
NODE_NAME="kup6s-agent-cax31-fsn1-yim"

# Check node version
kubectl get node $NODE_NAME -o jsonpath='{.status.nodeInfo.kubeletVersion}'

# Check if node is cordoned
kubectl get node $NODE_NAME -o jsonpath='{.spec.unschedulable}'
# Output: true (cordoned) or <empty> (schedulable)

3. Check Upgrade Plan Status¶

# Check agent upgrade plan
kubectl get plan k3s-agent -n system-upgrade -o yaml | grep -A 5 "status:"

# Check server upgrade plan
kubectl get plan k3s-server -n system-upgrade -o yaml | grep -A 5 "status:"

Look for:

latestVersion: Should match node version
conditions.type: Complete with status: "True"

4. Check for Active Upgrade Jobs¶

kubectl get jobs -n system-upgrade

Expected: No active jobs (all should be completed or failed)

If active jobs exist: Wait for them to complete before manually uncordoning

5. Review Auto-Uncordon Logs¶

# Get latest job logs
kubectl logs -n kube-system -l app.kubernetes.io/name=auto-uncordon --tail=100

Look for:

❌ Version mismatch: Node version doesn’t match plan target version
❌ Active jobs: Upgrade jobs still running for the node
❌ Plan incomplete: Upgrade plan not showing Complete: True
✅ All checks passed: Node should have been uncordoned

Manual Uncordon (If Needed)¶

If auto-uncordon hasn’t fixed it and all safety checks pass:

# Replace with your stuck node name
NODE_NAME="kup6s-agent-cax31-fsn1-yim"

# Verify node is ready
kubectl get node $NODE_NAME | grep Ready

# Verify no active upgrade jobs
kubectl get jobs -n system-upgrade | grep -i active || echo "No active jobs"

# Uncordon manually
kubectl uncordon $NODE_NAME

Verify:

kubectl get nodes $NODE_NAME
# Should show "Ready" (not "Ready,SchedulingDisabled")

Troubleshooting Auto-Uncordon CronJob¶

CronJob Not Running¶

Check CronJob exists:

kubectl get cronjob -n kube-system | grep auto-uncordon

If missing: Re-apply infrastructure manifests:

cd kube-hetzner
bash scripts/apply-and-configure-longhorn.sh

Check for suspended CronJob:

kubectl get cronjob auto-uncordon-stuck-nodes -n kube-system -o jsonpath='{.spec.suspend}'
# Output should be: false

If suspended: Resume the CronJob:

kubectl patch cronjob auto-uncordon-stuck-nodes -n kube-system \
  -p '{"spec":{"suspend":false}}'

CronJob Pods Failing¶

Check recent job failures:

kubectl get jobs -n kube-system -l app.kubernetes.io/name=auto-uncordon \
  --sort-by=.metadata.creationTimestamp | tail -10

Get failure logs:

# Get pod name from failed job
kubectl get pods -n kube-system -l app.kubernetes.io/name=auto-uncordon \
  --field-selector=status.phase=Failed

# View logs (replace POD_NAME)
kubectl logs -n kube-system POD_NAME

Common issues:

Image pull failures: Check cluster network and image availability
Permission errors: Verify RBAC permissions exist
API server errors: Check cluster health
Template-escape syntax error (observed 2026-06-29): if the deployed ConfigMap was re-applied from the .tpl without OpenTofu un-escaping, bash arithmetic like $((…)) may appear as $$((…)) — a hard bash syntax error. The script then only succeeds when there are no cordoned nodes (it exits before the for loop) and crashes exactly when a node needs uncordoning. Verify and fix:
```
# reproduce the parse error against the live script
kubectl get cm auto-uncordon-script -n kube-system -o jsonpath='{.data.auto-uncordon\.sh}' \
  | bash -n   # -> "syntax error near unexpected token '('"  means it is broken
```
In extra-manifests/85-auto-uncordon-cronjob.yaml.tpl, $(( needs a single $ (command-substitution $( is not OpenTofu interpolation); only ${…} brace-expansion needs the $${…} escape. Re-render via the apply script, or for a direct kubectl apply replace $$→$.

Re-enable Dry-Run Mode (For Testing)¶

If you suspect the CronJob is uncordoning incorrectly:

# Enable dry-run mode (logs only, no uncordoning)
kubectl set env cronjob/auto-uncordon-stuck-nodes -n kube-system DRY_RUN=true

# Wait for next scheduled run (within 5 minutes)
kubectl logs -n kube-system -l app.kubernetes.io/name=auto-uncordon --tail=100 -f

# Disable dry-run mode to resume automatic uncordoning
kubectl set env cronjob/auto-uncordon-stuck-nodes -n kube-system DRY_RUN=false

When NOT to Uncordon¶

Do NOT uncordon if:

❌ Node version doesn’t match cluster version (upgrade still in progress)
❌ Active upgrade jobs exist in system-upgrade namespace
❌ Node status is not Ready
❌ Node has been manually cordoned for maintenance (check annotations)

How to safely cordon for maintenance (without auto-uncordon interfering):

The auto-uncordon CronJob only uncordons nodes that:

Match the target K3S version
Have no active upgrade jobs
Are in Ready state

So manually cordoning a node for maintenance is safe - it won’t be auto-uncordoned unless it meets ALL criteria (which it won’t during active maintenance).

Prevention¶

The auto-uncordon CronJob prevents this issue automatically - no additional configuration needed.

Best practices:

✅ Keep auto-uncordon CronJob running (default)
✅ Monitor CronJob logs after K3S upgrades
✅ Verify all nodes are schedulable after upgrades complete

Check all nodes are schedulable:

# Should return empty (no cordoned nodes)
kubectl get nodes -o jsonpath='{.items[?(@.spec.unschedulable==true)].metadata.name}'

Technical Details¶

CronJob Configuration:

Manifest: kube-hetzner/extra-manifests/85-auto-uncordon-cronjob.yaml.tpl
Namespace: kube-system
Schedule: */5 * * * * (every 5 minutes)
Image: alpine/k8s:1.31.3 (Alpine Linux with kubectl and jq)
ServiceAccount: auto-uncordon (minimal RBAC permissions)

RBAC Permissions:

nodes: get, list, patch (read node status, uncordon)
plans.upgrade.cattle.io: get, list (read upgrade plan target versions)
jobs: get, list (check for active upgrade jobs)

Version Normalization: The script normalizes K3S version formats since:

Upgrade plans use: v1.31.13-k3s1 (hyphen)
Kubelet reports: v1.31.13+k3s1 (plus sign)

Both formats are valid and equivalent.