Reference

Auto-Uncordon CronJob

Type: Reference (Information-oriented)

Related: Stuck Nodes After Upgrade | Extra Manifests

Technical reference for the automated node uncordon system that resolves stuck nodes after K3S upgrades.

Overview

The Auto-Uncordon CronJob is a cluster infrastructure component that automatically detects and uncordons nodes left in SchedulingDisabled state after K3S system-upgrade-controller completes upgrades.

Purpose: Mitigate a known race condition in the K3S system-upgrade-controller where controller pod restarts cause it to lose track of which nodes need uncordoning.

Deployed: Since 2025-11-09 as part of core infrastructure

Location: kube-hetzner/extra-manifests/85-auto-uncordon-cronjob.yaml.tpl

Component Specifications

CronJob

Property	Value	Description
Name	auto-uncordon-stuck-nodes	CronJob resource name
Namespace	kube-system	System namespace for cluster management tools
Schedule	*/5 * * * *	Every 5 minutes
Concurrency Policy	Forbid	Prevents overlapping job executions
Active Deadline	120s	Jobs timeout after 2 minutes
Backoff Limit	2	Retry failed jobs up to 2 times
Success History	3	Keep last 3 successful jobs for logs
Failure History	5	Keep last 5 failed jobs for debugging

Container

Property	Value	Description
Image	alpine/k8s:1.31.3	Alpine Linux with kubectl and jq
Command	/bin/sh /scripts/auto-uncordon.sh	Execute detection script
CPU Request	10m	Minimal CPU requirement
CPU Limit	100m	Maximum CPU allowed
Memory Request	32Mi	Minimal memory requirement
Memory Limit	64Mi	Maximum memory allowed

Environment Variables

Variable	Default	Description
DRY_RUN	false	Set to true to log actions without uncordoning
NAMESPACE	system-upgrade	K3S upgrade controller namespace (hardcoded in script)

RBAC Permissions

ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: auto-uncordon
  namespace: kube-system

ClusterRole Permissions

API Group	Resources	Verbs	Purpose
"" (core)	nodes	get, list, patch	Read node status and uncordon nodes
upgrade.cattle.io	plans	get, list	Read K3S upgrade plan target versions
batch	jobs	get, list	Check for active upgrade jobs

Security principle: Minimal permissions required for operation (least privilege).

Detection Algorithm

Safety Checks

The script performs 5 safety checks before uncordoning any node. ALL checks must pass:

1. Node is Cordoned

spec.unschedulable == true

Rationale: Only uncordon nodes that are actually cordoned. Prevents interference with schedulable nodes.

2. Node is Ready

status.conditions[type=="Ready"].status == "True"

Rationale: Ensure node is healthy before allowing pod scheduling. Prevents scheduling to unhealthy nodes.

3. Version Matches Target

# Node version (normalized)
status.nodeInfo.kubeletVersion (e.g., v1.31.13+k3s1)

# Matches upgrade plan version (normalized)
status.latestVersion from Plan resource (e.g., v1.31.13-k3s1)

Version normalization: K3S uses + in kubelet version but - in plan version. Both are equivalent:

• v1.31.13+k3s1 → normalize to → v1.31.13-k3s1

Rationale: Ensure upgrade completed successfully. Prevents uncordoning during active upgrades.

4. No Active Upgrade Jobs

# Check for jobs in system-upgrade namespace
# matching node name with status.active > 0
kubectl get jobs -n system-upgrade

Rationale: Don’t interfere with in-progress upgrades. Prevents race conditions.

5. Upgrade Plan Complete

# Check plan status conditions
status.conditions[type=="Complete"].status == "True"

Rationale: Verify upgrade cycle finished. Prevents premature uncordoning.

Node Type Detection

The script determines whether a node is control-plane or agent:

# Check for control-plane label
metadata.labels["node-role.kubernetes.io/control-plane"]

# If present:
#   - Use k3s-server plan latestVersion
#   - Node type: "control-plane"
# If absent:
#   - Use k3s-agent plan latestVersion
#   - Node type: "agent"

Execution Flow

1. Fetch upgrade plan target versions
   - k3s-agent plan → AGENT_TARGET_VERSION
   - k3s-server plan → SERVER_TARGET_VERSION

2. Check plan completion status
   - If neither plan shows Complete=True → Exit (no action needed)

3. Get all cordoned nodes
   - Query: spec.unschedulable == true
   - If none found → Exit (no action needed)

4. For each cordoned node:
   a. Check node Ready status
   b. Determine node type (control-plane vs agent)
   c. Get target version for node type
   d. Normalize and compare versions
   e. Check for active upgrade jobs
   f. If all checks pass → Uncordon node

5. Log summary
   - Count of nodes uncordoned
   - Success/failure for each node

Log Format

Successful Run (No Stuck Nodes)

[2025-11-09 12:37:00 UTC] Starting stuck node detection...
[2025-11-09 12:37:04 UTC] Target versions - Agent: v1.31.13-k3s1, Server: v1.31.13-k3s1
[2025-11-09 12:37:07 UTC] Upgrade plans Complete status - Agent: True, Server: True
[2025-11-09 12:37:08 UTC] INFO: No cordoned nodes found. Nothing to do.

Successful Uncordon

[2025-11-09 12:40:03 UTC] Found cordoned nodes: kup6s-agent-cax31-fsn1-yim
[2025-11-09 12:40:03 UTC] Checking node: kup6s-agent-cax31-fsn1-yim
[2025-11-09 12:40:05 UTC]   ✅ Node is Ready
[2025-11-09 12:40:06 UTC]   Node type: agent, target version: v1.31.13-k3s1
[2025-11-09 12:40:08 UTC]   ✅ Version matches target: v1.31.13+k3s1
[2025-11-09 12:40:10 UTC]   ✅ No active upgrade jobs
[2025-11-09 12:40:10 UTC]   ✅ ALL CHECKS PASSED - Ready to uncordon
[2025-11-09 12:40:10 UTC]   🔧 Uncordoning node kup6s-agent-cax31-fsn1-yim
node/kup6s-agent-cax31-fsn1-yim uncordoned
[2025-11-09 12:40:11 UTC]   ✅ Successfully uncordoned kup6s-agent-cax31-fsn1-yim
[2025-11-09 12:40:11 UTC] Complete. Uncordoned 1 node(s).

Skipped Due to Safety Check

[2025-11-09 12:38:03 UTC] Checking node: kup6s-agent-cax31-fsn1-yim
[2025-11-09 12:38:05 UTC]   ✅ Node is Ready
[2025-11-09 12:38:06 UTC]   Node type: agent, target version: v1.31.13-k3s1
[2025-11-09 12:38:08 UTC]   ❌ SKIP: Version mismatch (current: v1.31.12+k3s1, target: v1.31.13-k3s1)
[2025-11-09 12:38:08 UTC] Complete. Uncordoned 0 node(s).

Monitoring and Observability

Check CronJob Status

# Verify CronJob exists and is active
kubectl get cronjob auto-uncordon-stuck-nodes -n kube-system

# Check if CronJob is suspended
kubectl get cronjob auto-uncordon-stuck-nodes -n kube-system \
  -o jsonpath='{.spec.suspend}'
# Should return: false

View Recent Jobs

# List recent job executions
kubectl get jobs -n kube-system \
  -l app.kubernetes.io/name=auto-uncordon \
  --sort-by=.metadata.creationTimestamp | tail -10

# Get job completion status
kubectl get jobs -n kube-system \
  -l app.kubernetes.io/name=auto-uncordon \
  -o custom-columns=NAME:.metadata.name,COMPLETIONS:.status.completions,DURATION:.status.completionTime

Access Logs

# View logs from most recent job
kubectl logs -n kube-system \
  -l app.kubernetes.io/name=auto-uncordon \
  --tail=100

# Follow logs in real-time (wait for next scheduled run)
kubectl logs -n kube-system \
  -l app.kubernetes.io/name=auto-uncordon \
  --tail=100 -f

# View logs from specific job
kubectl logs -n kube-system job/auto-uncordon-stuck-nodes-28923456

Metrics (Future Enhancement)

Currently no Prometheus metrics exported. Monitoring relies on:

• Job completion status (visible in kubectl get jobs)

• Log output (visible in kubectl logs)

• Manual node status checks (kubectl get nodes)

Potential metrics (not implemented):

• auto_uncordon_nodes_detected_total - Count of stuck nodes detected

• auto_uncordon_nodes_uncordoned_total - Count of nodes successfully uncordoned

• auto_uncordon_nodes_skipped_total - Count of nodes skipped by safety checks

• auto_uncordon_last_run_timestamp - Timestamp of last successful run

Configuration Options

Enable/Disable Dry-Run Mode

Dry-run mode: Logs what would be done without actually uncordoning nodes.

# Enable dry-run (logs only, no uncordoning)
kubectl set env cronjob/auto-uncordon-stuck-nodes -n kube-system \
  DRY_RUN=true

# Disable dry-run (resume automatic uncordoning)
kubectl set env cronjob/auto-uncordon-stuck-nodes -n kube-system \
  DRY_RUN=false

Suspend/Resume CronJob

Temporary disable (e.g., during cluster maintenance):

# Suspend CronJob (stop all scheduled runs)
kubectl patch cronjob auto-uncordon-stuck-nodes -n kube-system \
  -p '{"spec":{"suspend":true}}'

# Resume CronJob
kubectl patch cronjob auto-uncordon-stuck-nodes -n kube-system \
  -p '{"spec":{"suspend":false}}'

Adjust Schedule

Change run frequency (requires manifest edit):

# Edit CronJob directly (temporary change, lost on reapply)
kubectl edit cronjob auto-uncordon-stuck-nodes -n kube-system
# Update spec.schedule: "*/5 * * * *"

# Permanent change: Edit manifest and reapply
# kube-hetzner/extra-manifests/85-auto-uncordon-cronjob.yaml.tpl

Schedule examples:

• */5 * * * * - Every 5 minutes (default)

• */10 * * * * - Every 10 minutes

• 0 * * * * - Every hour at minute 0

• 0 0 * * * - Daily at midnight

Deployment

Initial Deployment

Deployed automatically as part of infrastructure provisioning:

cd kube-hetzner
bash scripts/apply-and-configure-longhorn.sh

Manifest location: kube-hetzner/extra-manifests/85-auto-uncordon-cronjob.yaml.tpl

Manual Deployment

kubectl apply -f kube-hetzner/extra-manifests/85-auto-uncordon-cronjob.yaml.tpl

Note: For OpenTofu deployments, the .tpl file contains $$ for variable escaping. For direct kubectl apply, replace $$ with $.

Verification

# Check all components deployed
kubectl get sa,clusterrole,clusterrolebinding,configmap,cronjob \
  -n kube-system -l app.kubernetes.io/name=auto-uncordon

# Expected output:
# serviceaccount/auto-uncordon
# clusterrole.rbac.authorization.k8s.io/auto-uncordon
# clusterrolebinding.rbac.authorization.k8s.io/auto-uncordon
# configmap/auto-uncordon-script
# cronjob.batch/auto-uncordon-stuck-nodes

Upgrade

# Re-apply manifest to update
kubectl apply -f kube-hetzner/extra-manifests/85-auto-uncordon-cronjob.yaml.tpl

# Delete existing ConfigMap to force script update
kubectl delete configmap auto-uncordon-script -n kube-system
kubectl apply -f kube-hetzner/extra-manifests/85-auto-uncordon-cronjob.yaml.tpl

Troubleshooting

See Troubleshoot Nodes Stuck After Upgrade for detailed troubleshooting guide.

Quick checks:

# 1. Verify CronJob is running
kubectl get cronjob auto-uncordon-stuck-nodes -n kube-system

# 2. Check recent job status
kubectl get jobs -n kube-system -l app.kubernetes.io/name=auto-uncordon \
  --sort-by=.metadata.creationTimestamp | tail -5

# 3. View recent logs
kubectl logs -n kube-system -l app.kubernetes.io/name=auto-uncordon --tail=100

# 4. Check RBAC permissions exist
kubectl get clusterrole auto-uncordon
kubectl get clusterrolebinding auto-uncordon

Security Considerations

Minimal Permissions

The auto-uncordon ServiceAccount has only the permissions required for operation:

• ✅ Can read node status

• ✅ Can patch nodes (uncordon operation)

• ✅ Can read upgrade plans

• ✅ Can list jobs

• ❌ Cannot delete or modify other resources

• ❌ Cannot access secrets or sensitive data

No Manual Cordon Interference

The CronJob respects manually cordoned nodes during maintenance:

• Only uncordons nodes matching K3S upgrade target version

• Skips nodes with active upgrade jobs

• Won’t uncordon nodes that are truly unhealthy (not Ready)

Safe to manually cordon for maintenance - auto-uncordon won’t interfere.

Image Provenance

Container image: alpine/k8s:1.31.3

• Official Alpine Kubernetes tools image

• Contains kubectl, helm, and jq

• Minimal attack surface (Alpine Linux base)

• Version pinned for reproducibility

Implementation History

Date	Event	Description
2025-11-09	Initial deployment	CronJob deployed to resolve recurring stuck node issues
2025-11-09	First successful uncordon	Node kup6s-agent-cax31-fsn1-yim automatically uncordoned
2025-11-09	Documentation created	How-to guide and reference documentation added

Related Resources

• Stuck Nodes After Upgrade - How-to guide for using this component

• K3S Upgrade Failures - Different issue: upgrade timing out

• Extra Manifests - All infrastructure component manifests

• Apply Infrastructure Changes - How to deploy updates