Pods on cloud agents cannot reach the API server (missing hcloud network routes)

Symptom

  • A pod-networked API client (e.g. kured) on a cloud agent node crashloops with dial tcp 10.43.0.1:443: i/o timeout or context deadline exceeded.

  • Direct curl from a pod on that node to any control-plane node IP (https://10.25x.0.101:6443) times out, while pod→pod (even cross-node) and pod→internet work.

  • Variant: the aggregated metrics API flaps (kubectl top fails intermittently) after a control plane node was replaced — the API server cannot reach pods (metrics-server) on the new node.

  • hostNetwork pods on the same node are unaffected (they do not use the pod-source path).

Root cause

Cilium runs in native routing mode with WireGuard node encryption (encrypt-node=true), but it intentionally excludes API-server nodes from node-to-node encryption (bootstrap chicken-and-egg; visible as encryptkey=0 for the control-plane node IPs in cilium-dbg bpf ipcache list, identity 7). Pod→control-plane-host traffic therefore leaves the node unencrypted with the pod IP as source and relies on the Hetzner cloud network to route it — which requires a route entry per pod CIDR (10.42.X.0/24 → node IP).

Those routes were historically created by the hcloud cloud-controller-manager, but since the Robot/Helm CCM setup it runs with HCLOUD_NETWORK_ROUTES_ENABLED=false: existing routes are frozen leftovers, and new or replaced cloud nodes never get a route (and stale routes keep pointing at deleted gateways, blackholing traffic).

Dedicated (Robot) workers are not affected: their raw pod-source traffic reaches the control planes over the vSwitch L2 coupling.

Diagnosis

# 1. Compare pod CIDRs with the cloud network routes
kubectl get nodes -o custom-columns='NAME:.metadata.name,IP:.status.addresses[?(@.type=="InternalIP")].address,PODCIDR:.spec.podCIDR'
source kube-hetzner/clusters/<cluster>/.env
hcloud network describe <network> -o json | jq -r '.routes[] | "\(.destination) -> \(.gateway)"'

# 2. Every CLOUD node's pod CIDR needs a route to its InternalIP.
#    Robot workers (10.6.x) need none. Watch for routes whose gateway is no longer a node.

# 3. Confirm the encryption exclusion (optional)
kubectl exec -n kube-system <cilium-pod> -c cilium-agent -- cilium-dbg bpf ipcache list | grep <cp-node-ip>
# -> identity=7 encryptkey=0 = unencrypted raw path, routes required

Fix

Add or correct the route for each affected cloud node:

hcloud network remove-route <network> --destination 10.42.X.0/24 --gateway <stale-gw>   # only if stale
hcloud network add-route    <network> --destination 10.42.X.0/24 --gateway <node-internal-ip>

The effect is immediate (no pod or agent restart needed); restart a crashlooping consumer (e.g. the kured pod) to clear its backoff.

Prevention

Route reconciliation is disabled (HCLOUD_NETWORK_ROUTES_ENABLED=false in the CCM Helm values), so every cloud node replacement re-creates this problem. The durable fix — re-enabling CCM route reconciliation and validating it against Robot nodes — is tracked in plans_epics/backlog/hcloud-ccm-route-reconciliation.md. Until then: after replacing any cloud node, check and fix the routes as above (incident references: helix-prod witness kured crashloop + kup6s metrics-server flapping, 2026-07-07).