How-to · Troubleshooting

Recover AppSec from a machine registration conflict

This guide shows you how to recover the AppSec pod when its init container loops with HTTP 403 “user already exist” after a LAPI or cluster event.

Important

AppSec is currently DISABLED (since 2026-06-15). It is turned off via appsec.enabled: false in dp-infra/crowdsec/config.yaml because the AppSec Deployment had been stuck 0/1 for 41 days (the registration-conflict loop described below) while being unused — the Traefik bouncer runs with CrowdsecAppsecEnabled: false. A broken AppSec also leaves stale LAPI machine entries, which contributed to a cluster-wide WAF outage on 2026-06-15.

AppSec must be repaired and re-enabled. To bring it back: apply the recovery in this runbook (make the init registration idempotent or clear the stale machine), confirm a fresh AppSec pod reaches 1/1, then set appsec.enabled: true in dp-infra/crowdsec/config.yaml, run npm run build, and push so ArgoCD redeploys it. The stale-machine-pruner CronJob (its preventive helper) was repaired in the same change — its image moved from the retired bitnami/kubectl to rancher/kubectl:v1.34.8.

When this happens

The AppSec init container wait-for-lapi-and-register enters CrashLoopBackOff. This pattern reappears each time the LAPI engine restarts while the AppSec pod stays alive. Common triggers:

  • LAPI pod restart caused by Postgres failover

  • Cluster restructuring with node maintenance windows

  • Helm upgrade rollouts of the engine

The AppSec pod survives the disruption, but its init container restarts and re-runs cscli lapi register --machine <pod-name>. LAPI rejects the registration because the machine name (which equals the pod name) is still recorded from the previous successful registration.

This is a chart-level limitation — the init container’s register step is not idempotent.

Symptom check

Confirm the issue with these checks.

Pod status reports Unknown or 0/1 with restart loop:

kubectl get pods -n crowdsec

The init container logs the 403:

kubectl logs -n crowdsec <appsec-pod-name> -c wait-for-lapi-and-register --tail=5

Expected output includes:

level=fatal msg="api client register: api register (http://crowdsec-service.crowdsec:8080/) http 403 Forbidden: API error: user '<pod-name>': user already exist"

The machine entry still exists in LAPI:

kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli machines list | grep appsec

Recovery

Delete the stale machine entry and force-recreate the pod. The new pod gets a fresh name from the ReplicaSet, so its init container registers cleanly.

kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli machines delete <appsec-pod-name>
kubectl delete pod -n crowdsec <appsec-pod-name>

Verify recovery in 30 seconds:

kubectl get pods -n crowdsec

The new AppSec pod should be 1/1 Running.

Why not “just fix it permanently”

Three options exist; none is clean for our setup.

The upstream chart’s init container command is hardcoded as a one-liner:

cscli lapi register --machine "$USERNAME" -u "$LAPI_URL" --token "$REGISTRATION_TOKEN"

It does not check whether /tmp_config/local_api_credentials.yaml already exists. Making it idempotent would require either a chart fork or overriding the entire init container — both are heavier than the manual recovery, given how rarely this fires.

A second option is to add a periodic cron that runs cscli machines prune to expire stale machines. This treats the symptom rather than the cause and risks pruning legitimate machines during slow heartbeat windows.

A third option is to enable auto_registration to validate on a stable token without name uniqueness. The chart’s auto_registration block already accepts duplicate registrations via the same token, but LAPI’s machine table still enforces unique names; the chart does not delete the previous entry before registering.

For Phase 2, the manual recovery in this runbook is the accepted approach.

Preventive: clean up old machine entries periodically

Stale machine entries from previous pod incarnations accumulate over time. List them:

kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli machines list

Entries with Last Heartbeat showing many hours of staleness (⚠️ 50h32m) and a pod IP that no longer exists are safe to delete:

kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli machines delete <old-pod-name>

Run this periodically (monthly) or after any LAPI restart event to keep the machine list clean and prevent future name collisions.

Recurrence log

  • 2026-06-05 — initial occurrence after node restructuring (yim retired, EX nodes pinned)

  • 2026-06-06 — second occurrence after another LAPI restart event