How-to · Troubleshooting

Recover from a self-lockout

This guide shows you how to recover when CrowdSec blocks your own access to kup6s sites.

When this happens

Your ISP IP rotates onto an address that CAPI has listed as a bad actor. This is uncommon but possible — the previous owner of that IP may have been part of a botnet. Phase 2 enables block-mode, so the plugin enforces the decision and returns HTTP 403.

Symptom check

Confirm the issue is a CrowdSec block, not a network problem.

Verify only kup6s sites fail. If any other public site also fails, the problem is your network, not CrowdSec.

Open https://argocd.ops.kup6s.net/ in your browser. A 403 with CrowdSec branding confirms the block. A timeout suggests a network problem instead — CrowdSec responds quickly. A 5xx is unrelated.

Confirm the IP appears in the decision list. From a machine on a different network with kubectl access:

kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli decisions list --ip <your-public-ip>

If the command returns one or more entries, the block is confirmed.

Required: backup access path

You need kubectl access from a network that is not using your blocked source IP.

Options:

  • Mobile tethering — your smartphone hotspot uses a different carrier IP

  • A second ISP (separate physical connection)

  • SSH hop through an external server, then kubectl from there

  • Hetzner Cloud Console — the browser-based VM console works from any network including Tor

The kubeconfig lives in kube-hetzner/clusters/kup6s/kup6s_kubeconfig.yaml. Keep a copy on your backup device.

Important

Set up and test the backup access path before activating Phase 2. Verify you can kubectl get pods -n crowdsec from your mobile hotspot or backup ISP.

Action 1: Immediate temporary recovery

Delete the offending decision. This takes effect within 60 seconds.

export KUBECONFIG=/path/to/kup6s_kubeconfig.yaml
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli decisions delete --ip <your-ip>

The site becomes reachable again.

This is a temporary fix. If the same IP appears in a blocklist with a high refresh frequency, the decision returns within hours. For a permanent fix, continue with Action 2.

Action 2: Permanent allowlist entry

Create a personal allowlist that survives CAPI refreshes.

kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli allowlists create personal \
  --description "Permanent allowlist - kup6s admin"

Add your current IP:

kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli allowlists add-item personal --value <your-ip>

Verify:

kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli allowlists list
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli allowlists inspect personal

Update the allowlist when your ISP IP rotates:

kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli allowlists remove-item personal --value <old-ip>
kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli allowlists add-item personal --value <new-ip>

Action 3: Emergency plugin bypass

If Action 1 and Action 2 do not work, or you cannot determine the cause, you can bypass the plugin entirely.

Option A: Restore the Phase 1 wildcard temporarily

Restore ClientTrustedIPs to its detect-only value via a direct CRD patch:

kubectl patch middleware -n crowdsec crowdsec-bouncer --type=json -p='[
  {"op": "replace",
   "path": "/spec/plugin/crowdsec-bouncer-traefik-plugin/ClientTrustedIPs",
   "value": "0.0.0.0/0"}
]'

The plugin stops blocking within 60 seconds.

ArgoCD will revert this patch on the next sync. For a longer fix, edit middlewares.ts in dp-infra/ and merge an MR.

Option B: Remove the default middleware from Traefik

Edit kube-hetzner/clusters/kup6s/traefik_overrides.yaml and comment out the entryPoints.websecure.http.middlewares=... argument. Run bash scripts/apply-and-configure-longhorn.sh kup6s. Traefik no longer attaches the plugin to incoming requests after the helm-upgrade completes (1-2 minutes).

Both options are reversible.

Final fallback: Hetzner Cloud Console

If every kubectl path is dead, log into the Hetzner Cloud Console at https://console.hetzner.cloud/ from any browser. Use the browser-based VM console to SSH into a cluster node and run kubectl from there. CrowdSec does not block Hetzner’s console infrastructure.

After the incident: post-mortem

Identify which decision blocked you:

kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli decisions list --ip <your-ip>

Note the origin field:

  • CAPI — generic community feed, low-action option

  • lists:crowdsecurity/<name> — premium blocklist, consider disabling if false-positives recur

  • cscli — manual addition (smoke test, operator action)

If a specific blocklist produces repeated false-positives on legitimate mobile-carrier or ISP pools, disable it. See install-blocklists.