How-to

Roll out Phase 2 block-mode

This guide shows you how to activate the CrowdSec bouncer block-mode for kup6s.

Prerequisites

You need three things before starting.

Phase 1 (Detect-Only) must be live and stable. Verify with:

kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli bouncers list

The source secret application-secrets/crowdsec-bouncer-key must exist. Verify with:

kubectl get secret -n application-secrets crowdsec-bouncer-key

You must have a backup kubectl path from a second network — mobile tethering, a different ISP, an SSH hop via an external server, or the Hetzner Cloud Console. See troubleshooting/self-rescue-lockout for why this is required.

Step 1: Narrow ClientTrustedIPs

Edit dp-infra/crowdsec/charts/constructs/middlewares.ts. Find the ClientTrustedIPs value in the crowdsec-bouncer Middleware and change it from the Phase 1 wildcard to the RFC1918 ranges:

ClientTrustedIPs: '127.0.0.1/32,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16'

Apply the same change to the crowdsec-appsec Middleware for consistency, even though AppSec is not yet attached to routes.

Re-synth, commit, push, MR, merge in the dp-infra subrepo following the standard workflow in dp-infra/CLAUDE.md. The plugin reads the new spec within 60 seconds (next stream poll).

Step 2: Verify immediately after activation

Check that the Middleware CRD reflects the new value:

kubectl get middleware -n crowdsec crowdsec-bouncer \
  -o jsonpath='{.spec.plugin.crowdsec-bouncer-traefik-plugin.ClientTrustedIPs}'

Expected output: 127.0.0.1/32,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16

Confirm the plugin is still pulling decisions:

kubectl exec -n crowdsec deploy/crowdsec-lapi -- cscli bouncers list -o json \
  | jq '.[] | select(.last_pull != null) | {name, last_pull}'

The last_pull timestamp should be within the last 60 seconds for each active Traefik pod.

Hit your critical production routes to confirm they respond:

curl -sI -m 5 https://argocd.ops.kup6s.net/ | head -1
curl -sI -m 5 https://grafana.ops.kup6s.net/ | head -1

Both should return HTTP 200 or 302.

Warning

If any of these return 4xx (other than your expected app-level 401/403) or 5xx, execute the rollback immediately — see “Rollback” below.

Step 3: Observe for three days

For each of the next three days, run these daily checks.

Pod health:

kubectl get pods -n crowdsec -o wide

Decision count stability:

kubectl exec -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  promtool query instant http://localhost:9090 'sum(cs_active_decisions)'

Production routes:

for url in argocd.ops.kup6s.net grafana.ops.kup6s.net longhorn.ops.kup6s.net; do
  echo "$url $(curl -sI -m 5 https://$url/ | head -1)"
done

Open the Grafana dashboard “CrowdSec Overview” and watch for sudden spikes or drops in cs_active_decisions.

If anyone reports that they cannot reach a kup6s site, see troubleshooting/self-rescue-lockout and troubleshooting/false-positive-handling.

Step 4: After three clean days, enroll

Once observation passes cleanly, proceed to enroll-community-console.

Step 5: After enrollment, install blocklists

Proceed to install-blocklists.

Rollback

If Phase 2 causes immediate issues and you need to revert quickly, edit middlewares.ts to restore the Phase 1 value:

ClientTrustedIPs: '0.0.0.0/0'

Re-synth, commit, push, merge. The plugin reverts to Detect-Only within 60 seconds.

For instant rollback without an MR cycle, patch the live Middleware CRD directly:

kubectl patch middleware -n crowdsec crowdsec-bouncer --type=json -p='[
  {"op": "replace",
   "path": "/spec/plugin/crowdsec-bouncer-traefik-plugin/ClientTrustedIPs",
   "value": "0.0.0.0/0"}
]'

Take care: ArgoCD will revert this patch on the next sync unless you also update the source code. Use the live patch as the immediate stop-gap and follow with the code revert.