How-To Guide

Manage helix-prod Admin Access

Type: How-To (Task-oriented)

Related: Security Model | Manage Contexts with ktx

Goal: Issue, renew, onboard, and offboard personal cluster-admin access to helix-production.

Time: ~10 minutes to issue a certificate; ~2 minutes to renew one.

Why needed: helix-production has no identity provider and no jumphost. The Kubernetes API (https://api.helixpass.net:6443) is reachable from anywhere, so a personal, short-lived client certificate is the access control.

Before You Start

  • Each admin has a personal client certificate, not a shared kubeconfig.

  • Authorization comes from a single ClusterRoleBinding named helix-admins.

  • Its user subjects are listed in helix-infra-charts/admin-access/config.yaml — that file is the authoritative admin list, deployed via ArgoCD.

  • Certificates are valid for 90 days (kubernetes.io/kube-apiserver-client signer). Kubernetes cannot revoke a client certificate, so removing a name from config.yaml — not certificate expiry — is what cuts off access.

  • For the full rationale (why certificates, why user subjects instead of a group binding, why 90 days) and a detailed risk discussion, see admin-access/README.md in the helix-infra-charts repository.

Prerequisites

  • kubectl and openssl installed locally.

  • To issue a certificate (approver role): realpath, and an active kubectl context pointing at helix-prod. Verify with kubectl cluster-info — it must show https://api.helixpass.net:6443.

  • A checkout of helix-infra-charts/admin-access/scripts/admin-cert.sh.

Run every command below from that script’s directory:

cd helix-infra-charts/admin-access/scripts

Your key, CSR, certificate, and ca.crt are written to ~/.kube/helix-admin/ (directory mode 0700, key mode 0600); the kubeconfig context is merged into your default kubeconfig (~/.kube/config, mode 0600). Override the location with the ADMIN_CERT_DIR environment variable. Never copy the private key elsewhere, and never commit it to a repository.

Issue a New Certificate

The three steps below apply both to a brand-new admin and to a renewal that has already expired (see “Renew Your Certificate” below for the one-command self-renewal path).

Step 1: Request a Key and CSR (You)

./admin-cert.sh request phil

Expected output:

Dateien in /home/phil/.kube/helix-admin: phil.key (bleibt hier), phil.csr
OK: /home/phil/.kube/helix-admin/phil.csr an einen helix-Admin senden. Der Key bleibt auf DIESEM Rechner.

This creates phil.key and phil.csr (CN=phil, O=helix-admins) in ~/.kube/helix-admin/. Send the .csr file to an existing helix-prod admin over any channel — its content is not secret. The .key file never leaves your machine.

Step 2: Issue the Certificate (Approver)

The approver runs this with an active kubectl context on helix-prod:

./admin-cert.sh issue phil ~/Downloads/phil.csr

Expected output:

certificatesigningrequest.certificates.k8s.io/helix-admin-phil created
certificatesigningrequest.certificates.k8s.io/helix-admin-phil approved
subject=O=helix-admins, CN=phil
notAfter=Oct 13 2026 GMT
OK: /home/jens/.kube/helix-admin/phil.crt + /home/jens/.kube/helix-admin/ca.crt an phil zurücksenden (nicht geheim).

Before signing, issue verifies the CSR subject: CN must equal the name you passed and O must be exactly helix-admins, otherwise it aborts unsigned. This blocks a substituted CSR from being signed under a mismatched name or a privileged organization such as O=system:masters. On success it creates a CertificateSigningRequest with expirationSeconds: 7776000 (90 days), approves it, downloads the signed certificate to phil.crt, and copies the cluster CA to ca.crt. It deletes the CSR object again once the certificate is retrieved. Send phil.crt and ca.crt back to the requester — neither is secret.

Step 3: Build the Kubeconfig (You)

./admin-cert.sh kubeconfig phil

Expected output:

OK: Kontext 'helix-prod' in /home/phil/.kube/config gemergt — sofort nutzbar mit: ktx helix-prod

This expects phil.crt, phil.key, and ca.crt to already sit in ~/.kube/helix-admin/, and merges a helix-prod cluster, user, and context into your default kubeconfig (~/.kube/config, or the first entry of $KUBECONFIG), embedding the certificate, key, and CA. Existing contexts for other clusters are left untouched, and ktx sees helix-prod immediately — no manual merge step.

Verify Access

ktx helix-prod
kubectl auth whoami

Expected output:

ATTRIBUTE   VALUE
Username    phil
Groups      [helix-admins system:authenticated]

kubectl auth can-i '*' '*'

Expected output:

yes

If can-i returns no, your name is not yet (or no longer) a subject in config.yaml — see “Onboard a New Admin” below.

Renew Your Certificate

Renewal is one command, run entirely by you, no second person needed — as long as your current certificate is still valid. No RBAC change is needed either: the ClusterRoleBinding is untouched, only the certificate is replaced.

Switch to your helix-prod context first, then renew:

ktx helix-prod
./admin-cert.sh renew phil

This runs the same request and issue steps shown above (new key, new CSR, self-approved via your still-valid old certificate), then finishes with:

OK: User 'phil' im aktiven Kubeconfig in-place aktualisiert (Kontexte unverändert).

Your credentials are updated in place inside the active kubeconfig. The context and its ktx entry are untouched — no re-import needed.

kubectl auth whoami

Confirms phil / helix-admins, now backed by a fresh 90-day certificate.

Tip

renew only updates credentials in place when <name> matches the active context’s user, not just any matching entry in the file — this is what keeps a same-named user of a different cluster in a merged kubeconfig safe from being overwritten (which is why you run ktx helix-prod first). If the active context is something else, renew instead merges a fresh helix-prod context into your default kubeconfig. If you merge kubeconfigs from multiple clusters, give your helix-prod identity a name that is not reused as an active-context user elsewhere, so the in-place path always applies where you expect it.

Check Expiration

openssl x509 -in ~/.kube/helix-admin/phil.crt -noout -enddate

Expected output:

notAfter=Oct 13 2026 GMT

If the certificate has already expired, self-renewal is not possible — an expired certificate cannot authenticate the renewal request itself. Fall back to the three-step issuing process above, with another admin approving your CSR.

Onboard a New Admin

  1. Add the name to admins: in helix-infra-charts/admin-access/config.yaml.

  2. Run npm run build in helix-infra-charts/admin-access/.

  3. Commit and push to main — ArgoCD syncs the updated ClusterRoleBinding automatically.

  4. Have the new admin run request, send you the CSR, and have an existing admin run issue for them, as described in “Issue a New Certificate” above.

Offboard an Admin

  1. Remove the name from admins: in helix-infra-charts/admin-access/config.yaml.

  2. Run npm run build, commit, and push to main.

  3. ArgoCD syncs within about a minute — authorization is revoked immediately, even though the certificate itself is still technically valid.

Warning

Kubernetes cannot revoke a client certificate. A leaked or offboarded certificate keeps authenticating until it expires (up to 90 days) — removing the name from the ClusterRoleBinding subject list is what revokes authorization, and that takes effect immediately. On a suspected key leak, remove the subject right away and report the incident; do not wait for certificate expiry.

Limits

  • No certificate revocation in Kubernetes — see the warning above; the ClusterRoleBinding subject list is the only immediate lever.

  • Every admin can approve CSRs for any CN, inherent to holding cluster-admin — the trust model assumes all admins are equally trusted.

  • No API audit logging yet, and the API server is reachable from the public internet — both are tracked as follow-ups, not blockers.

  • Break-glass access, the tofu-generated admin kubeconfig, is unaffected by this workflow and held by Jens for emergencies.

For the full rationale and risk discussion, see admin-access/README.md in helix-infra-charts.