How-To Guide

Set Up and Operate the Headscale VPN

Type: How-To (Task-oriented)

Related: Setup Environment | Troubleshooting

Goal: Operate the self-hosted WireGuard mesh VPN (Headscale) on vpn.kup.tirol and onboard new devices.

Time: ~10 minutes per device

Prerequisites

  • SSH access as root to vpn.kup.tirol (Hetzner project vpn, hcloud context kupvpn)

  • Repo checkout with the config sources in vpn-headscale/

  • For provisioning history and exact resource IDs, see vpn-headscale/PROVISIONING.md in the repo root

Architecture in one paragraph

Headscale (native .deb, systemd) on a dedicated Hetzner VM is the coordination server for official Tailscale clients. The embedded DERP relay (TCP 443) and STUN (UDP 3478) solve NAT traversal for peers without static IPs. Traffic between devices is end-to-end WireGuard; the server only distributes keys, routes, and ACLs. MagicDNS names use the suffix tail.kup.tirol. Design and phases: docs/superpowers/specs/2026-06-08-headscale-vpn-design.md.

Check service health

curl https://vpn.kup.tirol/health
ssh root@vpn.kup.tirol 'systemctl status headscale --no-pager'
ssh root@vpn.kup.tirol 'journalctl -u headscale -f'

Expect HTTP 200 and active (running).

Onboard a Linux or macOS device

Create a user once per person, then a short-lived pre-auth key:

ssh root@vpn.kup.tirol 'headscale users create <name>'
ssh root@vpn.kup.tirol 'headscale users list'
ssh root@vpn.kup.tirol 'headscale preauthkeys create --user <ID> --expiration 1h'

Important

headscale preauthkeys create --user requires the numeric user ID. headscale auth register --user requires the user name. This CLI inconsistency exists in Headscale 0.29.

Join the device with the official Tailscale client:

sudo tailscale up --login-server https://vpn.kup.tirol --authkey <key> --hostname <device-name>

Verify the registration:

ssh root@vpn.kup.tirol 'headscale nodes list'
tailscale status

Onboard a mobile device

  1. Install the Tailscale app.

  2. Open the gear icon, select Accounts, then the three-dot menu, then Use an alternate server.

  3. Enter https://vpn.kup.tirol and sign in.

  4. The browser shows a command containing an hskey-authreq-… value.

  5. Register the device on the server:

ssh root@vpn.kup.tirol 'headscale auth register --auth-id <hskey-authreq-...> --user <name>'

Verify mesh connectivity

tailscale ping <other-device-ip-or-name>

direct <ip:port> means peer-to-peer WireGuard works. via DERP(headscale) means traffic uses the relay on the VM; this is the expected fallback behind strict NAT. MagicDNS check:

ping <device-name>.tail.kup.tirol

Change configuration or ACLs

Always edit the files in the repo (vpn-headscale/config/), never only on the VM. Validate the policy before activating it:

scp vpn-headscale/config/acl.hujson root@vpn.kup.tirol:/tmp/acl-candidate.hujson
ssh root@vpn.kup.tirol 'headscale policy check --file /tmp/acl-candidate.hujson'
ssh root@vpn.kup.tirol 'install -m 640 -o headscale -g headscale /tmp/acl-candidate.hujson /etc/headscale/acl.hujson && systemctl restart headscale'

In policy v2, user references require an @ suffix, for example jens@.

Update Headscale

OS security updates install automatically via unattended-upgrades (reboot window 04:30 UTC). The systemd timer headscale-autopatch checks daily around 04:00 UTC:

  • Patch releases (same minor) install automatically with a database backup and health-check rollback.

  • Minor and major releases only trigger a notification mail to jk@kleinundpartner.at.

For a manual (minor/major) upgrade, read the release notes first, then:

ssh root@vpn.kup.tirol 'cp -a /var/lib/headscale/db.sqlite /var/lib/headscale/db.sqlite.bak-manual'
HEADSCALE_VERSION=<x.y.z> ssh root@vpn.kup.tirol 'HEADSCALE_VERSION=<x.y.z> bash /root/install-headscale.sh'
ssh root@vpn.kup.tirol 'systemctl restart headscale && sleep 5 && curl -fsS https://vpn.kup.tirol/health'

Compare the config schema after minor upgrades:

ssh root@vpn.kup.tirol 'diff /etc/headscale/config.yaml.dist /etc/headscale/config.yaml'

Troubleshooting

TLS handshake errors with mkdir /var/www: read-only file system

The ACME cache directory is not writable. Ensure tls_letsencrypt_cache_dir: /var/lib/headscale/cache is set in config.yaml.

Certificate not issued

Let’s Encrypt HTTP-01 needs port 80 reachable from the internet. Check both the Hetzner firewall vpn-headscale-fw and ufw status on the VM.

Ping only works via DERP, never direct

Hole punching failed, which some NATs cause. This is functional but adds latency; traffic stays end-to-end encrypted.

Policy parse errors after an ACL change

Run headscale policy check --file <candidate> before activating. Headscale does not start with an invalid policy file.

Notification mails missing

Check /var/log/msmtp.log on the VM. The Mailjet credentials live only in /etc/msmtprc (mode 600) on the VM.