How the migration from groot worked

This page records the July 2026 migration of Plausible from the Docker Swarm host groot to the kup6s cluster, as a recipe for similar migrations. The spec and step-by-step plan live in the umbrella repository under docs/superpowers/specs/2026-07-06-plausible-migration-design.md and docs/superpowers/plans/2026-07-06-plausible-migration.md.

Key facts

  • Source: Plausible plausible/analytics:latest (image from 2023-07, effectively v2.0.0), ClickHouse 22.6, PostgreSQL 14, Swarm stack plausible.

  • Target: Plausible CE v3.2.1, ClickHouse 24.12, CNPG PG 18, deployed via dp-infra/plausible/.

  • The “55 GB ClickHouse data” on groot was 53 GiB of untended system log tables; the actual analytics data was about 60 MiB on disk.

Sequence

  1. Truncate the ClickHouse system log tables on the source and set TTLs, freeing the disk.

  2. Take a full backup (pg_dump plus a stop-copy tar of the data directories).

  3. Upgrade in place on the source, one major at a time, per the CE release notes: v2.0.0 → v2.1.5 (ClickHouse 22.6 → 24.3.3.102, new TOTP_VAULT_KEY, drop db init-admin from the start command) → v3.0.1 (ClickHouse → 24.12, CLICKHOUSE_SKIP_USER_SETUP=1) → v3.2.1.

  4. Deploy the empty target on a temporary hostname and verify health, certificates, and backups.

  5. Stop the source app, copy PostgreSQL with pg_dump/pg_restore and ClickHouse tables as FORMAT Native dumps into the same app version.

  6. Cut over DNS with an explicit record that overrides the wildcard, flip BASE_URL, and verify certificate, live traffic, and GeoIP.

Lessons learned

  • Same-version copy is the safe path: upgrade first, then move data 1:1.

  • SECRET_KEY_BASE and TOTP_VAULT_KEY must travel with the data; PostgreSQL holds fields encrypted with them.

  • Physical row counts of VersionedCollapsingMergeTree and SummingMergeTree tables differ after re-insert because merges collapse rows; verify with count() ... FINAL and sum(value) instead.

  • Skip migration-seeded tables (location_data, acquisition_channel_*, *_dict) and migration leftovers (sessions_v2_tmp_versioned) when copying; the target’s own migrations create identical content.

  • Swarm on the source rejected freshly resolved tag@digest image references; a manual docker pull on the node fixed it, so pre-pull images before docker service update.

  • Plausible forces the first-launch registration page while the user table is empty, regardless of DISABLE_REGISTRATION; do not leave an empty instance publicly reachable for long.

Decommissioning

The groot stack was stopped (scaled to 0) at cutover and kept as a rollback reserve. Rollback: delete the explicit DNS record (the wildcard then routes to groot again) and scale the three services back up. After a 14-day stability window the stack and /data/plausible are to be deleted; see the project memory for the gate date.