Troubleshooting Solidtime¶

This guide lists problems hit during the initial deployment and their fixes. App logs go to stderr (LOG_CHANNEL=stderr), so check kubectl logs first.

kubectl logs -n solidtime -l app.kubernetes.io/component=http --tail=50
kubectl logs -n solidtime -l app.kubernetes.io/component=worker --tail=50

Worker crashloops with “WORKER_COMMAND is undefined”¶

The worker role execs $WORKER_COMMAND, so it must be set (php artisan queue:work --tries=3 --sleep=3 --max-time=3600). The http and scheduler roles need no command. Set WORKER_COMMAND on the worker deployment only.

http pods never become ready or restart repeatedly¶

Solidtime has no /health or /up route. Probe /login, which returns 200. Do not probe /, which 302-redirects to the absolute https:// URL from APP_URL; kubelet then follows the redirect over HTTPS against the plain-HTTP port 8000 and the probe fails with “server gave HTTP response to HTTPS client”.

Creating an API token fails with “Personal access client is not configured”¶

Solidtime uses Laravel Passport for API tokens. Issuing a personal-access token needs a personal-access client row in the oauth_clients table — the Passport keypair in solidtime-app-secrets is necessary but not sufficient. Solidtime’s self-host setup requires a one-time php artisan passport:client --personal (upstream Docker guide); the image entrypoint only runs migrate and optimize, so it never creates the client. Until it exists, the first API-token attempt throws Personal access client not found.

This deployment handles it automatically with the solidtime-ensure-passport-client ArgoCD PostSync hook Job, which creates the client only when none exists (idempotent, survives a DB rebuild). If API-token creation still fails:

# Is a personal-access client present?
kubectl exec -n solidtime deploy/solidtime-scheduler -c solidtime -- \
  php /var/www/html/artisan tinker --execute="echo \Laravel\Passport\Client::where('grant_types','like','%personal_access%')->count();"
# 0 -> create it manually (one-shot):
kubectl exec -n solidtime deploy/solidtime-scheduler -c solidtime -- \
  php /var/www/html/artisan passport:client --personal --name="API" --no-interaction

Never run the manual create more than once without checking the count first — each run adds another client.

The edge returns HTTP 404 while pods are healthy¶

The Ingress referenced a Traefik middleware that does not exist on this cluster (default-redirect-https@kubernetescrd; only www-redirect exists). A missing-middleware reference makes Traefik drop the router, which returns 404. Do not set a traefik.ingress.kubernetes.io/router.middlewares annotation unless the middleware exists; the entrypoint already redirects HTTP to HTTPS.

The Clockify import fails with “Missing required client configuration options”¶

The app could not resolve the S3 disk. Solidtime reads S3_* environment variables (S3_REGION, S3_BUCKET, S3_ENDPOINT, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_USE_PATH_STYLE_ENDPOINT), not AWS_*. With AWS_* set, the region, bucket, and endpoint resolve empty and the AWS SDK throws. Verify the resolved config inside a pod:

POD=$(kubectl get pod -n solidtime -l app.kubernetes.io/component=http -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n solidtime "$POD" -c solidtime -- php artisan tinker --execute="echo config('filesystems.disks.s3.region').' '.config('filesystems.disks.s3.bucket');"

The Clockify import fails with “Invalid CSV header, missing field”¶

The CSV is the wrong export or the wrong format. The “Clockify Time Entries” importer requires 13 columns including Billable, and the US date format. See How to migrate from Clockify.

A config.yaml change did not take effect¶

Values that land in the solidtime-env ConfigMap (for example APP_ENABLE_REGISTRATION) are read through envFrom only at pod start. ArgoCD updates the ConfigMap but does not restart the pods. Force a restart:

kubectl rollout restart deploy/solidtime-http deploy/solidtime-scheduler deploy/solidtime-worker -n solidtime

A change to the pod template (environment variable names or image tag) triggers a rollout automatically.

Database backups or WAL archiving look stalled¶

The firstRecoverabilityPoint field updates lazily, so use the cluster conditions as the authoritative signal.

kubectl get cluster.postgresql.cnpg.io solidtime-postgres -n solidtime -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'

Expect ContinuousArchiving=True and LastBackupSucceeded=True. Each postgres pod must be 2/2 (postgres plus the plugin-barman-cloud sidecar). If a pod is 1/1, the sidecar was not injected; recreate the pod, replica first. See ../cnpg/index.