Production recovery¶

Ordered by what users see. Each entry says how to confirm the diagnosis before acting; most "outages" in the September log were one container or one header, not the platform. Database recovery procedures (PITR, standby failover, logical restore, the monthly drill) are in the DR runbook and are not repeated here.

0. First minute, always¶

curl -sI https://bim.addmedad.top/ and curl -s https://bim.addmedad.top/api/health.
Dokploy → the compose service → container list: which of web, docs, api, worker, postgres, redis, minio is not running, and did a deploy just happen?
Read the last 200 lines of the failing container's log before restarting anything — a restart erases the evidence you need for the fix.

1. The site returns Traefik's plain-text `404 page not found` on every path¶

TLS is fine but nothing is routed: web (or, for doc.addmedad.top, docs) is not on dokploy-network. Confirm with any path (/api/health included) returning the same plain 404. Fix: the compose path in Dokploy must be docker-compose.dokploy.yml (which attaches web and docs to the network); redeploy. Do not "fix" it by adding ports. For doc.addmedad.top specifically, the domain must also exist in Dokploy (compose app → Domains: host doc.addmedad.top, service docs, port 80, HTTPS) and in DNS (an A record to the production host); without either, Traefik has nothing to route and answers this same 404.

A failed docs image build (mkdocs build --strict refuses a broken link or a nav entry with no file) fails the whole compose up --build: web and api stay on their previous containers and the next push is blocked until the documentation link is fixed. The build log names the file.

2. The SPA loads but every `/api/*` call is 502¶

nginx in web cached the old api container's IP after a deploy. Confirm: api container log shows it serving fine. Fix: restart web from Dokploy. If api itself is restarting, see 3.

3. `api` crash-looping¶

Read its log. Three usual causes:

Config validation — config.ts refuses to start with a missing/short secret (APP_SECRET must be at least 32 chars, missing DATABASE_URL): set the variable in Dokploy (and check the compose allowlist), redeploy.
Migration not completed — api waits for migrate; if migrate failed the api never starts. See Migrations.
Database unreachable — postgres unhealthy or password rotated without updating DATABASE_URL.

4. Uploads stay `queued`; clash runs never finish; exports hang¶

The worker, Redis, or their agreement on REDIS_KEY_PREFIX. Confirm: worker container log has no [worker] heartbeat / job lines; redis-cli ping inside the network. Fix: restart worker; if it crash-loops read its log (same config rules as the api; it also needs METRICS_PORT free). Jobs are persisted by pg-boss in Postgres — nothing queued is lost by a restart; in-flight jobs retry.

5. Models converted before 2026-09-08 take a minute to show¶

Not an outage: they have no tile manifest. Reconvert from the Models row menu, model by model.

6. Documents / models / photos download as 404 or the viewer gets no geometry¶

Object storage. Confirm: api log shows S3 errors; migrate's ensureBucket() failed on the last deploy. Fix: S3_* / STORAGE_DRIVER in Dokploy, bucket exists and the key can read/write it; redeploy. Nothing is deleted by a storage outage; uploads made during it fail visibly.

7. Emails not sent (invitations, scheduled reports, acknowledgements)¶

SMTP. Confirm: [mailer] SMTP not configured — skipped email to … (unset) or [report-schedule] delivery failed … mailer down (unreachable) in logs. Fix the SMTP_* variables. Nothing else depends on email: invitations have Copy link, transmittal acknowledgement links can be copied, in-app notifications keep working.

8. Copilot answers 503¶

Read the response body. "No copilot model is configured…" = no key reaches the container (Dokploy env + compose allowlist). A vendor error after ~20 s × vendors = every configured vendor failed; Check model status on the Copilot page shows which. See Copilot credentials.

9. Integrations: every push/pull errors¶

INTEGRATIONS_ENABLED kill switch is off (browse and disconnect still work — it is designed for a vendor incident). Turn it back on when the vendor recovers.
Vendor token expired → users Re-authorize.
config.baseUrl rejected on connect → the URL failed the SSRF check (must be https, public).

10. Map tab black, "Failed to fetch" in the browser console, nothing in server logs¶

The web CSP connect-src does not allow the basemap hosts (the compose/nginx conf was edited). apps/web/tests/structural/csp-tile-hosts.test.ts guards this; restore https://tile.openstreetmap.org https://server.arcgisonline.com in both nginx confs and redeploy.

11. A deploy broke the UI (error boundary "Something went wrong")¶

Roll back in Dokploy to the previous deployment (previous image) while the fix is prepared; the schema is additive so the previous code runs against the new schema. Then fix forward on BIMzone_Master_push — gate locally (typecheck, web suite, server suite alone) whenever CI is unavailable; a push to that branch is a production deploy.

12. Data damage (bad import, accidental purge, wrong migration)¶

Stop the writes (scale api/worker to zero in Dokploy), then PITR to just before the event, then bring the app back. Object storage is not rolled back by PITR: purged files are gone; uploaded files since the restore point remain orphaned and are harmless.

13. Redis lost (flushed or replaced)¶

Sessions are JWTs (unaffected). pg-boss state is in Postgres (unaffected). Redis carries the worker→browser realtime relay (job progress and completion events), multi-node Socket.IO fan-out and the shared SAML replay cache; a flush costs nothing durable — restart worker so it resubscribes. Without Redis the API still serves but every realtime event from a job is dropped (Cloud activity stops updating until a refresh).

After any incident¶

Write what happened, what you saw, and what fixed it into docs/PHASE-EXECUTION-LOG-2026-09.md.
If a user-visible defect was involved, add the symptom to Troubleshooting so the next person recognises it.
If the cause was a configuration that nothing tests, add a structural test (the repo's pattern: apps/*/tests/structural/*.test.ts reads the deployed artefact as text and asserts the rule).