Skip to content

Monitoring & health checks

What to look at, in what order, to know Production is healthy — and the thresholds at which it is not. Objectives and Prometheus alert rules live in operations/SLO.md; this page is the operator's checklist.

Endpoints

Check How Healthy looks like
Web up curl -sI https://bim.addmedad.top/ 200, HSTS and CSP headers present
Docs up curl -sI https://doc.addmedad.top/ 302 to /BIMZONE-USER-GUIDE/; the hub then 200
API up curl -s https://bim.addmedad.top/api/health {"ok":true,"ts":<epoch ms>}
API metrics GET /metrics on the api — private networks only (the route refuses public origins) Prometheus text: http_server_request_duration_*, job_queue_depth, job_execution_duration_*, db_pool_connections
Worker metrics GET http://worker:9091/metrics inside the compose network (METRICS_PORT=9091) same families for jobs; the worker answers /metrics only
Backups fresh GET /api/admin/backup-status as a platform admin (PLATFORM_ADMIN_EMAILS) {"lastBackupAt":"…","ageSeconds":N,"stale":false} — stale flips at 26 h
Compliance evidence GET /api/admin/compliance/evidence (platform admin) a JSON summary of controls in force
Postgres compose healthcheck pg_isready -U bimzone healthy
Redis compose healthcheck redis-cli ping PONG
MinIO / S3 migrate runs ensureBucket() at every deploy deploy log shows no storage error

Daily glance (2 minutes)

  1. /api/health returns ok.
  2. Dokploy: all app profile containers running (api, worker, web, docs, postgres, redis, minio); migrate exited 0.
  3. /api/admin/backup-status → stale:false (if the dr profile is enabled).
  4. In the app: Cloud activity shows no job stuck for hours; a project's Copilot › Check model status shows ok for the vendors you pay for.
  5. Worker log tail: no repeating [report-schedule] delivery failed, [convert-dwg] … DWG converter not installed, [workspace-purge] storage purge failed beyond what you expect.

What the numbers mean

  • job_queue_depth{queue} — pg-boss backlog per queue. SLO-3: no queue above 1,000 pending for 15 minutes. A draining-but-failing queue looks healthy on depth alone — read job_execution_duration_count{outcome="failure"} per queue as well.
  • http_server_request_duration_bucket — SLO-2: 95 % of requests under 500 ms over a 30-day window (BimZoneLatencyBudgetBurn in the SLO doc).
  • Availability — SLO-1: 99.5 % on /api/health plus successful 2xx/3xx share; the SLO doc carries multi-window burn-rate alerts (BimZoneErrorBudgetBurnFast / …Slow).
  • db_pool_connections{state} — idle near zero with waiting rising means pool exhaustion; look for a stuck long query before scaling.

Log lines worth alerting on

Line Meaning Action
Migration failed: (migrate) deploy stopped before the api started Migrations
copilot provider failed; trying the next one (api, repeated) a vendor is down or its key/credits are gone Copilot credentials
No copilot model is configured in responses every copilot key absent from the container compose allowlist / Dokploy env
[report-schedule] delivery failed for … : mailer down SMTP unreachable fix SMTP; the schedule stays active
[reconcile-uploads] … failed N incomplete uploads whose finalisation never came usually a client that disconnected; grows = storage problem
[workspace-purge] storage purge failed … row kept for retry object storage refused a delete check S3 credentials/permissions; it retries
[metrics-server] error on host:port — this worker may not be scrapeable port clash change METRICS_PORT
nginx 502 on /api/ after a deploy stale upstream IP restart web

Capacity signals

  • Model conversion is CPU/RAM bound on the worker; a 200–300 MB IFC takes minutes. Many concurrent uploads queue rather than fail.
  • Clash runs are capped per run (see the clash settings modal's cap report); capped runs are a product signal, not an outage.
  • Storage growth: models keep their IFC and converted geometry; photos are EXIF-stripped JPEGs; the 30-day document trash holds deleted files until purge.

Synthetic checks worth automating

  • GET /api/health every minute from outside (availability SLO).
  • GET /api/admin/backup-status hourly → alert on stale:true.
  • Once a day: upload a 1 MB IFC to a monitoring project and expect ready within 5 minutes (proves worker + Redis + storage end to end). Delete it afterwards (Test-data cleanup).