Skip to content

Service Level Objectives & Alert Rules

Status: active from Phase 0 (T-0.15) · Scope: apps/server HTTP API and its job queue

Every rule below is expressed against a metric this server actually emits today (T-0.13). That is a hard constraint, not a style note: an alert referencing a metric that does not exist never fires, and a monitoring system that silently never fires is worse than none, because it is trusted. tests/slo-rules.test.ts enforces it — it extracts every metric name used in this document and asserts each one appears in the live /metrics exposition. Adding an aspirational rule here fails the build.

1. Emitted metrics

Series Type Labels Source
http_server_request_duration_count histogram count route, method, status_class every response, via the Fastify onResponse hook
http_server_request_duration_bucket histogram bucket + le as above
http_server_request_duration_sum histogram sum as count as above
db_pool_connections gauge state = total|idle|waiting pg.Pool, read at scrape time
job_queue_depth gauge queue (fixed enum) one grouped query over pgboss.job in created+retry
job_execution_duration_count histogram count queue (fixed enum), outcome = success|failure every job, via the runInJobSpan wrapper each boss.work handler already passes through
job_execution_duration_bucket histogram bucket + le as above
job_execution_duration_sum histogram sum as count as above

Labels are deliberately bounded (R-0.5): route is a template with ids collapsed to :id, status_class is 2xx/4xx/5xx, and queue is a fixed list. No tenant, user, project or resource id is a label anywhere, and that is asserted by telemetry-metrics.test.ts, not merely reviewed.

Gap CLOSED in Phase 1 (T-1.6, TD-10). Depth alone answers "are workers keeping up" and cannot answer "did this job fail" — and the failure mode that matters inverts the signal, because a queue draining because every job throws drains faster than a healthy one. job_execution_duration now carries both: throughput is its count, failure rate is its count split by outcome.

Closing it exposed a second gap in the same area: job_queue_depth observed only 20 of the platform's 25 queues. The five cron queues are created by their own job modules rather than by startQueue(), so scheduled work — reminders, audit retention, access reviews, progress snapshots, anomaly scans — had no backlog signal at all. All 25 are now observed, and worker-registry.test.ts fails the build if a queue is drained without being measurable.

Useful expressions:

# failure rate per queue over 30m
sum by (queue) (rate(job_execution_duration_count{outcome="failure"}[30m]))
  / sum by (queue) (rate(job_execution_duration_count[30m]))

# a queue that is draining but failing — the case depth alone reports as healthy
sum by (queue) (rate(job_execution_duration_count{outcome="failure"}[15m])) > 0
  and sum by (queue) (job_queue_depth) == 0

2. Objectives

Windows are 30 days rolling. Targets are deliberately modest for a system with no production history — an SLO invented above measured performance produces alert fatigue, and the first honest re-baseline happens once real traffic exists.

⚠ 99.5% is below the competitive bar, deliberately and temporarily. Procore, Autodesk ACC and Bentley commit to 99.9% contractually. 99.5% permits ~3.6 h of downtime per 30 days; 99.9% permits ~43 minutes. A customer's security questionnaire asks for this number, and ours is visibly the weaker answer.

The target is honest for a system with no production history — an SLO invented above measured performance produces alert fatigue — but it stops being defensible the moment a contract quotes it. Treat 99.9% as the number the architecture must be able to support (it constrains the Phase-1 hosting choice: a single-AZ deployment cannot credibly claim it), and re-baseline the published figure once there is real traffic to measure. See docs/roadmap/Phase-1/QUESTIONS.md.

SLO-1 — Availability · target 99.5%

Proportion of requests that do not return 5xx. Client errors (4xx) are excluded: a rejected malformed request is the API working correctly.

sum(rate(http_server_request_duration_count{status_class!="5xx"}[30d]))
  / sum(rate(http_server_request_duration_count[30d]))

Error budget at 99.5% = 0.5% of requests, ≈ 3.6 hours of full outage per 30 days.

SLO-2 — Latency · target 95% of requests under 500 ms

sum(rate(http_server_request_duration_bucket{le="500"}[30d]))
  / sum(rate(http_server_request_duration_count[30d]))

le="500" is a bucket boundary the histogram actually has — a rule using an absent boundary silently interpolates nonsense. Measured on the server's own handling time; it excludes network and client render, so it is not a user-perceived-latency objective and must not be quoted as one.

SLO-3 — Job backlog · target: no queue above 1,000 pending for 15 minutes

max by (queue) (job_queue_depth)

Not a ratio, so it carries no error budget — it is a saturation objective. A backlog that clears is healthy; one that grows monotonically means workers are not keeping up with intake.

3. Alert rules

Multi-window, multi-burn-rate for the ratio SLOs (the Google SRE workbook pattern): a fast window catches an outage in minutes, a slow window catches a slow bleed, and requiring both to be burning suppresses the single-spike false positives that make people mute alerts.

groups:
  - name: bimzone-slo
    rules:
      # ---- SLO-1 availability ----
      - alert: BimZoneErrorBudgetBurnFast
        # 14.4x burn over 1h exhausts a 30d budget in ~2 days → page.
        expr: |
          (
            sum(rate(http_server_request_duration_count{status_class="5xx"}[1h]))
              / sum(rate(http_server_request_duration_count[1h]))
          ) > (14.4 * 0.005)
          and
          (
            sum(rate(http_server_request_duration_count{status_class="5xx"}[5m]))
              / sum(rate(http_server_request_duration_count[5m]))
          ) > (14.4 * 0.005)
        for: 2m
        labels: { severity: page }
        annotations:
          summary: "Fast error-budget burn — 5xx rate is 14x the SLO-1 budget"

      - alert: BimZoneErrorBudgetBurnSlow
        # 6x burn over 6h — a slow bleed that still exhausts the budget within the window.
        expr: |
          (
            sum(rate(http_server_request_duration_count{status_class="5xx"}[6h]))
              / sum(rate(http_server_request_duration_count[6h]))
          ) > (6 * 0.005)
          and
          (
            sum(rate(http_server_request_duration_count{status_class="5xx"}[30m]))
              / sum(rate(http_server_request_duration_count[30m]))
          ) > (6 * 0.005)
        for: 15m
        labels: { severity: ticket }
        annotations:
          summary: "Sustained error-budget burn — 5xx rate is 6x the SLO-1 budget"

      # ---- SLO-2 latency ----
      - alert: BimZoneLatencyBudgetBurn
        expr: |
          (
            1 - (
              sum(rate(http_server_request_duration_bucket{le="500"}[1h]))
                / sum(rate(http_server_request_duration_count[1h]))
            )
          ) > (14.4 * 0.05)
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "Latency budget burning — well under 95% of requests are completing in 500 ms"

      # ---- SLO-3 saturation ----
      - alert: BimZoneJobBacklog
        expr: max by (queue) (job_queue_depth) > 1000
        for: 15m
        labels: { severity: ticket }
        annotations:
          summary: "Queue {{ $labels.queue }} has over 1000 pending jobs for 15 minutes"

      # The alert BimZoneJobBacklog structurally cannot fire for (T-1.6, TD-10). A handler that throws
      # immediately drains its queue FASTER than a healthy one, so depth goes to zero, the backlog
      # alert stays quiet, and the work is simply never done. Failure rate is the only signal that
      # distinguishes the two, and it did not exist until job_execution_duration did.
      - alert: BimZoneJobFailureRate
        expr: >
          sum by (queue) (rate(job_execution_duration_count{outcome="failure"}[15m]))
            / sum by (queue) (rate(job_execution_duration_count[15m])) > 0.25
        for: 15m
        labels: { severity: page }
        annotations:
          summary: "Over 25% of {{ $labels.queue }} jobs are failing — the queue may be draining without doing the work"

      # ---- Leading indicator, not an SLO ----
      - alert: BimZoneDbPoolSaturated
        # `waiting` above zero means requests are queuing for a connection. This shows up BEFORE the
        # latency SLO does, which is the point of having it.
        expr: db_pool_connections{state="waiting"} > 0
        for: 5m
        labels: { severity: ticket }
        annotations:
          summary: "Requests are waiting for a Postgres connection — pool saturated"

4. Known limitations

  1. No production baseline. Targets are engineering judgement, not measurement. First honest re-baseline once real traffic exists; until then treat breaches as information about the target as much as about the system.
  2. Single-process topology. Workers run in-process with the API today, so a job storm and an API slowdown are the same incident and these rules cannot separate them. Phase 1's split changes that, and SLO-3 should gain a per-deployable dimension then.
  3. No job success/failure metric (see §1). Queue depth is a proxy: a queue that drains because every job is failing fast looks identical to a healthy one.
  4. Scrape-time gauges. db_pool_connections and job_queue_depth are sampled when scraped, so a spike between scrapes is invisible. Acceptable for saturation signals; not a basis for billing or compliance claims.
  5. SOC 2 (Phase 2) needs evidence of acting on these alerts, not just their definition. Routing, on-call ownership and a response record are Phase 2 deliverables; this document is the input to them, not the completed control.