Copilot runbook — providers, credentials, permissions, failure modes¶

The operator's page for the in-product copilot. It is ON in Production, verified end to end against three vendors on 2026-09-12 and re-verified through the real UI on 2026-09-16 (every vendor ok). The user-facing page is guide/10-copilot.md. Every change below is a Dokploy environment change; saving the environment redeploys the compose.

Requirements coverage¶

Requirement	Section
Supported providers	Providers
Environment variables	Environment variables
Secure configuration	Secure configuration
Model selection	Model selection
Permissions	Permissions and tenancy
Confirmation of writes	Write confirmation
Tool usage	Tools
Failures and failover	Failures and failover
Rate limits and cost	Rate limits and cost
The 503 "no model configured" state	The off state
Troubleshooting	Troubleshooting
What never goes in source control or logs	Never in git, never in logs

Providers¶

Selected by configuration, never by a code change (apps/server/src/lib/copilot/factory.ts):

Provider id	Wire format	Default model (verified end to end)	Env key
`google`	Gemini	`gemini-2.5-flash`	`GOOGLE_API_KEY`
`openrouter`	OpenAI-compatible	`deepseek/deepseek-v4.1-flash`	`OPENROUTER_API_KEY`
`deepseek`	OpenAI-compatible	`deepseek-flash`	`DEEPSEEK_API_KEY`
`anthropic`	Anthropic Messages	(set `COPILOT_MODEL`)	`COPILOT_API_KEY` + `COPILOT_PROVIDER=anthropic`
`openai`	OpenAI-compatible (OpenAI, Together, Groq, vLLM…)	(set `COPILOT_MODEL`)	`COPILOT_API_KEY` + `COPILOT_PROVIDER=openai` + `COPILOT_BASE_URL`

Every vendor call has a 20 s timeout (the health check uses 15 s).

Environment variables¶

Two modes, resolved by resolveCopilot:

Vendor keys (simplest). Drop in any of GOOGLE_API_KEY, OPENROUTER_API_KEY, DEEPSEEK_API_KEY. Every key present joins the chain in that fixed order; the first present is the primary and all others are fallbacks. COPILOT_PROVIDER=<google|openrouter|deepseek> moves that vendor to the front without duplicating its key; naming a vendor with no key is ignored. COPILOT_MODEL overrides the primary's model only.
Explicit mode (wins when COPILOT_API_KEY is set). COPILOT_PROVIDER (default anthropic), COPILOT_API_KEY, COPILOT_MODEL, COPILOT_BASE_URL, plus an optional COPILOT_FALLBACK_PROVIDER / COPILOT_FALLBACK_API_KEY / COPILOT_FALLBACK_MODEL / COPILOT_FALLBACK_BASE_URL. Vendor keys are ignored in this mode.

What Production has (Dokploy → project Bimzone → environment production → compose app → Environment): the three vendor keys and nothing else copilot-related.

docker-compose.yml forwards these three and every COPILOT_* name to the api and worker; apps/server/tests/structural/compose-env-allowlist.test.ts fails the build if one is dropped from the allowlist, because a variable absent from it is silently invisible inside the container. Empty values are safe — every copilot setting tolerates "", which compose writes for any unset variable.

Secure configuration¶

Keys live only in Dokploy's environment store and reach the api container through the compose allowlist. They are never written to the database, never returned by any route, never logged.
The api container needs outbound HTTPS to the vendors; nothing inbound.
Prefer per-vendor keys with the smallest scope the vendor offers, one key per deployment, so a rotation affects one environment.
COPILOT_MODEL deliberately has no default in config.ts. It used to default to an Anthropic model name, which made "unset" unreadable: Google was called with that name, 404'd on every request, and the copilot answered from its fallback while looking healthy. A test reads config.ts and fails if a default comes back. Set the model and provider together or not at all.

Model selection¶

Vendor-key mode: each vendor's verified default (table above). Override the primary with COPILOT_MODEL; reorder with COPILOT_PROVIDER.
Explicit mode: COPILOT_MODEL names the model for COPILOT_PROVIDER; the fallback pair names its own.
A model name steered onto the wrong vendor (a Google name on DeepSeek) is the misconfiguration the health check exists to surface: every vendor row shows its vendor:model and its reply.

Permissions and tenancy¶

The copilot acts as an in-product API key scoped to the caller's workspace with scope read, or write when the caller's project role is anything but viewer. It cannot read or change anything the user's own account could not.
Every tool call is a normal API call through the same handlers and tenancy checks; the project id in the route is the only project the tools can see.
A viewer is never offered write tools; a confirmation sent by a viewer is refused.
Check model status (GET /api/projects/:projectId/copilot/providers) is project-admin only.

Write confirmation¶

The model never executes a write. When it proposes one, the route returns a pending card with the tool name and the exact arguments, plus a confirmation token that is an HMAC (under APP_SECRET) over ${userId}:${projectId}:${conversationId}, the tool and a stable serialisation of the arguments.
The client posts confirm: { tool, args, token } back on the same conversation; the server recomputes the HMAC. A token obtained for one action cannot be replayed for another, by another user, on another project, or on another conversation — which is why the pending arguments are shown but not editable.
Exactly one write runs per confirmation; the action is audited as copilot.write_confirmed with the tool and arguments and shows in the project Activity feed as Copilot Write Confirmed.
If it ever creates something without asking, stop and raise it — that is the guardrail failing and the only outcome here worth an incident.

Tools¶

The copilot uses the MCP tool set (apps/server/src/mcp/tools.ts). Nine reads and one write:

Tool	Scope	What it returns
`list_projects`	read	the projects the key can see
`search_elements`, `get_element`	read	elements by text / by GUID with properties
`list_issues`, `list_rfis`, `list_wirs`, `list_mars`	read	the registers, filterable
`clash_summary`	read	the project's clash figures
`project_stats`	read	project-level statistics
`create_issue`	write	creates an issue, optionally attached to elements by GUID — always behind confirmation

Loop bound: maxSteps = 6 tool calls per question; the model then stops and says so ("Stopped after 6 tool calls without reaching an answer."). Tool results are truncated at 4,000 characters each. Citations shown to the user are the real tool results, not model prose.

The system prompt pins the behaviour: answer from the tools; never state a figure a tool did not show; if the tools do not contain the answer, say so; keep answers short enough for a site tablet.

Failures and failover¶

With N keys, a question survives N−1 vendors being down. Failover happens only on a vendor failure (rate limit, 5xx, timeout, bad model name) and never on a content refusal.
The chain is walked in series, so the worst case — every vendor timing out — is 20 s × vendors per model call before the user sees a 503 carrying the vendors' reasons; nginx's /api/ block allows 600 s so it is a real 503, not a proxy 504.
Each failover logs copilot provider failed; trying the next one with the vendor's own message and no project data. Repeated lines for one vendor = check its key, credits and model name.
Observed on 2026-09-12: Google 503 "high demand" and OpenRouter 402 (in_flight_budget_exhausted, a free-tier budget) in the same minute, DeepSeek answered. On 2026-09-16 all three ok (Google 640 ms, OpenRouter 5.2 s, DeepSeek 958 ms).

Rate limits and cost¶

Route limit: 30 requests per minute per session, keyed on the bearer token; the health check shares it. A runaway client is bounded at 30 × 7 model calls a minute on one session.
One question is up to 6 tool calls and 7 model calls. Budget vendor spend accordingly; the health check sends the fixed prompt ping with no tools.
Over the limit the route answers 429; there is no queueing — the client waits.

The off state (503 "No copilot model is configured…")¶

Exact text returned when no provider resolves:

No copilot model is configured. Set a provider key — GOOGLE_API_KEY, OPENROUTER_API_KEY or DEEPSEEK_API_KEY — or COPILOT_API_KEY with COPILOT_PROVIDER. The tools, tenancy checks and write confirmations are in place and wait only on a credential.

It is a 503, not a 500, on purpose: "not configured yet" is an operator's next step. The page still loads and says so; nothing is invented. To turn the feature off deliberately, remove every copilot key and save — this state is the result.

Checking it¶

Project → Copilot → Check model status (admins): per vendor Primary / Fallback, vendor:model, ok · N ms or the vendor's error. ok proves credential + model name + network.
Ask "how many issues are open?" — a correct answer cites at least one tool call and the footer names the model.
Ask it to create an issue — it must stop with This would change the project and a Confirm and run button. Confirm once, verify exactly one issue exists, delete it (the write is real).

Troubleshooting¶

Symptom	Cause	Fix
503 "No copilot model is configured…"	no key reaches the container	set the key in Dokploy; confirm the name is in the compose allowlist
Health strip shows a vendor 401/403	wrong or revoked key	replace the key
Vendor 404 on every request but answers still arrive	wrong model name on that vendor (fallback is answering)	fix `COPILOT_MODEL` / remove the override
Vendor 402 (OpenRouter `in_flight_budget_exhausted`)	free-tier budget	fund the account or accept the skip
Vendor 503 "high demand"	vendor capacity	nothing; failover covers it
Every request ~20 s × vendors then 503	all vendors down or outbound HTTPS blocked from the api container	check egress, then keys
429	30/min/session	wait a minute
"Stopped after 6 tool calls…"	the question needs more than the loop bound	narrower question
A viewer gets no write tools	by design	correct
Confirmation refused (403 / invalid token)	different user, project or conversation than the pending card; or a viewer	re-ask on the same conversation

Rotating or disabling¶

Replace a vendor key and save. Remove all of them and the copilot returns the honest 503 above.

Adding a fourth vendor¶

Any OpenAI-compatible endpoint works through explicit mode: COPILOT_API_KEY + COPILOT_PROVIDER=openai + COPILOT_BASE_URL + COPILOT_MODEL. To make it a first-class vendor key instead, add one row to VENDOR_KEY_ENV in factory.ts, forward the name in docker-compose.yml, and add it to MUST_FORWARD in the structural net — the tests tell you if you miss one.

Never in git, never in logs¶

Never in source control: any vendor key, COPILOT_API_KEY, .env files with values, Dokploy exports, screenshots of the environment page.
Never in logs: vendor keys; the confirmation token; full prompts and tool results (they contain project data and go to the vendor by design — the api logs only the vendor's error message on failover); bearer tokens.
Never in the database: vendor keys (the copilot's acting key is synthesised per request and not persisted).
What is recorded: the audit row for every confirmed write (tool + arguments), the rate-limit counters, and the health-check results shown on the page.