Runbook
Health endpoints
Section titled “Health endpoints”| Service | Path | What it proves |
|---|---|---|
| Web | /api/health |
The process is up |
| API | /health |
The process is up |
Both are liveness only and touch no database, on purpose: a DB-touching probe on the container healthcheck lets a Neon blip restart every container at once. A green healthcheck is not evidence that the database is reachable.
Symptom → first thing to check
Section titled “Symptom → first thing to check”| Symptom | Check |
|---|---|
Local ECONNREFUSED / CONNECT_TIMEOUT |
DATABASE_URL is the -pooler host with sslmode=require&channel_binding=require; restart the dev server |
| A domain cannot sign in | It is missing from BETTER_AUTH_TRUSTED_ORIGINS (comma-separated) |
| 403 on a new admin page | The path is not registered in PAGE_ACCESS in packages/core/src/lib/rbac.ts |
| Asset URLs have a double slash | CLOUDFLARE_R2_PUBLIC_URL has a trailing slash |
| Revenue numbers look low | A query counted confirmed but not partially_paid |
| Stripe retrying a webhook for days | Something in the handler throws on a structural condition — capture, log, return 200 |
| A customer paid but has no booking | Look for an orphan payment row; a succeeded PaymentIntent always leaves one |
Deploy fails at exporting layers |
A production and a beta deploy ran at the same time — retry one at a time |
| WhatsApp number logged out | More than one gateway container ran against the same auth volume. One only |
| WhatsApp number blocked or restricted | Cold sends went out too fast or too many. Lower WHATSAPP_COLD_DAILY_CAP, and do not blast for a few days — WhatsApp outreach |
Vendor blast shows stopped_* |
offline: reconnect the gateway. risk: 3+ STOP replies or >10% failures, so read the replies before sending more. limit: the daily cold budget is spent, so send the rest tomorrow |
| “Daily WhatsApp limit” when starting a blast | The list is bigger than today’s remaining cold budget. Pick fewer shops |
| Feature toggled but nothing changed | Other replicas take up to 30 s to see it. If it still looks wrong after that, check whether an override exists or the env default applies |
| Petition returns 404 | The petition_pc flag is off (production defaults to off). Check /admin/feature-flags |
| Petition sign fails with “Notice updated” / “Petition notice not published” | A newer privacy notice was published after the form loaded (reload), or none exists in this environment yet (republish-legal-pages.ts --create) |
| Migration journal disagrees with the database | Someone ran DDL outside Drizzle. Stop, check bun run db:check:migrations, fix forward with a custom migration |
During an event
Section titled “During an event”- Staff roles are granted on
/admin/users(auditedusers.updateRoles). There is no OTP shortcut in production. /admin/scanning-monitorshows whether every station is scanning.- The signing queue display belongs on the venue screen, not a staff device.
After a payment-path change
Section titled “After a payment-path change”Append an entry to docs/booking-payments/decisions.md — append-only, a
reversal is a new entry. The log is the proof trail for two prior incidents.
