Skip to content

Runbook

Service Path What it proves
Web /api/health The process is up
API /health The process is up

Both are liveness only and touch no database, on purpose: a DB-touching probe on the container healthcheck lets a Neon blip restart every container at once. A green healthcheck is not evidence that the database is reachable.

Symptom Check
Local ECONNREFUSED / CONNECT_TIMEOUT DATABASE_URL is the -pooler host with sslmode=require&channel_binding=require; restart the dev server
A domain cannot sign in It is missing from BETTER_AUTH_TRUSTED_ORIGINS (comma-separated)
403 on a new admin page The path is not registered in PAGE_ACCESS in packages/core/src/lib/rbac.ts
Asset URLs have a double slash CLOUDFLARE_R2_PUBLIC_URL has a trailing slash
Revenue numbers look low A query counted confirmed but not partially_paid
Stripe retrying a webhook for days Something in the handler throws on a structural condition — capture, log, return 200
A customer paid but has no booking Look for an orphan payment row; a succeeded PaymentIntent always leaves one
Deploy fails at exporting layers A production and a beta deploy ran at the same time — retry one at a time
WhatsApp number logged out More than one gateway container ran against the same auth volume. One only
WhatsApp number blocked or restricted Cold sends went out too fast or too many. Lower WHATSAPP_COLD_DAILY_CAP, and do not blast for a few days — WhatsApp outreach
Vendor blast shows stopped_* offline: reconnect the gateway. risk: 3+ STOP replies or >10% failures, so read the replies before sending more. limit: the daily cold budget is spent, so send the rest tomorrow
“Daily WhatsApp limit” when starting a blast The list is bigger than today’s remaining cold budget. Pick fewer shops
Feature toggled but nothing changed Other replicas take up to 30 s to see it. If it still looks wrong after that, check whether an override exists or the env default applies
Petition returns 404 The petition_pc flag is off (production defaults to off). Check /admin/feature-flags
Petition sign fails with “Notice updated” / “Petition notice not published” A newer privacy notice was published after the form loaded (reload), or none exists in this environment yet (republish-legal-pages.ts --create)
Migration journal disagrees with the database Someone ran DDL outside Drizzle. Stop, check bun run db:check:migrations, fix forward with a custom migration
  • Staff roles are granted on /admin/users (audited users.updateRoles). There is no OTP shortcut in production.
  • /admin/scanning-monitor shows whether every station is scanning.
  • The signing queue display belongs on the venue screen, not a staff device.

Append an entry to docs/booking-payments/decisions.md — append-only, a reversal is a new entry. The log is the proof trail for two prior incidents.