2026-05-04 — bricks-api production: 2/3 replicas crashed silently

TL;DR

On 2026-05-04 at 14:30 UTC, two of three production bricks-api replicas crashed during a Postgres connection drop and Railway gave up retrying after ~9 seconds, leaving the service running degraded on a single replica until a manual restart on 2026-05-05 at 08:19 UTC (~17h of degraded capacity).

No alert fired. The incident was discovered visually by checking the Railway UI.

Timeline (UTC)

Time	Event
2026-05-04 14:30:02.040Z	Replica A — Error: Connection terminated unexpectedly from pg/lib/client.js:136 (unhandled error event on the pg pool → uncaught exception)
2026-05-04 14:30:02.262Z	Replica B — same
2026-05-04 14:30:02.564Z	Replica C — same
2026-05-04 14:30:03.633Z	Railway emits Deployment.restarted ×2 (replicas A and B)
2026-05-04 14:30:10.223Z	Replica C boots Nest successfully (Starting Nest application)
2026-05-04 14:30:12.118Z	Railway emits Deployment.crashed ×2 — CRITICAL (replicas A and B abandoned after retries)
2026-05-04 14:30:29.858Z	Replica C catches Failed to acquire permit to connect to the database while reconnecting, retries successfully
2026-05-05 08:19:32.409Z	Manual Deployment.deployed triggered by operator → 3 replicas back online

Root cause

Layer 1 — Unhandled error event on the pg pool.

Every node-postgres Pool re-emits the error event of its idle clients on the pool itself. The daily ~09:30 / 14:30 UTC connection drops are caused by one of our own crons triggering a Neon compute scale up/down — this is an intentional, expected behaviour, not a Neon-side incident. When the compute scales, idle clients in our pools emit Error: Connection terminated unexpectedly. None of the four pools we instantiate had a listener attached:

• src/common/services/typeorm-config.service.ts — the TypeORM DataSource master pool
• src/__new/lib/kysely/database.ts — the Kysely pool used by espace-fi modules
• src/lib/better-auth/better-auth.factory.ts — two pools created by createAuth

Without a listener, Node's EventEmitter rethrows the error as an uncaught exception and the process dies. There is no stack on stdout because the crash happens too early for the logger to flush. The pool itself is self-healing — the broken client is destroyed and the next acquire opens a fresh connection — so a listener that just logs the error keeps everything working.

Layer 2 — restartPolicyMaxRetries: 10 is too tight.

projects/api/railway/railway.api.json had restartPolicyType: "ON_FAILURE" with the Railway default restartPolicyMaxRetries: 10. Once the three replicas crashed simultaneously, they all tried to reconnect at the same time, hit the Postgres "Too many database connection attempts are currently ongoing" rate limit, and re-crashed. Two replicas burned through all ten retries in about nine seconds and Railway moved them to CRITICAL crashed state, where they remained until a manual redeploy. The third replica got lucky on its first reconnect and survived.

Layer 3 — No alert wired up.

notificationRules for project 00 - Core Application was empty when queried via the Railway GraphQL API on 2026-05-05. Datadog receives the service logs (the agent runs as a sibling service on Railway with DD_LOGS_ENABLED=true and container_collect_all), but no monitor was configured against the crash signature or against replica health.

This was fixed out of band on 2026-05-05: Railway crash notifications are now active on the 00 - Core Application project. Datadog log monitors on the pg connection signature are tracked as a separate follow-up.

Fixes shipped on this branch

1. attachPgPoolErrorHandler helper

New helper in projects/api/src/__new/lib/postgres/attach-pg-pool-error-handler.ts. Logs the idle-client error via the Nest logger and lets the pool self-heal on the next acquire. Wired up at every new Pool(...) call site in projects/api:

• common/services/typeorm-config.service.ts (TypeORM master pool, attached in onModuleInit after dataSource.initialize())
• __new/lib/kysely/database.ts (espace-fi Kysely pool)
• lib/better-auth/better-auth.factory.ts (better-auth pool — also collapsed two redundant new Pool(...) calls into one)

The better-auth.check-and-migrate.script.ts is left as-is on purpose: it is a short-lived migration runner where crashing on connection issues is the correct behaviour (we do not want to report "migration OK" if the connection died mid-run).

2. restartPolicyMaxRetries: 10 → 20

projects/api/railway/railway.api.json. Doubles the retry budget while keeping the same ON_FAILURE policy. Together with fix (1), Railway now has roughly twice as long to wait out a Postgres reconnection burst before giving up.

We deliberately did not push this much higher: the fundamental fix is the error handler. The bumped budget is a safety net for the next unknown failure mode.

Follow-ups

• Datadog monitor on the pg connection-drop log signature, scoped to alert only on sustained spikes (we now self-heal on each individual drop, but a sudden surge would still indicate something abnormal — e.g. the scaling cron firing more often than intended, or Neon-side instability on top of our cron).
• TypeORM pool tuning: keepAlive: true on the underlying pg client and extra.idleTimeoutMillis set slightly under Neon's idle-disconnect window so we close clients ourselves before Neon does. Reduces the rate of idle-client errors even if it does not change correctness now that the handler is in place.

How to verify the fix

Manual test on a PR environment:

1. Deploy this branch to a PR environment
2. From the Neon console, terminate the compute for that branch (Operations → Restart Compute)
3. Watch the API logs: you should see one pg.Pool[typeorm-master] idle client error — connection lost, pool will reconnect on next acquire line per pool, then normal operation resumes within seconds
4. Without this branch the same action would crash the process and Railway would emit Deployment.restarted