Outbox Dispatcher — Runbook & Local Verification¶

Inter-module events flow through an in-process dispatcher, not Kafka. The transactional outbox pattern is documented in conventions.md § Transactional Outbox; the @on_event handler convention in conventions.md § Event Handlers. This runbook covers the relay — the OutboxPoller that drains committed outbox rows and dispatches each event to the registered @on_event handlers via the EventDispatcher.

Kafka is deferred until a module is extracted to its own service. When that happens, swap the dispatcher target for a Kafka producer/consumer — the @on_event handler signatures and the published events stay identical, only the transport changes.

Topology¶

┌────────────────────────┐   same SQL txn    ┌─────────────────────┐
│ command handler        │ ────────────────▶ │ outbox table        │
│  + OutboxEventPublisher │  INSERT row +     │  status = pending   │
└────────────────────────┘  NOTIFY outbox_new └──────────┬──────────┘
                                                         │ LISTEN/NOTIFY
                                                         ▼
                                              ┌─────────────────────┐
                                              │ OutboxPoller         │
                                              │  FOR UPDATE          │
                                              │  SKIP LOCKED         │
                                              └──────────┬──────────┘
                                                         │ per event:
                                                         │ fresh AsyncSession
                                                         ▼
                                              ┌─────────────────────┐
                                              │ EventDispatcher      │
                                              │  asyncio.gather over │
                                              │  @on_event handlers  │
                                              └─────────────────────┘

All of this runs inside the FastAPI process (src/main.py lifespan starts the poller as a background asyncio task). There are no external broker services.

Startup wiring¶

src/main.py lifespan, in order:

1. load_all_handlers() walks src/modules/*/application/event_handlers.py and imports each so the @on_event decorators fire and register into the default dispatcher. This must run before the poller starts — otherwise the dispatcher is empty and every outbox row drains to completed without side effects.
2. OutboxPoller is constructed against get_jobs_session_factory() (JOBS_DB_URL → the grinsystem_jobs role, which holds BYPASSRLS). The poller's claim query drains every tenant's events without setting app.current_tenant, so it must bypass the outbox table's tenant_isolation RLS policy. See GRI-152.
3. The poller runs as a named asyncio task (outbox-poller) via run_forever().

Shutdown cancels the poller task before disposing the engine, so the in-flight batch can roll back cleanly. Partially claimed rows are left pending on restart — never half-marked.

Poll loop¶

OutboxPoller.run_forever() repeats: process_batch(), then wait for a NOTIFY or timeout.

• Claim (_claim_batch): SELECT ... WHERE status = 'pending' ORDER BY created_at LIMIT OUTBOX_BATCH_SIZE FOR UPDATE SKIP LOCKED. The SKIP LOCKED means any number of pollers can run safely against the same table — peers skip rows another worker already holds.
• Dispatch (_dispatch_one): each event runs in its own fresh AsyncSession minted from the session factory, wrapped in a CONSUMER span parented on the publishing request's trace. Handlers never share the poller's lock-holding session, so a handler's rollback can't poison the batch's marking transaction. The batch's events are dispatched concurrently via asyncio.gather(..., return_exceptions=True).
• Mark: rows whose handlers all returned cleanly are bulk-updated to completed with processed_at; failed rows get a per-row update to failed with processed_at, retry_count + 1, and a truncated error string (OUTBOX_ERROR_MAX_LEN).
• Wait (_wait_for_notify): blocks on Postgres LISTEN outbox_new for a low-latency wake-up, falling back to OUTBOX_POLL_TIMEOUT_SECS so it still drains on a missed NOTIFY.

Tuning¶

Status lifecycle¶

An event whose event_type has no registered subclass reconstructs as the bare DomainEvent envelope; the dispatcher sees zero handlers and the row is marked completed without side effects — the correct semantics for an unknown event type.

Metrics¶

The dispatcher emits four metrics (full table in observability.md § Outbox dispatcher):

Each dispatch also emits a shared.consume_<EventType> CONSUMER span linked back to the HTTP request that produced the row, so one SigNoz trace spans HTTP request → handler → outbox row → consume → downstream handler.

Common failure modes¶

Rows stay pending, never dispatched¶

• Poller not running. Confirm the outbox-poller task started — look for the outbox_dispatcher_starting log line at startup. If handler_counts is empty, no handlers registered (see next).
• RLS hides the rows. The poller must connect as a BYPASSRLS role. In production JOBS_DB_URL points at grinsystem_jobs; in dev/test it falls back to DATABASE_URL (the bootstrap superuser, which also bypasses RLS). If JOBS_DB_URL is misconfigured to a non-BYPASSRLS role, the claim query returns nothing.

Rows drain to completed but nothing happens downstream¶

The dispatcher is empty — load_all_handlers() didn't import the handler module, or no module wires @on_event for that event type yet. Check the outbox_dispatcher_starting log's handler_counts. A module's handlers must live in src/modules/<m>/application/event_handlers.py to be autodiscovered.

Rows land in failed¶

A handler raised. Inspect the row's error column (truncated to OUTBOX_ERROR_MAX_LEN) and retry_count. There is no automatic retry today — re-dispatch is a manual/operational decision until the retry milestone lands. Because delivery is at-least-once, handlers must be idempotent (processed_events dedupe — see conventions.md § Handler-side idempotency).

Dispatch lag climbs (outbox_lag_seconds P95 high)¶

Publish rate outruns dispatch. Confirm dispatcher_batch_size P95 is pinned near OUTBOX_BATCH_SIZE (the tell). Raise OUTBOX_BATCH_SIZE, or — once a module is extracted — this is the signal to move that module's events onto Kafka.

Local verification¶

The outbox_dispatcher integration tests (GRI-116) exercise the full publish → poll → dispatch → mark path against a real Postgres:

make test    # includes the outbox_dispatcher integration fixture

No broker, no Kafka Connect, no extra services — the dispatcher runs in-process, so the tests need only the database.