CRM architecture

The decision record is ADR-017. This page describes how the pieces fit together. For the product behaviour, see the CRM guide; for the endpoints, the CRM API.

Components

Piece	Where	Responsibility
Domain logic	`packages/crm`	Lead capture and merge (one open lead per contact), field provenance, lead rules and scoring, qualification, assignment strategies, SLA deadlines, deals and fractional board ranks, conversion, companies, click-to-chat references, form mapping, visibility, AI insight application
Inbound capture	`apps/whatsapp-worker/src/message-events.ts`	`captureInbound` in the transaction that records and threads an inbound message (in a savepoint: a CRM failure never loses a message); `markLeadResponded` for messages typed on the phone
API	`apps/api/src/modules/crm`	Leads, deals, pipelines, companies, notes, tasks, settings, custom fields, scoring and assignment rules, forms and links (management and public endpoints), reports
Automation	`apps/scheduler/src/crm.ts`	`CrmAutomation` loop (assignment, woken by `crm-nudge`, 30 s poll); SLA sweep, task reminders, rescoring and retention jobs
AI	`apps/ai-worker/src/lead-intel.ts`, `packages/ai/src/lead-prompt.ts`	`LeadIntelWorker`: classify conversations (AI capture mode) and extract qualification details; qualifying goals for AI replies
Imports	`apps/campaign-worker/src/imports.ts`	Optional lead per imported row, in the same savepoint as the contact
Dashboard	`apps/web/app/app/[orgId]/p/[projectId]/crm`, `apps/web/app/f/[publicId]`, `apps/web/public/embed/v1.js`, `apps/web/proxy.ts`	CRM pages, hosted form, embed script, per-form framing policy

Data

contacts ──< leads (one open per contact) ──> companies
   │           └── conversation_id ··· conversations
   └── company_id ──> companies
pipelines ──< pipeline_stages ──< deals ──< deal_stage_history
                                    └── lead_id, contact_id, company_id
crm_activities (append-only timeline; subject + denormalized lead/deal/company/contact ids)
crm_notes, crm_tasks (subject: lead | deal | company)
crm_settings (project), crm_field_definitions, crm_scoring_rules, crm_assignment_rules
lead_forms ──< lead_form_submissions        lead_links ──< lead_link_clicks (ref_token)
lead_ai_jobs (one active per conversation)
inbox_notifications (shared feed; CRM rows carry subject_type/subject_id)

One open lead per contact is a partial unique index (leads_open_contact_uq, status in new/working/qualified, not deleted). Every capture path locks the contact row first, so concurrent captures (a message, a form fill, an API call) serialize and the later ones merge into the lead the first one created.
Field provenance (leads.field_sources) records who set each value: human > api/form/import > ai. A writer may replace a value only when it ranks at least as high as the previous writer; only people clear values.
Board order uses fractional ranks (base-36 strings compared with collate "C"): moving a card rewrites only that card.
Money is stored in minor units (bigint) with an ISO currency per deal; reports group by currency (no FX).

Flows

inbound message (worker)
  └─ one transaction: message → contact (locked) → thread → [savepoint] captureInbound:
        open lead?        → counters, link attribution, schedule extraction (debounced, rate-limited)
        (ref: XXXXXX)     → lead from click-to-chat with the click's UTM
        mode all_new      → lead unless one closed within reopenAfterDays
        mode keywords     → lead when a keyword matches (whole words, any script)
        mode ai           → lead_ai_jobs(classify)
  └─ after commit: crm-events, ai-nudge

capture (API / form / import)
  └─ one transaction: upsertContact (plan limit, opt-in with proof) → captureLead (lock contact → open lead?) →
     activity, outbox lead.*, rescore → automation_due_at when it needs an owner

automation loop (scheduler, woken by crm-nudge, 30 s poll)
  └─ per lead: SELECT … WHERE automation_due_at <= now() FOR UPDATE SKIP LOCKED
     → lock crm_settings (round-robin cursor) → conversation assignee | first matching rule | fallback → lead.assigned

lead AI job (ai-worker)
  └─ claim (SKIP LOCKED, lease, attempts++) → no locks: history + engine.extract (budget reserve/settle, kind lead_intel)
     → short transaction: lock contact → job (fence on attempts, rerun) → capture (classify) → applyAiInsights → rescore
     → optional handoff of the conversation to the lead owner when the lead becomes SQL

Realtime

Every CRM transaction collects IDs-only events (lead.changed, deal.changed, company.changed, task.changed, notification) in a CrmEventBuffer and publishes them to Redis crm-events:<projectId> after commit, with the owner and team as visibility hints. The dashboard-only SSE endpoint GET /app/orgs/{orgId}/projects/{projectId}/crm/stream (CrmStreamController, needs leads:read) relays them:

Events are filtered in memory by canSeeCrmEvent, which mirrors the list rules: in own_team mode agents get leads and deals that are unassigned, theirs or owned by a teammate (leads also by team); tasks that are unassigned or assigned to them; every company. Notifications go only to their recipient, whatever the viewer's role.
The viewer (visibility setting, teams, teammates) is re-read every 60 s, so membership changes apply to open streams.
A ready event on connect, ping every 15 s, gauge whatstig_crm_streams_open; unsubscribed on disconnect.

useCrmStream (mounted once in CrmGate) invalidates the matching React Query keys with 250 ms batching and refetches everything on (re)connect or drop. CRM_REFRESH_MS polls every 15 s without a stream and every 120 s with one. An event only names the record's current owner, so someone who just lost access to a record sees it disappear on the next fallback poll rather than instantly.

Locking order

Contact → lead → deal → crm_settings (assignment only). The AI worker locks the contact, then the job. The inbound path already holds the contact lock when it captures, so capture never waits on a lock the inbox path takes later.

Qualification

scoreLead sums the points of every active scoring rule whose condition matches (conditions are evaluated in memory against the lead's facts: fields, custom fields, source and UTM, contact tags, engagement counters, AI signals, forms, instance, phone prefix). MQL and SQL are thresholds; SQL also needs every requiredForSql detail. A person can override the level. Rescoring happens in every transaction that changes the lead, after rule changes (crm-rescore, resumable, batches of 200) and nightly for time-based rules.

AI

Capture mode ai: a classify job reads the conversation and creates a lead when isLead and the confidence is at least classifyMinConfidence.
Extraction: standard fields and custom fields marked AI-extractable are filled above fieldMinConfidence, with provenance ai (never overwriting people, forms or integrations), plus intent, urgency, sentiment, a summary and the details still missing. Re-extraction is debounced and rate-limited by reextractAfterMinutes.
Qualifying questions: when AI auto-reply is on, the reply prompt lists the missing details and allows one short question when it fits; maxQualificationQuestions bounds it per lead.
Budgets: the same per-organization reserve-and-settle budget as replies, ledger kind lead_intel. When it is exhausted, leads keep working with rules only.
The conversation is data in the prompt; output is a JSON object validated field by field (invalid values are dropped).

Forms and links

Public endpoints (@Public): GET /public/v1/forms/{publicId}, POST /public/v1/forms/{publicId}/submissions, GET /l/{code}. Submissions are rate-limited per IP hash and form, protected by a honeypot field, a minimum fill time and optional Cloudflare Turnstile; cross-origin submissions must come from an allowed origin.
The embed is an iframe of the hosted page. apps/web/proxy.ts sets frame-ancestors from the form's allowed origins (cached 60 s); every other dashboard page keeps frame-ancestors 'none'.
Click-to-chat: each click gets a 6-character reference in the prefilled WhatsApp text. The first inbound message that carries it within 14 days claims the click for that contact and attributes the lead.

Failure behaviour

Failure	Effect
CRM error during inbound capture	Savepoint rolled back, error logged; the message, thread and AI reply job are kept
AI provider down / budget exhausted	Job retried with backoff / skipped; rules-only scoring continues
Worker dies mid-AI-job	Lease expires; the sweeper re-queues (or fails after `AI_MAX_JOB_ATTEMPTS`); stale results are fenced
Nobody can take a lead	`automation_due_at` cleared (no hot loop); assignment rules or members changing do not retry old leads automatically
Redis down	Nudges lost; loops poll every 30 s; the CRM stream delivers nothing and dashboards fall back to polling
API unreachable from the web proxy	Hosted form frames only allowed from `'self'`

Observability

Metrics: whatstig_crm_leads_created_total{source}, whatstig_crm_lead_ai_jobs_total{kind,outcome}, whatstig_crm_automation_total{task,outcome}, whatstig_crm_automation_lag_seconds, whatstig_crm_form_submissions_total{outcome}, whatstig_crm_streams_open. The scheduler reports last_job_success for crm-sla-sweep, crm-task-reminders, crm-rescore-all and crm-retention. Alerts: see infrastructure/monitoring/alerts.yml.