Plan Court Report: Migrate subscription billing from Stripe to the internal ledger

Date: 2026-09-07T14:01:40.550Z
Verdict: PASS
Rounds: 3 (max 3 by setting)
Stopped: max rounds reached
Seats: Red Team (deepseek-v4-flash), Scope Cop (Google: Gemini 3.7 Flash), Architect (Qwen: Qwen3.8 27B), Tracer (Ox Alpha), Stress Test (Tencent: Hy-MT2-1.8B) | Judge: deepseek / deepseek-v4-pro
Tokens: 69092 in / 134138 out | Est. cost: $0.3617

Round 1

Red Team — deepseek-v4-flash — FAIL

Lens: Correctness & failure modes

• [major] Unparseable critique
  Critic returned non-JSON: {
  "verdict": "FAIL",
  "issues": [
  {
  "title": "Renewal cron can double-charge or skip due to race/partial failure",
  "severity": "blocker",
  "detail": "The cron renews any subscr
  Fix: Check model/prompt config.

Scope Cop — Google: Gemini 3.7 Flash — FAIL

Lens: Completeness & hidden assumptions

• [blocker] Active Stripe subscriptions are never cancelled, causing double billing
  Phase 3 flips renewals to the ledger cron using PaymentIntents, but nowhere in the plan are the existing Stripe Subscription objects paused or cancelled. Because Stripe's subscription engine runs autonomously on Stripe's infrastructure, both Stripe Subscriptions and the new internal cron will charge customers concurrently.
  Fix: Add an explicit migration step to disable/cancel Stripe Subscriptions at cutover (or set them to pause collection), ensuring each subscription is owned by exactly one billing engine at any point.
• [blocker] Rollback is impossible without a reverse-sync mechanism
  Step 7 asserts that the deploy is 'fully revertible' by flipping the feature flag. However, during the period the flag is active, new subscriptions, cancellations, and renewals processed by the ledger will not exist in Stripe Subscriptions. Reverting the flag will cause dropped renewals and out-of-sync customer billing states.
  Fix: Specify a bi-directional sync strategy during the cutover window or define a forward-fix-only protocol with explicit state reconciliation if cutover fails.
• [blocker] Missing dunning, off-session 3DS/SCA, and failed payment lifecycle
  Stripe Subscriptions handles off-session card authentication (SCA/3DS), smart retries, payment method expiration, and dunning sequences. The proposed ledger schema and cron directly charge PaymentIntents without handling requires_action (off-session 3DS challenges), retry backoff, or state tracking for past-due subscriptions.
  Fix: Design a dunning and retry state machine in the ledger schema (e.g., payment_failed, past_due), and implement customer webhook/email flows for PaymentIntents requiring off-session 3DS authorization.
• [major] No backfill migration for existing Stripe subscriptions
  Phase 1 adds the table and Phase 2 dual-writes ongoing events, but the plan contains no step to extract, transform, and backfill the pre-existing active subscriptions from Stripe into the ledger table prior to cutover.
  Fix: Add an explicit data migration job before Phase 2/3 to import all active Stripe subscriptions, payment method IDs, and period timestamps into the ledger.
• [major] Fire-and-forget dual-write guarantees data divergence
  Phase 2 uses fire-and-forget background jobs for subscription events without transaction guarantees or sequence ordering. Race conditions (e.g., a cancellation processing before a creation job completes) and unhandled worker failures will cause persistent ledger drift.
  Fix: Use a transactional outbox pattern with deterministic event versioning/timestamps instead of fire-and-forget background jobs.

Architect — Qwen: Qwen3.8 27B — FAIL

Lens: Design & simplicity

• [blocker] Hourly renewal cron creates a thundering herd and early-billing risk
  Phase 1 step 3 scans for subscriptions where current_period_end < now() + 1h and charges them via Stripe PaymentIntents. This processes every subscription due in the next hour in one batch, can charge up to an hour early, and has no batching, backoff, rate-limit handling, or idempotency key. Overlapping cron runs or Stripe retries can produce duplicate charges.
  Fix: Use an exact-due-time scheduler or partitioned queue with idempotency keys, retries, and Stripe rate-limit handling. Do not rely on a single hourly scan for financial renewals.
• [blocker] Fire-and-forget dual-write makes ledger state unreliable
  Phase 2 step 4 writes subscription events to the ledger as a background job so checkout stays fast. That is the wrong abstraction for financial state: dropped jobs, ordering issues, and silent failures will make the ledger diverge from Stripe. It also undermines the success criteria of zero failed renewals and a reconciliation diff trending to zero.
  Fix: Make the ledger write synchronous or use a transactional outbox with idempotent operations, retries, dead-letter handling, and alerting. The ledger should not be treated as a best-effort mirror.
• [major] Count-only reconciliation hides dangerous state drift
  Phase 2 step 5 compares subscription counts between Stripe and the ledger and writes a CSV reviewed on Monday. Counts can match while status, current_period_end, cancel_at_period_end, invoice IDs, or payment IDs differ. A Monday CSV review is too slow and too weak for a billing cutover.
  Fix: Compare full subscription state and invoice/payment references, alert on diffs in near real time, and gate cutover on zero unresolved diffs.
• [blocker] One-day cutover and flag rollback are not a safe migration
  Phase 3 flips a feature flag in one day and claims the deploy is fully revertible. It does not address in-flight renewals, already-created invoices, Stripe webhook state, backfill validation, or how to roll back after charges have been created. A flag flip is not a rollback for financial operations.
  Fix: Use a canary by customer cohort, shadow renewals, staged cutover, and an explicit rollback runbook that covers in-flight charges and already-created invoices.
• [major] Subscription schema and renew endpoint are not idempotent or auditable
  Phase 1 defines a minimal subscriptions table and a POST /subscriptions/:id/renew endpoint with no idempotency key, plan snapshot, effective dates, invoice/payment IDs, attempt history, or versioning. This makes retries, audits, and duplicate-charge prevention unsafe.
  Fix: Model subscription state transitions with idempotent operations, store external Stripe IDs and invoice/payment references, and require idempotency keys on renew.

Tracer — Ox Alpha — FAIL

Lens: Execution & rollout

• [major] Critic call failed
  HTTP 404: Thank you for participating in the Stealth Ox Alpha testing period. This model was ZAI's GLM-5.3 Flash. Use it now: https://openrouter.ai/z-ai/glm-5.3-flash
  Fix: Check provider config and API key.

Stress Test — Tencent: Hy-MT2-1.8B — FAIL

Lens: Data & integrity

• [major] Missing duplicate event detection for Stripe webhooks
  The current implementation dedups by event id but does not filter out duplicate events from the ledger side, which could cause double charges or missed updates.
  Fix: Add a unique identifier field to the ledger subscription event and use it for deduplication in the webhook handler.
• [major] No validation for status transitions
  There is no validation logic to ensure a subscription cannot be canceled while still being renewed, risking invalid state changes.
  Fix: Implement a business rule to block cancel/renew operations if the subscription is in 'suspended' or 'canceled' state.
• [minor] No monitoring for ledger write latency
  No tracking of ledger write performance, which could impact renewal processing speed.
  Fix: Add metrics to the ledger service to monitor write latency and trigger alerts if thresholds are exceeded.
• [minor] No fallback for ledger failure
  If the ledger service becomes unavailable, the billing flow will stop, causing service disruption.
  Fix: Implement a circuit breaker pattern that retries Stripe calls on ledger side failures and logs the error.
• [minor] No test environment for cutover
  The cutover assumes a live production environment, which may not be available during testing.
  Fix: Prepare a staging environment with the same schema and configurations to simulate the cutover process.

Round 2

Red Team — deepseek-v4-flash — FAIL

Lens: Completeness & hidden assumptions

• [blocker] Full-state reconciliation criterion is incompatible with Stripe Subscription cancellation/pausing after migration
  Phase 2 item 10 requires nightly full-state reconciliation (status, current_period_end, cancel_at_period_end, latest invoice/payment ID) to reach zero unresolved diffs before cutover and Phase 4 item 16 requires 30 consecutive days of zero full-state diffs before removing dual-write. But Phase 3 item 12 intentionally cancels or pauses the corresponding Stripe Subscriptions as each cohort is migrated to ledger-owned renewals. After that, any migrated subscription will legitimately diverge from Stripe: the Stripe object is canceled/paused while the ledger shows active, and the ledger's current_period_end advances on renewal while the Stripe object's period does not. The reconciliation as specified will therefore report non-zero diffs forever after cutover, making the Phase 4 cleanup gate impossible and likely blocking expansion if a nightly reconciliation runs after any cohort has moved.
  Fix: Define separate reconciliation semantics for migrated subscriptions: mark them as ledger-owned/Stripe-paused in the ledger, exclude them from full-state comparison against the Stripe Subscription object, or compare them only against Stripe PaymentIntents/charges. State explicitly that the pre-cutover zero-diff gate applies only to non-migrated dual-write subscriptions, and redefine the Phase 4 cleanup criterion accordingly.
• [major] Rollback protocol relies on reactivating a canceled Stripe Subscription, which Stripe does not support
  Phase 3 item 14 says for completed ledger renewals whose Stripe subscription was already canceled, either 'reverse-sync changed subscription state back to Stripe before reactivating the Stripe subscription' or use forward-fix-only. Stripe does not provide a way to resume/reactivate a canceled Subscription object. Restoring Stripe-owned billing would require creating a new Subscription with a new stripe_subscription_id, re-establishing the billing cycle anchor, and reconciling the period already paid through the ledger. The plan does not specify that recreation flow, how to preserve stripe_subscription_id mappings, or how to avoid double-charging when the new subscription overlaps a period already billed by the ledger. This makes the rollback protocol an unverified hand-wave.
  Fix: Prefer pause_collection over cancellation for any canary cohort that might need rollback, and document a tested Stripe API procedure for resuming paused subscriptions. If cancellation is unavoidable, specify that rollback creates a brand-new Stripe Subscription, defines billing_cycle_anchor, keeps stripe_subscription_id references in the ledger, and includes a reconciliation step to void or credit any overlapping new-subscription invoice.
• [major] Backfill and status enum omit trialing and other non-terminal Stripe subscription states
  Phase 2 item 9 backfills only 'active and trialing' Stripe Subscriptions, but Phase 1's ledger status field is required to support only active, past_due, canceled, and paused — there is no trialing status. Subscriptions in past_due, unpaid, incomplete, or paused states are real lifecycle states that the ledger scheduler and dunning will need to own. The plan never says how to map trialing into the ledger, nor how to backfill non-active subscriptions that may still require retries or conversions. This can silently drop valid subscriptions during the migration and violate the 100% renewal target.
  Fix: Expand the ledger subscription status enum to include trialing (or add a separate trial_end field) and backfill all non-terminal Stripe subscription states that can still renew, including past_due, paused, and incomplete where appropriate. Define explicit status mapping and migration validation for each backfilled Stripe state.
• [major] Subscription lifecycle coverage is incomplete: no plan for changes, proration, payment-method updates, or trial conversion
  The stated goal is 'all subscription billing (creation, renewal, cancellation)', and Phase 2 item 8 says dual-write covers 'creation, renewal, cancellation, and status change events'. But real subscription billing also includes plan changes/upgrades/downgrades, quantity changes, proration, payment method updates, and trial-to-active conversion. The plan only defines POST /subscriptions, /cancel, and /renew endpoints; there is no update or change endpoint, no event type for plan changes, and no statement about how existing subscriptions modified in Stripe during the dual-write week are propagated to the ledger. Reconciliation will only detect these diffs late, and the plan does not say who fixes them or how the ledger remains authoritative for the migrated lifecycle.
  Fix: Add explicit update/change endpoints or generic subscription-change event ingestion for plan/price/quantity changes, proration, and trial conversion. Specify how Stripe webhooks and the transactional outbox propagate these mutations to the ledger during dual-write and after cutover, and include them in reconciliation field comparisons.
• [major] Rollout timeline cannot satisfy the 7-day reconciliation gate before Friday cutover
  The Rollout section says: 'Start dual-write on a Monday and let it run for one week with full reconciliation. On cutover Friday after standup...' If dual-write starts Monday and runs 'one week', the natural completion is the following Monday, not Friday. Even if 'one week' means a five-day business week, success criteria require 'Full-state reconciliation diff ... is zero for 7 days before cutover'. With a Monday start and Friday cutover, there are only four nights of reconciliation data before cutover morning, making the seven-day-zero-diff gate impossible. The plan also says 'Start dual-write on a Monday' and Phase 2 is '1 week', but then Phase 3 begins on Friday, which is inconsistent with 'one week' unless the Friday is in the following week; that is not stated.
  Fix: Set an explicit calendar: e.g., start dual-write on Monday, run for at least seven full calendar days with nightly reconciliation, and begin canary cutover on the following Friday only after seven zero-diff reports. Alternatively reduce the gate to 'at least N days with zero diffs during the dual-write period' and state that number explicitly so the rollout is testable.

Scope Cop — Google: Gemini 3.7 Flash — FAIL

Lens: Design & simplicity

• [blocker] Per-minute lookahead query causes dropped renewals and race conditions
  Step 6 specifies selecting subscriptions due 'in the next minute'. If a query uses a strict forward window [now, now + 1 min], any scheduler hiccup or transient failure causes past-due subscriptions (current_period_end < now) to fall out of the query window and never renew. Conversely, if long-running payment processing does not update current_period_end within 60 seconds, the subsequent minute tick will re-fetch the exact same row, flooding queues with redundant jobs.
  Fix: Query subscriptions where current_period_end <= now and status = 'active', combined with transactional row-level leasing (SELECT FOR UPDATE SKIP LOCKED or a distinct processing_state column with a lease expiration timeout) to handle retries and prevent double-dispatch.
• [major] Custom sharded queue and token bucket scheduler is excessive overengineering
  Building a custom per-minute sharded queue infrastructure with token-bucket concurrency management for a 3-developer team in a 3-day Phase 1 window is high-friction overengineering. Stripe's standard rate limits (100 req/s in live mode) easily accommodate batch renewal with standard off-the-shelf worker queues or DB job tables with worker concurrency limits.
  Fix: Drop custom queue sharding. Use an existing standard job runner/queue library with basic worker pool concurrency limits, or use standard database-backed job polling with row-level locks.
• [major] Nightly full-state reconciliation breaks during canary phase
  Step 10 demands zero full-state diffs (including status, latest invoice/payment ID, cancel_at_period_end) against Stripe, but Step 12 explicitly cancels or pauses Stripe Subscription objects as cohorts migrate. As soon as canary cohorts move to the ledger and charge via standalone PaymentIntents, Stripe Subscription objects will be paused/canceled with no matching Stripe invoice, creating permanent full-state diffs that will continuously fail the zero-diff cutover gate.
  Fix: Segment reconciliation logic by cohort ownership: validate full state against Stripe Subscription objects only for unmigrated cohorts; for ledger-owned cohorts, reconcile ledger records against Stripe PaymentIntents and customer objects rather than Stripe Subscription records.
• [major] Undefined rollback path relies on impossible Stripe subscription reactivation
  Step 14 offers an unresolved fork: 'either reverse-sync changed subscription state back to Stripe before reactivating... or use forward-fix-only'. Canceled Stripe Subscription objects cannot be un-canceled or reactivated via the Stripe API—they must be recreated from scratch with custom billing_cycle_anchor calculations and prorations disabled. Leaving this unarchitected renders rollback untestable during Phase 0 drills.
  Fix: Explicitly adopt forward-fix-only or use Stripe's pause_collection (keeping Stripe subscriptions alive in a paused state until full cutover) rather than canceling them, ensuring a clean resumption path without recreating Stripe subscription objects.

Architect — Qwen: Qwen3.8 27B — FAIL

Lens: Execution & rollout

• [blocker] Dual-write week can double-charge because ledger scheduler ownership is undefined
  Phase 1 step 6 says the per-minute scheduler charges subscriptions whose current_period_end is due, while Phase 2 step 8 says the billing API writes every renewal to Stripe and the ledger. Phase 3 only cancels or pauses Stripe subscriptions when a cohort moves to ledger-owned renewals. The plan never says whether the ledger scheduler is disabled, shadow-only, or allowed to charge during the one-week dual-write/backfill window. A coding agent could enable the scheduler and cause Stripe and the ledger to charge the same subscription for the same period, or disable it and violate the stated goal of validating ledger renewals before cutover.
  Fix: Add an explicit ownership field such as billing_engine = stripe | ledger | shadow_ledger. During dual-write, run the ledger in shadow/no-charge mode or restrict the scheduler to ledger_owned=true only. Add an executable test that asserts no two charges exist for the same subscription_id and current_period_end across Stripe and ledger during the dual-write window.
• [blocker] Full-state reconciliation gate is unsatisfiable because expected invoice/payment/pending-event drift is not classified
  Phase 2 step 10 blocks cutover until zero unresolved full-state diffs remain, and it compares latest invoice/payment ID and pending event IDs. During dual-write, Stripe will have real subscription invoice/payment data while the ledger may only have pending events or no charges. After cutover, Stripe subscriptions are canceled or paused in step 12, so Stripe and ledger will intentionally diverge on latest invoice/payment IDs. The plan provides no rules for expected differences, mapping between Stripe and ledger identifiers, or how to mark a diff resolved.
  Fix: Define a per-phase reconciliation contract. During dual-write, compare lifecycle fields only or classify invoice/payment/pending-event diffs as expected until cutover. After cutover, treat the ledger as source of truth and reconcile Stripe only for payment-processor artifacts. Provide the exact diff classification logic and a report query that distinguishes expected drift from unresolved drift.
• [blocker] Backfill scope and timing leave reconciliation and rollback incomplete
  Phase 2 step 9 backfills only active and trialing Stripe subscriptions and says to run it before the dual-write window ends. The rollout starts dual-write on Monday and cutover on Friday after standup, so the backfill may occur after several nights of reconciliation. Missing past_due, unpaid, paused, canceled, incomplete, and incomplete_expired subscriptions will create full-state diffs and can also break rollback because step 14 snapshots only active Stripe subscriptions for the affected cohort.
  Fix: Backfill all non-deleted Stripe subscription states before dual-write starts, not before it ends. If some states are intentionally excluded, encode the exclusion in reconciliation and rollback. Include past_due, unpaid, paused, and canceled subscriptions in the pre-cutover snapshot and validation, and add checks for missing rows and state distribution mismatches.
• [major] Rollback runbook is not executable because failure classes, triggers, and reverse-sync steps are unspecified
  Phase 3 step 14 says the runbook must specify which protocol applies to each failure class, but the plan does not define the failure classes, metric thresholds, or exact reverse-sync actions. It says to stop the ledger scheduler, cancel incomplete PaymentIntents, mark pending events failed, and either reverse-sync or use forward-fix-only, but it does not say how to restore current_period_end, status, cancel_at_period_end, or payment method state on Stripe, nor how to handle renewals that were due while the scheduler was stopped. "Metrics look wrong" is not an operable trigger.
  Fix: Enumerate failure classes such as ledger outage, duplicate charge, missed renewal, stuck SCA/3DS, webhook drift, and reconciliation drift. Define exact rollback triggers, for example any duplicate charge, more than one failed renewal in 15 minutes, p95 renewal latency above 2s for 10 minutes, or unresolved full-state diff above zero. Provide the exact Stripe API calls and ledger SQL needed to restore or forward-fix each class.
• [major] Cohort cutover definition is too vague to guarantee exactly one billing engine per subscription
  Phase 3 step 11 says to start with 1% of renewals, but does not define whether the cohort is by customer, subscription, renewal batch, or hash. A customer with multiple subscriptions or renewals in different minutes can be split across Stripe and ledger. Step 12 says to cancel or pause the corresponding Stripe Subscription objects, but if the cohort is renewal-based, "corresponding" is ambiguous, and pausing a Stripe subscription may affect other subscriptions for the same customer. There is also no verification step after pause/cancel to prove exactly one engine owns each subscription.
  Fix: Define a deterministic cohort by customer_id hash or subscription_id hash, and ensure all subscriptions for a customer are assigned to the same cohort. Before each expansion, verify that the Stripe subscription is paused or canceled, the ledger ownership flag is set, and no mixed ownership exists. Add an alert for any subscription where both Stripe and ledger are active for the same billing period.

Tracer — Ox Alpha — FAIL

Lens: Data & integrity

• [major] Critic call failed
  HTTP 404: Thank you for participating in the Stealth Ox Alpha testing period. This model was ZAI's GLM-5.3 Flash. Use it now: https://openrouter.ai/z-ai/glm-5.3-flash
  Fix: Check provider config and API key.

Stress Test — Tencent: Hy-MT2-1.8B — FAIL

Lens: Security & permissions

• [blocker] Active Stripe subscriptions never cancelled causing double billing
  Phase 3 explicitly states Stripe Subscriptions must be canceled/paused as each ledger-owned cohort expands, yet no implementation is described
  Fix: Add implementation details for the pause/cancel logic in services/stripe-subscriptions.
• [blocker] Fire-and-forget dual-write leads to data divergence
  Replaced with transactional outbox, idempotent events, retries, and dead-letter handling but no verification that divergence is prevented
  Fix: Specify how duplicate events are detected and resolved in Phase 1.
• [major] Hourly renewal cron creates thundering herd and early-billing risk
  Replaces hourly scan with per-minute scheduler, but no handling of concurrency limits or charge initiation timing
  Fix: Add details on rate-limit handling and charge initiation logic.
• [major] Missing dunning, off-session 3DS/SCA and failed payment lifecycle
  Added past_due/payment_failed state tracking and requires_action flow in Phase 1, but no implementation details
  Fix: Describe the dunning flow and 3DS/SCA handling mechanisms.
• [minor] No monitoring for ledger write latency
  Added monitoring in Phase 0, but no thresholds or alerting defined
  Fix: Specify p50/p95 latency thresholds and alert conditions.

Round 3

Red Team — deepseek-v4-flash — FAIL

Lens: Design & simplicity

• [blocker] Scheduler selects past_due rows without respecting next_retry_at and dunning state
  Phase 1 step 6 says the per-minute scheduler selects subscriptions with status active, trialing, or past_due when current_period_end <= now. Step 7 then says that when a PaymentIntent returns requires_action you must mark the subscription past_due, set next_retry_at, and 'do not retry the charge until the PaymentIntent is confirmed or the dunning schedule advances.' For a failed past_due subscription, current_period_end is already <= now, so the scheduler will find that row every minute and immediately attempt another charge, completely bypassing the 3/7/14-day dunning schedule. This is both a design contradiction and a real double-billing/retry hazard.
  Fix: Add dunning-aware predicates to the scheduler selection, e.g. only select past_due rows when next_retry_at <= now AND no unresolved PaymentIntent exists for that subscription; alternatively select only active/trialing rows for normal renewal and drive past_due retries from a separate dunning job. Also add an integration test proving a failed requires_action payment is not retried before the dunning wait expires.
• [blocker] No ownership-based routing stops dual-write to Stripe Subscriptions API after cohort cutover
  Phase 2 step 8 states the billing API writes every lifecycle event to Stripe and the ledger during dual-write. Phase 3 moves cohorts to ledger ownership and pauses/cancels the corresponding Stripe Subscription objects, but the plan never specifies that billing API calls for billing_engine='ledger' rows must stop calling the Stripe Subscriptions endpoints. Without that routing switch, a plan change, payment-method update, or trial conversion for a migrated subscription will still attempt to write to a paused/canceled Stripe Subscription. That violates the exactly-one-owner invariant, can fail or inadvertently modify Stripe state, and makes the canary cutover unsafe.
  Fix: Add an explicit ownership gate in the billing API/outbox producer: for rows with billing_engine='ledger', subscription lifecycle operations are written only to the ledger and Stripe is touched only for PaymentIntents/payment methods, not for Subscription objects. Make this gate part of the cohort cutover business logic and verify it with the phase-3 exactly-one-engine checks.
• [major] shadow_ledger state is dead weight and shadow-mode scheduler behavior is contradictory
  Phase 1 step 3 introduces a billing_engine enum with 'stripe', 'ledger', and 'shadow_ledger', but nothing in the plan ever says how a row enters shadow_ledger or what writes/reads apply to it. Step 6 says during dual-write the scheduler runs in shadow mode and only charges rows where billing_engine='ledger', while billing_engine remains 'stripe' and the scheduler records pending events without creating PaymentIntents. Since no rows are set to 'ledger' until Phase 3, the scheduler during Phase 2 would never have a chargeable row, and 'recording pending events' duplicates the transactional outbox from step 8. This is unnecessary complexity and creates ambiguity about what happens if any row is accidentally set to shadow_ledger.
  Fix: Remove shadow_ledger from the enum and delete all 'shadow scheduler' code paths from Phase 1/2. Keep billing_engine as a simple two-state flag (stripe/ledger), leave the ledger scheduler disabled during dual-write, and use the outbox + daily reconciliation as the shadow-validation mechanism. Start the scheduler only when Phase 3 moves rows to billing_engine='ledger'.
• [minor] Optimistic-locking version field and stripe_pause_* columns lack defined semantics
  Phase 1 step 3 adds a version column and stripe_pause_status/stripe_pause_event_id columns, but Phase 3 only says to record the Stripe pause/cancel status and event id. The plan never defines how version is incremented or checked, which endpoint uses it, or what happens on a version conflict. Similarly, the pause columns are written during cutover but are not reconciled or read anywhere else, so they can silently drift from Stripe's actual pause state.
  Fix: Define explicit optimistic concurrency semantics for version (every update must compare-and-set version; on mismatch return a retryable conflict) or remove it. For stripe_pause_status/stripe_pause_event_id, specify that they are authoritative for rollback and are verified during each phase-3 expansion and by the ownership-aware reconciliation; otherwise strip them from the schema to reduce surface area.

Scope Cop — Google: Gemini 3.7 Flash — FAIL

Lens: Execution & rollout

• [blocker] Scheduler query ignores next_retry_at, causing immediate retry loops on failed renewals
  Step 6 defines the scheduler query as selecting subscriptions where current_period_end <= now and status is in (active, trialing, past_due). In Step 7, failed renewals and 3DS requires_action flows mark the subscription past_due, set next_retry_at (e.g., +3 days), and forbid retrying until that time. However, because current_period_end remains in the past, the per-minute scheduler query will re-select and re-attempt to lease and process these past_due subscriptions every minute, violating the dunning backoff policy.
  Fix: Update Step 6 scheduler query to filter for current_period_end <= now AND (next_retry_at IS NULL OR next_retry_at <= now) so dunning backoffs are respected.
• [major] Backfill migration runs before dual-write starts, creating a blind mutation gap
  Step 9 specifies running the backfill migration 'Before the dual-write window starts'. Any subscription creations, cancellations, payment method updates, or renewals that occur in Stripe between the time the backfill extract begins and the moment Step 8's transactional outbox is deployed and enabled will be missed, resulting in stale or missing ledger records prior to cutover.
  Fix: Enable the transactional outbox dual-write path in the billing API first (writing to ledger with idempotent upserts), and then run the historical backfill with updated_at/version checks to prevent overwriting newer outbox events.
• [major] Canary cohort 2-hour verification gate is incompatible with asynchronous SCA/3DS flows
  Step 11 sets 2-hour verification gates between canary cohort expansions, and Step 15 enforces a gate of 'zero failed renewals on cutover dashboard... before expanding'. Under Step 7, cards requiring SCA/3DS transition to past_due and dispatch an email/hosted flow to the customer with a multi-day dunning window. If past_due/requires_action is counted as a failure on the dashboard, legitimate SCA challenges will permanently block cohort expansion within 2 hours; if it is not counted as a failure, cohorts expand without verifying that off-session confirmation and webhook resolution actually succeed.
  Fix: Explicitly distinguish in Step 15 metrics between hard payment failures (which block canary expansion) and pending SCA requires_action states, and define synthetic SCA end-to-end verification during the 1% canary window rather than relying solely on passive 2-hour organic traffic.

Architect — Qwen: Qwen3.8 27B — FAIL

Lens: Data & integrity

• [blocker] Renewal idempotency key is not a stable billing-period identity
  Phase 6 derives each renewal idempotency key from (subscription_id, current_period_end). That is not a unique billing period: current_period_end can be unchanged across dunning retries, can be altered by plan/proration/trial/manual corrections, and does not identify the period start. A dunning retry after 3/7/14 days would collide with the original failed attempt, while a period mutation could create a new key for the same billing period and allow a second charge.
  Fix: Add a durable renewal_id or store period_start/period_end and use an idempotency key such as subscription_id + period_start + period_end + attempt_number. Enforce a unique successful-charge constraint on the billing period, not just the mutable current_period_end.
• [blocker] Scheduler can dispatch past_due renewals before dunning or SCA resolution
  Phase 6 selects rows where current_period_end <= now and status is active, trialing, or past_due, but it does not filter on next_retry_at, an open requires_action PaymentIntent, or an in-flight payment attempt. Phase 7 says not to retry until the PaymentIntent is confirmed or the dunning schedule advances, yet the scheduler query can still pick the row every minute and create conflicting attempts or duplicate dispatch work.
  Fix: Make the scheduler predicate explicit: current_period_end <= now AND (next_retry_at IS NULL OR next_retry_at <= now) AND no open requires_action/in-flight PaymentIntent. Use the lease/processing state to serialize attempts per subscription and per billing period.
• [blocker] Backfill and reconciliation baseline do not match the fields being compared
  Phase 9 backfills payment method IDs, current_period_end, cancel_at_period_end, and trial end, but Phase 10 reconciles latest invoice/payment ID and pending event IDs. The plan never populates latest invoice/payment IDs during backfill, and it does not define how Stripe webhook-driven renewals produce the outbox events required for the 'expected transient drift' classification. Normal Stripe renewals or webhook lag can therefore appear as unresolved diffs and block cutover, or be silently ignored if not classified.
  Fix: Backfill latest invoice/payment IDs and any pending event identifiers. Define webhook ingestion as an idempotent event log or outbox path, and classify webhook lag with an explicit TTL. Define exactly what pending event IDs means in both Stripe and ledger.
• [blocker] Migrated-row reconciliation can miss a still-active Stripe subscription
  Phase 10 says migrated rows are compared only against Stripe PaymentIntents, customer records, and payment method identifiers, and that Stripe Subscription pause/cancel status is treated as expected. That omits the dangerous case: ledger owns the subscription while the Stripe subscription is still active because pause_collection failed, was unsupported, or was not verified. Phase 3 has an alert during expansion, but the 30-day cleanup gate in Phase 4 relies on ownership-aware reconciliation, which would not catch this double-billing state.
  Fix: For migrated rows, assert that the Stripe subscription is paused or canceled, or explicitly alert/block if it is active. Do not treat all Stripe Subscription status as expected; only pause/cancel should be acceptable, and active should be an unresolved ownership diff.
• [blocker] Cutover and rollback ownership changes are not atomic and lack a safe fallback
  Phase 12 calls Stripe pause_collection, records pause status, and sets billing_engine='ledger' as separate operations, with no transactional or verified state machine. If pause fails or is unsupported, the plan has no safe intermediate state, yet Phase 11 requires Stripe to be paused/canceled before ledger ownership. Phase 14 also reverse-syncs ledger state to paused Stripe subscriptions and recreates canceled Stripe subscriptions without checking whether the ledger subscription itself is canceled, which can resurrect a canceled customer or create inconsistent billing anchors.
  Fix: Use a two-phase cutover: mark shadow_ledger, pause Stripe, verify pause, then set ledger. If pause fails, remain stripe and alert. Rollback must branch on ledger status: do not recreate or resume Stripe billing for ledger-canceled subscriptions, and only reverse-sync fields that Stripe can safely update.

Tracer — Ox Alpha — FAIL

Lens: Security & permissions

• [major] Critic call failed
  HTTP 404: Thank you for participating in the Stealth Ox Alpha testing period. This model was ZAI's GLM-5.3 Flash. Use it now: https://openrouter.ai/z-ai/glm-5.3-flash
  Fix: Check provider config and API key.

Stress Test — Tencent: Hy-MT2-1.8B — PASS

Lens: UX & edge cases

No issues raised.

Revised Plan — after Round 1

Judge dispositions (addressed 15 · deferred 0 · rejected 2):

• [REJECTED] Unparseable critique — Not a plan critique; the critic returned non-JSON, so no actionable issue.
• [ADDRESSED] Active Stripe subscriptions are never cancelled, causing double billing — Folded into Phase 3 canary cutover: Stripe Subscriptions are canceled/paused as each cohort moves ledger-owned.
• [ADDRESSED] Rollback is impossible without a reverse-sync mechanism — Added Phase 3 rollback protocol with snapshot, in-flight charge handling, reverse-sync or forward-fix.
• [ADDRESSED] Missing dunning, off-session 3DS/SCA, and failed payment lifecycle — Added past_due/payment_failed state handling, requires_action flow, and retry policy in Phase 1.
• [ADDRESSED] No backfill migration for existing Stripe subscriptions — Added Phase 2 backfill migration for existing Stripe subscriptions before cutover.
• [ADDRESSED] Fire-and-forget dual-write guarantees data divergence — Replaced fire-and-forget with transactional outbox, idempotent events, retries, and dead-letter.
• [ADDRESSED] Hourly renewal cron creates a thundering herd and early-billing risk — Replaced hourly scan with per-minute scheduler, sharded queues, rate-limit handling, and idempotency keys.
• [ADDRESSED] Fire-and-forget dual-write makes ledger state unreliable — Replaced fire-and-forget with transactional outbox; ledger unavailable handled by retry/dead-letter and no orphan charges.
• [ADDRESSED] Count-only reconciliation hides dangerous state drift — Reconciliation now compares full state and gates cutover on zero unresolved diffs.
• [ADDRESSED] One-day cutover and flag rollback are not a safe migration — Added canary cohorts, staged gates, and explicit rollback runbook.
• [ADDRESSED] Subscription schema and renew endpoint are not idempotent or auditable — Extended schema with Stripe ids, version, payment error, and idempotency keys on all endpoints.
• [REJECTED] Critic call failed — Not a plan critique; the critic call failed with HTTP 404, so no actionable issue.
• [ADDRESSED] Missing duplicate event detection for Stripe webhooks — Added ledger-side subscription event id/idempotency key and webhook dedupe by both Stripe and ledger ids.
• [ADDRESSED] No validation for status transitions — Added state transition validation rejecting invalid cancel/renew combinations.
• [ADDRESSED] No monitoring for ledger write latency — Added Phase 0 monitoring for ledger write latency and alerts.
• [ADDRESSED] No fallback for ledger failure — Added circuit-breaker/outbox retry and dead-letter behavior so ledger unavailability does not block API or create orphan charges.
• [ADDRESSED] No test environment for cutover — Added Phase 0 staging environment and cutover drills.

Migrate subscription billing from Stripe to the internal ledger

Goal

Move all subscription billing (creation, renewal, cancellation) off Stripe's Subscriptions API
onto our internal ledger service, keeping Stripe as the payment processor only. Target: 100% of
renewals flowing through the ledger by Oct 31.

Background

The ledger service (services/ledger) already records one-time invoices. It does not yet own
subscription lifecycles. The billing team (3 devs) owns both codebases.

Phases

Phase 0 — Staging and safety net before dual-write

• Deploy the same ledger schema, endpoints, scheduler, webhook handler, and ledger_billing
  feature flag to a staging environment with production-like Stripe test mode, payment method
  fixtures, and SCA/3DS test cards. Run full cutover drills in staging and document the rollback
  runbook.
• Add monitoring before cutover: ledger write latency p50/p95 (alert when p95 > 500ms), duplicate
  event rate, scheduler lag, repeat renewal attempt rate, and a circuit-breaker/fallback status
  metric for ledger calls.

Phase 1 — Ledger subscription support (3 days)

• Extend the subscriptions table with the original NOT NULL fields (id, customer_id,
  plan_id, status, current_period_end, cancel_at_period_end) plus stripe_subscription_id,
  stripe_payment_method_id, version, updated_at, and last_payment_error. The status field
  must support at least active, past_due, canceled, and paused.
  [DECISION: add an idempotency-keyed subscription event/outbox table to the ledger schema; it must
  record subscription_id, event type, idempotency key, payload version, processed_at, and external
  Stripe event id where available.]
• Implement ledger endpoints: POST /subscriptions, POST /subscriptions/:id/cancel,
  POST /subscriptions/:id/renew. Every endpoint requires an idempotency key and derives a
  deterministic event key from it. Duplicate events return the stored result and do not re-apply
  side effects.
• Enforce state transition validation: cancel and renew are rejected for subscriptions whose
  status is canceled; renew on past_due is allowed only after payment resolution; cancel
  sets cancel_at_period_end and does not renew beyond the current period.
• Replace the hourly renewal cron with a per-minute scheduler. It selects subscriptions whose
  current_period_end is due in the next minute, shards them by subscription_id into worker
  queues, and applies a Stripe rate-limit-aware concurrency/token bucket. Each renewal uses an
  idempotency key derived from (subscription_id, current_period_end), retries transient failures
  with exponential backoff, records a pending event before charging, and stores invoice/payment IDs
  after the Stripe PaymentIntent succeeds.
• Implement off-session SCA/3DS handling: when a PaymentIntent returns requires_action, mark
  the subscription past_due with last_payment_error, trigger the customer action flow, and do
  not retry the charge until the PaymentIntent is confirmed or the dunning schedule advances. Add
  payment-failure state tracking and retry/backoff policy documented in the ledger.

Phase 2 — Dual-write and backfill (1 week)

• Billing API writes every subscription creation, renewal, cancellation, and status change event
  to Stripe and the ledger using a transactional outbox, not fire-and-forget: the event is written
  in the same database transaction as the local change, and a background worker publishes it to the
  ledger with retries, dead-letter handling, and alerting. The ledger rejects duplicate event keys
  from Phase 1. If the ledger is unavailable, the worker retries with exponential backoff and
  dead-letters the event; local billing API remains available, but no new ledger-owned renewals are
  charged until the event is durably recorded.
• Before the dual-write window ends, run a backfill migration for existing Stripe subscriptions:
  extract all active and trialing Stripe Subscription objects, payment method IDs,
  current_period_end, and cancel_at_period_end from Stripe and upsert them into the ledger
  subscriptions table preserving stripe_subscription_id. Validate row count, plan ids, period
  timestamps, and status distributions against Stripe before Phase 3.
• Nightly reconciliation compares full subscription state between Stripe and the ledger, not just
  counts: customer_id, plan_id, status, current_period_end, cancel_at_period_end, latest
  invoice/payment ID, and pending event IDs. Write the diff to the S3 CSV report and emit
  near-real-time alerts to a billing drift channel. Cutover is blocked until zero unresolved
  full-state diffs remain on cutover morning.

Phase 3 — Canary cutover with rollback protocol

• Do not rely on a one-step global flag flip. Move customer cohorts through ledger_billing in
  stages on cutover day: start at 1% of renewals, verify for two hours, then expand to 25%, 50%,
  and 100% with the phase-0 metrics as gates.
• As each cohort moves to ledger-owned renewals, cancel or pause the corresponding Stripe
  Subscription objects (pause_collection where available) so Stripe does not autonomously
  renew. Record the Stripe cancellation status and event id on the ledger subscription row, and
  ensure exactly one billing engine owns each subscription.
• Stripe webhooks are handled by the new handler in services/billing-webhooks and deduplicated
  by both Stripe event id and ledger-side subscription event id; state changes already present in
  the ledger are ignored.
• Rollback protocol: before cutover, snapshot active Stripe subscriptions for the affected cohort.
  If metrics look wrong or a cohort fails, pause expansion and follow the runbook:
  - stop the ledger scheduler before any rollback;
  - cancel incomplete Stripe PaymentIntent charges and mark ledger pending events as failed;
  - for completed ledger renewals whose Stripe subscription was already canceled, either
  reverse-sync changed subscription state back to Stripe before reactivating the Stripe
  subscription, or use forward-fix-only where ledger remains the source of truth and the
  nightly reconciliation resolves every affected subscription before the next cycle.
  The runbook must specify which protocol applies to each failure class.
• Post-cutover monitoring: zero failed renewals on cutover dashboard, p95 renewal latency under
  2s, zero duplicate charges from idempotency violations, and no dropped renewals for a cohort
  before expanding.

Phase 4 — Cleanup (later)

• Remove the dual-write path and the Stripe Subscriptions integration only after 30 consecutive
  days of zero full-state reconciliation diffs and zero failed renewals; keep Stripe
  PaymentIntents as the payment processor.

Success criteria

• Zero failed renewals on the cutover dashboard.
• Full-state reconciliation diff (not count-only) is zero for 7 days before cutover and trends to
  zero during the dual-write week.
• Renewal latency p95 under 2s.
• No duplicate charge events due to missing idempotency/deduplication.

Rollout

Run the Phase 0 staging cutover drill first. Start dual-write on a Monday and let it run for one
week with full reconciliation. On cutover Friday after standup, use the customer-cohort gates from
Phase 3: 1% for two hours, then 25%, 50%, 100% after passing the Phase 0 metrics. Ping #billing in
Slack at each expansion and when done. If any cohort shows a failed renewal or duplicate charge,
pause expansion and follow the rollback runbook.

Revised Plan — after Round 2

Judge dispositions (addressed 18 · deferred 0 · rejected 2):

• [ADDRESSED] Full-state reconciliation criterion is incompatible with Stripe Subscription cancellation/pausing after migration — Reconciliation now segments non-migrated vs migrated subscriptions; migrated rows are compared against Stripe PaymentIntents/customer records, not Stripe Subscription objects.
• [ADDRESSED] Rollback protocol relies on reactivating a canceled Stripe Subscription, which Stripe does not support — Rollback now prefers Stripe pause_collection and specifies new-Subscription recreation with billing_cycle_anchor, mapping, and duplicate-charge prevention if cancel was used.
• [ADDRESSED] Backfill and status enum omit trialing and other non-terminal Stripe subscription states — Status enum now includes trialing/unpaid/incomplete, and backfill covers all non-deleted Stripe states with explicit mapping before dual-write.
• [ADDRESSED] Subscription lifecycle coverage is incomplete: no plan for changes, proration, payment-method updates, or trial conversion — Added a subscription change/update endpoint and included plan/quantity/proration/payment-method/trial-conversion events in dual-write, webhooks, and reconciliation.
• [ADDRESSED] Rollout timeline cannot satisfy the 7-day reconciliation gate before Friday cutover — Rollout now requires seven full days of zero-diff reconciliations before cutover, with cutover on the following Friday after completing the seven-day window.
• [ADDRESSED] Per-minute lookahead query causes dropped renewals and race conditions — Scheduler query changed to current_period_end <= now with row-level leasing and SKIP LOCKED to prevent missed renewals and duplicate dispatch.
• [ADDRESSED] Custom sharded queue and token bucket scheduler is excessive overengineering — Removed custom sharded queues/token-bucket infrastructure; scheduler now uses standard DB-backed jobs with worker pool concurrency and Stripe rate limits.
• [ADDRESSED] Nightly full-state reconciliation breaks during canary phase — Reconciliation is segmented by ownership; migrated cohorts reconcile against PaymentIntents/customers rather than Stripe Subscription objects.
• [ADDRESSED] Undefined rollback path relies on impossible Stripe subscription reactivation — Rollback now uses pause_collection and defines exact recreation/protection steps for canceled subscriptions.
• [ADDRESSED] Dual-write week can double-charge because ledger scheduler ownership is undefined — Added billing_engine ownership flag and lock the ledger scheduler to shadow mode during dual-write; no ledger-owned charges until cutover.
• [ADDRESSED] Full-state reconciliation gate is unsatisfiable because expected invoice/payment/pending-event drift is not classified — Defined per-phase reconciliation contract with expected vs unresolved drift classification and ownership-aware comparison.
• [ADDRESSED] Backfill scope and timing leave reconciliation and rollback incomplete — Backfill now runs before dual-write and includes all non-deleted Stripe states, with pre-dual-write validation and snapshots.
• [ADDRESSED] Rollback runbook is not executable because failure classes, triggers, and reverse-sync steps are unspecified — Rollback runbook now enumerates failure classes, triggers, and exact pause-resume/recreation/forward-fix steps.
• [ADDRESSED] Cohort cutover definition is too vague to guarantee exactly one billing engine per subscription — Cutover cohort is now deterministic by customer_id hash; all subscriptions for a customer move together and exactly-one-engine verification is required before each expansion.
• [REJECTED] Critic call failed — This is an infrastructure/critic-call failure, not a plan issue.
• [ADDRESSED] Active Stripe subscriptions never cancelled causing double billing — Added implementation details for pause/cancel via Stripe pause_collection and record pause status/event id on ledger row.
• [ADDRESSED] Fire-and-forget dual-write leads to data divergence — Duplicate events are rejected by ledger idempotency keys and verified by reconciliation/alerts; Phase 1 and Phase 2 now state this explicitly.
• [ADDRESSED] Hourly renewal cron creates thundering herd and early-billing risk — Charge initiation and concurrency limits are now specified: charge only on due rows with leases and worker pool constrained by Stripe rate limits.
• [ADDRESSED] Missing dunning, off-session 3DS/SCA and failed payment lifecycle — Dunning/3DS handling expanded with retry schedule, failure counter/next_retry_at, and SCA confirmation flow.
• [REJECTED] No monitoring for ledger write latency — Phase 0 already defines p95 > 500ms alert; no further threshold is required.

Migrate subscription billing from Stripe to the internal ledger

Goal

Move all subscription billing (creation, renewal, cancellation) off Stripe's Subscriptions API
onto our internal ledger service, keeping Stripe as the payment processor only. Target: 100% of
renewals flowing through the ledger by Oct 31.

Background

The ledger service (services/ledger) already records one-time invoices. It does not yet own
subscription lifecycles. The billing team (3 devs) owns both codebases.

Phases

Phase 0 — Staging and safety net before dual-write

• Deploy the same ledger schema, endpoints, scheduler, webhook handler, and ledger_billing
  feature flag to a staging environment with production-like Stripe test mode, payment method
  fixtures, and SCA/3DS test cards. Run full cutover drills in staging and document the rollback
  runbook.
• Add monitoring before cutover: ledger write latency p50/p95 (alert when p95 > 500ms), duplicate
  event rate, scheduler lag, repeat renewal attempt rate, and a circuit-breaker/fallback status
  metric for ledger calls.

Phase 1 — Ledger subscription support (3 days)

• Extend the subscriptions table with the original NOT NULL fields (id, customer_id,
  plan_id, status, current_period_end, cancel_at_period_end) plus stripe_subscription_id,
  stripe_payment_method_id, version, updated_at, and last_payment_error. Add
  billing_engine NOT NULL DEFAULT 'stripe' with allowed values 'stripe', 'ledger',
  'shadow_ledger'; add trial_end nullable timestamp; add payment_failure_count integer default 0
  and next_retry_at nullable timestamp; add stripe_pause_status and stripe_pause_event_id
  nullable columns. The status field must support at least active, past_due, canceled,
  paused, trialing, unpaid, and incomplete.
  [DECISION: add an idempotency-keyed subscription event/outbox table to the ledger schema; it must
  record subscription_id, event type, idempotency key, payload version, processed_at, and external
  Stripe event id where available.]
• Implement ledger endpoints: POST /subscriptions, POST /subscriptions/:id/cancel,
  POST /subscriptions/:id/renew, and POST /subscriptions/:id/change for plan/price/quantity
  changes, proration, trial conversion, and payment method updates. Every endpoint requires an
  idempotency key and derives a deterministic event key from it. Duplicate events return the stored
  result and do not re-apply side effects.
• Enforce state transition validation: cancel and renew are rejected for subscriptions whose
  status is canceled; renew on past_due is allowed only after payment resolution; cancel
  sets cancel_at_period_end and does not renew beyond the current period; change is rejected
  for canceled subscriptions, trial conversion sets trial_end to null and status to active,
  and payment-method updates are allowed for any non-canceled status.
• Replace the hourly renewal cron with a per-minute scheduler. It selects subscriptions whose
  current_period_end <= now and status is one of active, trialing, or past_due, and uses
  transactional row-level leasing (SELECT ... FOR UPDATE SKIP LOCKED or a lease-expiry
  processing_state/lease column) to prevent missed renewals and duplicate dispatch. It does not
  use custom sharded queues or bespoke token-bucket infrastructure; it uses standard DB-backed job
  queues or a standard worker pool. Concurrency is limited by Stripe's documented rate limits
  (100 requests/sec live mode) and per-worker max in-flight. Charge initiation happens only on due
  rows; no early billing. During dual-write before cutover the scheduler runs in shadow mode and
  only charges rows where billing_engine='ledger'; until a cohort is cut over billing_engine
  stays 'stripe' and the scheduler records pending events without creating PaymentIntents. Each
  renewal uses an idempotency key derived from (subscription_id, current_period_end), retries
  transient failures with exponential backoff, records a pending event before charging, and stores
  invoice/payment IDs after the Stripe PaymentIntent succeeds.
• Implement off-session SCA/3DS and dunning handling. When a PaymentIntent returns
  requires_action, mark the subscription past_due with last_payment_error, increment
  payment_failure_count, set next_retry_at, trigger the customer action flow (SCA/3DS email or
  hosted confirmation), and do not retry the charge until the PaymentIntent is confirmed or the
  dunning schedule advances. Default dunning schedule: first retry 3 days after failure, second
  retry 7 days after the previous retry, final retry 14 days after that; after the final failed
  retry mark the subscription canceled and send the cancellation event. Document the policy in
  the ledger.

Phase 2 — Dual-write and backfill (1 week)

• Billing API writes every subscription creation, renewal, cancellation, status change, plan/price
  or quantity change, proration, payment method update, and trial conversion event to Stripe and
  the ledger using a transactional outbox, not fire-and-forget: the event is written in the same
  database transaction as the local change, and a background worker publishes it to the ledger with
  retries, dead-letter handling, and alerting. The ledger rejects duplicate event keys from Phase 1.
  During the dual-write window billing_engine remains 'stripe' for existing and new
  subscriptions, and the ledger scheduler runs in shadow/no-charge mode: it records pending events
  and ingests outbox events but does not create PaymentIntents or charge. If the ledger is
  unavailable, the worker retries with exponential backoff and dead-letters the event; local billing
  API remains available, but no new ledger-owned renewals are charged until the event is durably
  recorded. Add an executable test asserting that during dual-write no two successful charges exist
  for the same subscription_id and current_period_end across Stripe and ledger.
• Before the dual-write window starts, run a backfill migration for all non-deleted Stripe
  subscriptions. Extract every active, trialing, past_due, unpaid, paused, incomplete, and canceled
  Stripe Subscription object, payment method IDs, current_period_end, cancel_at_period_end,
  and trial end where present, and upsert them into the ledger subscriptions table preserving
  stripe_subscription_id. Map Stripe statuses as active->active, trialing->trialing,
  past_due->past_due, unpaid->unpaid, paused->paused, incomplete->incomplete,
  canceled->canceled, and incomplete_expired->canceled. Set billing_engine='stripe' for
  all backfilled rows. Validate row count, plan IDs, period timestamps, status distributions, and
  missing-row check against Stripe before Phase 3.
• Nightly reconciliation compares subscription state according to ownership. For non-migrated
  rows (billing_engine='stripe') compare full subscription state between Stripe and the ledger:
  customer_id, plan_id, status, current_period_end, cancel_at_period_end, latest invoice/payment
  ID, and pending event IDs. During dual-write a transient missing outbox event or in-flight
  payment ID is classified as expected only if the outbox event is published and clears the next
  reconciliation; any unresolved lifecycle or payment/id mismatch is unresolved. Cutover is blocked
  until all non-migrated rows have zero unresolved full-state diffs for seven consecutive nights
  before cutover morning. For migrated rows (billing_engine='ledger') do not compare against
  Stripe Subscription objects; compare the ledger record against Stripe PaymentIntents, customer
  records, and payment method identifiers only, and treat Stripe Subscription pause/cancel status as
  expected. Write the diff to the S3 CSV report and emit near-real-time alerts to a billing drift
  channel.

Phase 3 — Canary cutover with rollback protocol

• Do not rely on a one-step global flag flip. Move customer cohorts through ledger_billing in
  stages on cutover day. Cohort assignment is deterministic by customer_id hash, so all
  subscriptions for a customer move together. Start at 1% of customers, verify for two hours, then
  expand to 25%, 50%, and 100% with the phase-0 metrics as gates. Before each expansion verify
  exactly one billing engine owns each subscription: Stripe subscription is paused/canceled when
  the ledger owns it, billing_engine matches the assigned cohort, and no mixed ownership exists.
  Alert on any subscription where both Stripe and ledger are active for the same billing period.
• As each cohort moves to ledger-owned renewals, call Stripe pause_collection on the
  corresponding Stripe Subscription objects wherever the API supports it. Do not cancel Stripe
  subscriptions during canary cutover unless pause_collection is unavailable for that state and
  the failure class explicitly requires cancellation. Record the Stripe pause/cancel status and
  event id on the ledger subscription row (stripe_pause_status, stripe_pause_event_id), set
  billing_engine='ledger', and ensure exactly one billing engine owns each subscription.
• Stripe webhooks are handled by the new handler in services/billing-webhooks and deduplicated
  by both Stripe event id and ledger-side subscription event id; state changes already present in
  the ledger are ignored. Webhook handling includes plan/price/quantity changes, proration, payment
  method updates, and trial conversion events.
• Rollback protocol: before cutover, snapshot active Stripe subscriptions for the affected cohort.
  Defined failure classes and triggers:
  - duplicate charge: any subscription with two successful charges for the same
  (subscription_id, current_period_end) across Stripe and ledger.
  - more than one failed renewal in 15 minutes.
  - p95 renewal latency above 2s for 10 minutes.
  - any unresolved non-migrated full-state diff above zero.
  - ledger outage or outbox publication lag greater than 5 minutes.
  - stuck SCA/3DS PaymentIntent still requires_action after 24 hours.
  On trigger, pause expansion and follow the runbook:
  - stop the ledger scheduler before any rollback;
  - cancel incomplete Stripe PaymentIntent charges and mark ledger pending events as failed;
  - for paused Stripe subscriptions, end pause_collection and reverse-sync
  current_period_end, status, cancel_at_period_end, and payment method state from the ledger
  back to Stripe before resuming Stripe-owned billing;
  - for canceled Stripe subscriptions, create a new Stripe Subscription with the same customer,
  plan/quantity, and billing_cycle_anchor aligned to the ledger's current period, proration
  disabled, map the new stripe_subscription_id in the ledger, and reconcile/credit any overlap
  before resuming Stripe-owned billing;
  - otherwise use forward-fix-only where ledger remains the source of truth, and the ownership-aware
  nightly reconciliation resolves every affected subscription before the next cycle.
  The runbook must specify which protocol applies to each failure class.
• Post-cutover monitoring: zero failed renewals on cutover dashboard, p95 renewal latency under
  2s, zero duplicate charges from idempotency violations, and no dropped renewals for a cohort
  before expanding.

Phase 4 — Cleanup (later)

• Remove the dual-write path and the Stripe Subscriptions integration only after 30 consecutive
  days of zero unresolved ownership-aware reconciliation diffs (for non-migrated rows vs Stripe
  Subscriptions, for migrated rows vs Stripe PaymentIntents/customers) and zero failed renewals;
  keep Stripe PaymentIntents as the payment processor.

Success criteria

• Zero failed renewals on the cutover dashboard.
• Ownership-aware reconciliation: non-migrated full-state diff is zero for 7 days before cutover and
  trends to zero during the dual-write week; migrated rows reconcile against Stripe PaymentIntents
  and customer records, not Stripe Subscription objects.
• Renewal latency p95 under 2s.
• No duplicate charge events due to missing idempotency/deduplication.

Rollout

Run the Phase 0 staging cutover drill first. Start dual-write on Monday of week 1 and run it for at
least seven full calendar days with nightly ownership-aware reconciliation. Begin canary cutover on
Friday of week 2 only after seven consecutive zero-diff reports for non-migrated subscriptions and
all Phase 0 metrics passing. Use the customer-cohort gates from Phase 3: 1% of customers for two
hours, then 25%, 50%, 100% after passing the Phase 0 metrics and exactly-one-engine verification.
Ping #billing in Slack at each expansion and when done. If any cohort shows a failed renewal or
duplicate charge, pause expansion and follow the rollback runbook.

Revised Plan — after Round 3

Judge dispositions (addressed 12 · deferred 0 · rejected 2):

• [ADDRESSED] Scheduler selects past_due rows without respecting next_retry_at/dunning state — Updated step 6 scheduler predicate to respect next_retry_at and exclude in-flight/open PaymentIntents, and added an integration test.
• [ADDRESSED] No ownership-based routing stops dual-write to Stripe Subscriptions API after cohort cutover — Added explicit billing_engine ownership gate in step 8: ledger-owned rows write only to ledger; Stripe touched only for PaymentIntents/payment method identifiers.
• [ADDRESSED] shadow_ledger state is dead weight and shadow-mode scheduler behavior contradictory — Removed shadow_ledger from the enum and removed shadow scheduler; step 6 disables scheduler during dual-write.
• [ADDRESSED] Optimistic-locking version field and stripe_pause columns lack defined semantics — Defined version compare-and-set semantics and made stripe_pause columns authoritative for rollback/reconciliation in step 3.
• [ADDRESSED] Scheduler query ignores next_retry_at causing immediate retry loops on failed renewals — Same scheduler predicate update in step 6.
• [ADDRESSED] Backfill migration runs before dual-write starts creating a blind mutation gap — Step 9 now enables the outbox first and uses updated_at/version checks during backfill.
• [ADDRESSED] Canary cohort 2-hour verification gate incompatible with asynchronous SCA/3DS flows — Step 15 now distinguishes hard failures from pending SCA requires_action and adds synthetic SCA E2E verification in the 1% window.
• [ADDRESSED] Renewal idempotency key is not a stable billing-period identity — Step 6 now uses subscription_id + current_period_start + current_period_end + attempt_number and forbids duplicate successful charges per billing period.
• [ADDRESSED] Scheduler can dispatch past_due renewals before dunning or SCA resolution — Step 6 now also filters out in-flight attempts and open requires_action PaymentIntents.
• [ADDRESSED] Backfill and reconciliation baseline do not match the fields being compared — Step 9 backfills latest invoice/payment IDs and pending event identifiers; step 10 defines pending event IDs and idempotent TTL classification.
• [ADDRESSED] Migrated-row reconciliation can miss a still-active Stripe subscription — Step 10 now asserts corresponding Stripe Subscription is paused/canceled and treats active as unresolved; Phase 4 gate updated.
• [ADDRESSED] Cutover and rollback ownership changes are not atomic and lack a safe fallback — Step 12 verifies pause/cancel before setting billing_engine='ledger'; step 14 branches on ledger canceled status before rollback.
• [REJECTED] Critic call failed HTTP 404 — Provider/API error for the critic model; no plan change is actionable.
• [REJECTED] Stressor test passed with no issues — No actionable critique was provided.

Migrate subscription billing from Stripe to the internal ledger

Goal

Move all subscription billing (creation, renewal, cancellation) off Stripe's Subscriptions API onto our internal ledger service, keeping Stripe as the payment processor only. Target: 100% of renewals flowing through the ledger by Oct 31.

Background

The ledger service (services/ledger) already records one-time invoices. It does not yet own subscription lifecycles. The billing team (3 devs) owns both codebases.

Phases

Phase 0 — Staging and safety net before dual-write

• Deploy the same ledger schema, endpoints, scheduler, webhook handler, and ledger_billing feature flag to a staging environment with production-like Stripe test mode, payment method fixtures, and SCA/3DS test cards. Run full cutover drills in staging and document the rollback runbook.
• Add monitoring before cutover: ledger write latency p50/p95 (alert when p95 > 500ms), duplicate event rate, scheduler lag, repeat renewal attempt rate, and a circuit-breaker/fallback status metric for ledger calls.

Phase 1 — Ledger subscription support (3 days)

• Extend the subscriptions table with the original NOT NULL fields (id, customer_id, plan_id, status, current_period_start, current_period_end, cancel_at_period_end) plus stripe_subscription_id, stripe_payment_method_id, version, updated_at, and last_payment_error. Add billing_engine NOT NULL DEFAULT 'stripe' with allowed values 'stripe' and 'ledger'; add trial_end nullable timestamp; add payment_failure_count integer default 0 and next_retry_at nullable timestamp; add stripe_pause_status and stripe_pause_event_id nullable columns. version is used for optimistic concurrency: every update must provide the expected current version and increments it atomically; a version mismatch returns a retryable conflict. stripe_pause_status and stripe_pause_event_id are authoritative for rollback and are verified during each Phase 3 expansion and by the ownership-aware reconciliation. The status field must support at least active, past_due, canceled, paused, trialing, unpaid, and incomplete. [DECISION: add an idempotency-keyed subscription event/outbox table to the ledger schema; it must record subscription_id, event type, idempotency key, payload version, processed_at, and external Stripe event id where available.]
• Implement ledger endpoints: POST /subscriptions, POST /subscriptions/:id/cancel, POST /subscriptions/:id/renew, and POST /subscriptions/:id/change for plan/price/quantity changes, proration, trial conversion, and payment method updates. Every endpoint requires an idempotency key and derives a deterministic event key from it. Duplicate events return the stored result and do not re-apply side effects.
• Enforce state transition validation: cancel and renew are rejected for subscriptions whose status is canceled; renew on past_due is allowed only after payment resolution; cancel sets cancel_at_period_end and does not renew beyond the current period; change is rejected for canceled subscriptions, trial conversion sets trial_end to null and status to active, and payment-method updates are allowed for any non-canceled status.
• Replace the hourly renewal cron with a per-minute scheduler. It selects subscriptions whose current_period_end <= now and status is one of active, trialing, or past_due, and whose next_retry_at is null or already due (next_retry_at <= now), and which have no in-flight renewal attempt and no open requires_action PaymentIntent, and uses transactional row-level leasing (SELECT ... FOR UPDATE SKIP LOCKED or a lease-expiry processing_state/lease column) to prevent missed renewals and duplicate dispatch. It does not use custom sharded queues or bespoke token-bucket infrastructure; it uses standard DB-backed job queues or a standard worker pool. Concurrency is limited by Stripe's documented rate limits (100 requests/sec live mode) and per-worker max in-flight. Charge initiation happens only on due rows; no early billing. The scheduler is disabled during dual-write before cutover; it is enabled only for rows where billing_engine='ledger'. Each renewal uses an idempotency key derived from (subscription_id, current_period_start, current_period_end, attempt_number); the billing period start and end are immutable for a given renewal attempt, and no successful charge may share the same subscription and billing period. Retries transient failures with exponential backoff, records a pending event before charging, and stores invoice/payment IDs after the Stripe PaymentIntent succeeds. Add an integration test proving a failed requires_action payment is not retried before next_retry_at.
• Implement off-session SCA/3DS and dunning handling. When a PaymentIntent returns requires_action, mark the subscription past_due with last_payment_error, increment payment_failure_count, set next_retry_at, trigger the customer action flow (SCA/3DS email or hosted confirmation), and do not retry the charge until the PaymentIntent is confirmed or the dunning schedule advances. Default dunning schedule: first retry 3 days after failure, second retry 7 days after the previous retry, final retry 14 days after that; after the final failed retry mark the subscription canceled and send the cancellation event. Document the policy in the ledger.

Phase 2 — Dual-write and backfill (1 week)

• Billing API writes every subscription creation, renewal, cancellation, status change, plan/price or quantity change, proration, payment method update, and trial conversion event to Stripe and the ledger using a transactional outbox, not fire-and-forget: the event is written in the same database transaction as the local change, and a background worker publishes it to the ledger with retries, dead-letter handling, and alerting. The outbox producer routes by ownership: billing_engine='stripe' rows are written to Stripe and the ledger; billing_engine='ledger' rows are written only to the ledger, and Stripe is touched only for PaymentIntents/payment method identifiers, never for Subscription objects. The ledger rejects duplicate event keys from Phase 1. During the dual-write window billing_engine remains 'stripe' for existing and new subscriptions, and the ledger scheduler is disabled; the outbox worker publishes events to the ledger but does not create PaymentIntents or charge. If the ledger is unavailable, the worker retries with exponential backoff and dead-letters the event; local billing API remains available, but no new ledger-owned renewals are charged until the event is durably recorded. Add an executable test asserting that during dual-write no two successful charges exist for the same subscription_id and billing period across Stripe and ledger.
• Enable the transactional outbox dual-write path in the billing API first. Then run a backfill migration for all non-deleted Stripe subscriptions. Extract every active, trialing, past_due, unpaid, paused, incomplete, and canceled Stripe Subscription object, payment method IDs, current_period_start, current_period_end, cancel_at_period_end, trial end where present, latest invoice/payment IDs, and pending event identifiers from Stripe/webhook event log where available, and upsert them into the ledger subscriptions table preserving stripe_subscription_id. Map Stripe statuses as active->active, trialing->trialing, past_due->past_due, unpaid->unpaid, paused->paused, incomplete->incomplete, canceled->canceled, and incomplete_expired->canceled. Set billing_engine='stripe' for all backfilled rows. Use updated_at/version checks so a newer outbox event is never overwritten by the backfill. Validate row count, plan IDs, period timestamps, status distributions, and missing-row check against Stripe before Phase 3.
• Nightly reconciliation compares subscription state according to ownership. For non-migrated rows (billing_engine='stripe') compare full subscription state between Stripe and the ledger: customer_id, plan_id, status, current_period_end, cancel_at_period_end, latest invoice/payment ID, and pending event IDs. Pending event IDs are the idempotent event-log/outbox keys and Stripe webhook event IDs expected for the row; webhook ingestion must be an idempotent event log with an explicit TTL. During dual-write a transient missing outbox event or in-flight payment ID is classified as expected only if the idempotent event-log entry exists and clears the next reconciliation; any unresolved lifecycle or payment/id mismatch is unresolved. Cutover is blocked until all non-migrated rows have zero unresolved full-state diffs for seven consecutive nights before cutover morning. For migrated rows (billing_engine='ledger') do not compare against Stripe Subscription objects except to assert the corresponding Stripe Subscription is paused or canceled; compare the ledger record against Stripe PaymentIntents, customer records, and payment method identifiers only. An active Stripe Subscription for a ledger-owned row is an unresolved ownership diff, not an expected pause/cancel state. Write the diff to the S3 CSV report and emit near-real-time alerts to a billing drift channel.

Phase 3 — Canary cutover with rollback protocol

• Do not rely on a one-step global flag flip. Move customer cohorts through ledger_billing in stages on cutover day. Cohort assignment is deterministic by customer_id hash, so all subscriptions for a customer move together. Start at 1% of customers, verify for two hours, then expand to 25%, 50%, and 100% with the phase-0 metrics as gates. Before each expansion verify exactly one billing engine owns each subscription: Stripe subscription is paused/canceled when the ledger owns it, billing_engine matches the assigned cohort, and no mixed ownership exists. Alert on any subscription where both Stripe and ledger are active for the same billing period.
• As each cohort moves to ledger-owned renewals, call Stripe pause_collection on the corresponding Stripe Subscription objects wherever the API supports it. Do not cancel Stripe subscriptions during canary cutover unless pause_collection is unavailable for that state and the failure class explicitly requires cancellation. Record the Stripe pause/cancel status and event id on the ledger subscription row (stripe_pause_status, stripe_pause_event_id). Verify the Stripe subscription is paused or canceled before setting billing_engine='ledger'; if the pause/cancel cannot be verified, leave billing_engine='stripe' and alert. After verification set billing_engine='ledger', and ensure exactly one billing engine owns each subscription.
• Stripe webhooks are handled by the new handler in services/billing-webhooks and deduplicated by both Stripe event id and ledger-side subscription event id; state changes already present in the ledger are ignored. Webhook handling includes plan/price/quantity changes, proration, payment method updates, and trial conversion events.
• Rollback protocol: before cutover, snapshot active Stripe subscriptions for the affected cohort. Defined failure classes and triggers:
  - duplicate charge: any subscription with two successful charges for the same subscription and billing period across Stripe and ledger.
  - more than one failed renewal in 15 minutes.
  - p95 renewal latency above 2s for 10 minutes.
  - any unresolved non-migrated full-state diff above zero.
  - ledger outage or outbox publication lag greater than 5 minutes.
  - stuck SCA/3DS PaymentIntent still requires_action after 24 hours.
  On trigger, pause expansion and follow the runbook:
  - check the ledger subscription status before any rollback step; if the ledger subscription is canceled, do not recreate or resume Stripe billing for that subscription, and settle it only via the forward-fix/credit path;
  - stop the ledger scheduler before any rollback;
  - cancel incomplete Stripe PaymentIntent charges and mark ledger pending events as failed;
  - for paused Stripe subscriptions, end pause_collection and reverse-sync only the fields Stripe can safely update (current_period_end, status, cancel_at_period_end, and payment method state) from the ledger back to Stripe before resuming Stripe-owned billing;
  - for canceled Stripe subscriptions, create a new Stripe Subscription with the same customer, plan/quantity, and billing_cycle_anchor aligned to the ledger's current period, proration disabled, map the new stripe_subscription_id in the ledger, and reconcile/credit any overlap before resuming Stripe-owned billing;
  - otherwise use forward-fix-only where ledger remains the source of truth, and the ownership-aware nightly reconciliation resolves every affected subscription before the next cycle.
  The runbook must specify which protocol applies to each failure class.
• Post-cutover monitoring: zero hard payment failures on the cutover dashboard (legitimate pending SCA requires_action states are not counted as hard failures), p95 renewal latency under 2s, zero duplicate charges from idempotency violations, and no dropped renewals for a cohort before expanding. During the 1% canary window run a synthetic SCA/3DS end-to-end verification using a test card and confirm the off-session confirmation and webhook resolution path before expanding.

Phase 4 — Cleanup (later)

• Remove the dual-write path and the Stripe Subscriptions integration only after 30 consecutive days of zero unresolved ownership-aware reconciliation diffs (for non-migrated rows vs Stripe Subscriptions, for migrated rows vs Stripe PaymentIntents/customers and with no active Stripe Subscription for ledger-owned rows) and zero failed renewals; keep Stripe PaymentIntents as the payment processor.

Success criteria

• Zero failed renewals on the cutover dashboard.
• Ownership-aware reconciliation: non-migrated full-state diff is zero for 7 days before cutover and trends to zero during the dual-write week; migrated rows reconcile against Stripe PaymentIntents and customer records, not Stripe Subscription objects, and have no active Stripe Subscription for ledger-owned rows.
• Renewal latency p95 under 2s.
• No duplicate charge events due to missing idempotency/deduplication.

Rollout

Run the Phase 0 staging cutover drill first. Start the transactional outbox dual-write path on Monday of week 1, then run the backfill migration, and run dual-write for at least seven full calendar days with nightly ownership-aware reconciliation. Begin canary cutover on Friday of week 2 only after seven consecutive zero-diff reports for non-migrated subscriptions and all Phase 0 metrics passing. Use the customer-cohort gates from Phase 3: 1% of customers for two hours, then 25%, 50%, 100% after passing the Phase 0 metrics and exactly-one-engine verification. Ping #billing in Slack at each expansion and when done. If any cohort shows a failed renewal or duplicate charge, pause expansion and follow the rollback runbook.

Judge rulings

• Red Team — "Scheduler selects past_due rows without respecting next_retry_at and dunning state": OVERRULE — Step 6 already filters by next_retry_at and excludes open requires_action PaymentIntents, and step 7 explicitly forbids retries before next_retry_at.
• Red Team — "No ownership-based routing stops dual-write to Stripe Subscriptions API after cohort cutover": OVERRULE — Step 8 already routes billing_engine='ledger' rows to ledger only and touches Stripe solely for PaymentIntents/payment method identifiers.
• Red Team — "shadow_ledger state is dead weight and shadow-mode scheduler behavior is contradictory": OVERRULE — The plan defines billing_engine as only 'stripe' and 'ledger' and disables the scheduler during dual-write, so the described shadow_ledger/shadow-mode does not exist.
• Red Team — "Optimistic-locking version field and stripe_pause_* columns lack defined semantics": OVERRULE — Step 3 already defines version compare-and-set semantics and states stripe_pause_status/stripe_pause_event_id are authoritative for rollback and verified by Phase 3/reconciliation.
• Scope Cop — "Scheduler query ignores next_retry_at, causing immediate retry loops on failed renewals": OVERRULE — Step 6 already includes the next_retry_at predicate and excludes open requires_action/in-flight payments.
• Scope Cop — "Backfill migration runs before dual-write starts, creating a blind mutation gap": OVERRULE — Step 9 and the rollout order explicitly enable the transactional outbox dual-write path first, then run the backfill with updated_at/version checks.
• Scope Cop — "Canary cohort 2-hour verification gate is incompatible with asynchronous SCA/3DS flows": OVERRULE — Step 15 already distinguishes hard failures from pending SCA requires_action and requires synthetic SCA E2E verification during the 1% canary before expansion.
• Architect — "Renewal idempotency key is not a stable billing-period identity": OVERRULE — Step 6 derives the key from subscription_id, current_period_start, current_period_end, and attempt_number, and forbids multiple successful charges for the same subscription and billing period.
• Architect — "Scheduler can dispatch past_due renewals before dunning or SCA resolution": OVERRULE — Step 6 already filters on next_retry_at and no open requires_action/in-flight PaymentIntent, and step 7 prohibits retries before next_retry_at.
• Architect — "Backfill and reconciliation baseline do not match the fields being compared": OVERRULE — Step 9 backfills latest invoice/payment IDs and pending event identifiers, and step 10 defines pending event IDs with an idempotent event log and explicit TTL.
• Architect — "Migrated-row reconciliation can miss a still-active Stripe subscription": OVERRULE — Step 10 explicitly treats an active Stripe Subscription for a ledger-owned row as an unresolved ownership diff, not an expected pause/cancel state.
• Architect — "Cutover and rollback ownership changes are not atomic and lack a safe fallback": OVERRULE — Step 12 verifies Stripe pause/cancel before setting billing_engine='ledger', and step 14 branches on ledger canceled status and lists safe reverse-sync fields.
• Tracer — "Critic call failed HTTP 404": OVERRULE — No substantive critique was provided.
• Stress Test — "No issues raised": OVERRULE — No critique to uphold.

Finalized Plan

Migrate subscription billing from Stripe to the internal ledger

Goal

Move all subscription billing (creation, renewal, cancellation) off Stripe's Subscriptions API onto our internal ledger service, keeping Stripe as the payment processor only. Target: 100% of renewals flowing through the ledger by Oct 31.

Background

The ledger service (services/ledger) already records one-time invoices. It does not yet own subscription lifecycles. The billing team (3 devs) owns both codebases.

Phases

Phase 0 — Staging and safety net before dual-write

• Deploy the same ledger schema, endpoints, scheduler, webhook handler, and ledger_billing feature flag to a staging environment with production-like Stripe test mode, payment method fixtures, and SCA/3DS test cards. Run full cutover drills in staging and document the rollback runbook.
• Add monitoring before cutover: ledger write latency p50/p95 (alert when p95 > 500ms), duplicate event rate, scheduler lag, repeat renewal attempt rate, and a circuit-breaker/fallback status metric for ledger calls.

Phase 1 — Ledger subscription support (3 days)

• Extend the subscriptions table with the original NOT NULL fields (id, customer_id, plan_id, status, current_period_start, current_period_end, cancel_at_period_end) plus stripe_subscription_id, stripe_payment_method_id, version, updated_at, and last_payment_error. Add billing_engine NOT NULL DEFAULT 'stripe' with allowed values 'stripe' and 'ledger'; add trial_end nullable timestamp; add payment_failure_count integer default 0 and next_retry_at nullable timestamp; add stripe_pause_status and stripe_pause_event_id nullable columns. version is used for optimistic concurrency: every update must provide the expected current version and increments it atomically; a version mismatch returns a retryable conflict. stripe_pause_status and stripe_pause_event_id are authoritative for rollback and are verified during each Phase 3 expansion and by the ownership-aware reconciliation. The status field must support at least active, past_due, canceled, paused, trialing, unpaid, and incomplete. [DECISION: add an idempotency-keyed subscription event/outbox table to the ledger schema; it must record subscription_id, event type, idempotency key, payload version, processed_at, and external Stripe event id where available.]
• Implement ledger endpoints: POST /subscriptions, POST /subscriptions/:id/cancel, POST /subscriptions/:id/renew, and POST /subscriptions/:id/change for plan/price/quantity changes, proration, trial conversion, and payment method updates. Every endpoint requires an idempotency key and derives a deterministic event key from it. Duplicate events return the stored result and do not re-apply side effects.
• Enforce state transition validation: cancel and renew are rejected for subscriptions whose status is canceled; renew on past_due is allowed only after payment resolution; cancel sets cancel_at_period_end and does not renew beyond the current period; change is rejected for canceled subscriptions, trial conversion sets trial_end to null and status to active, and payment-method updates are allowed for any non-canceled status.
• Replace the hourly renewal cron with a per-minute scheduler. It selects subscriptions whose current_period_end <= now and status is one of active, trialing, or past_due, and whose next_retry_at is null or already due (next_retry_at <= now), and which have no in-flight renewal attempt and no open requires_action PaymentIntent, and uses transactional row-level leasing (SELECT ... FOR UPDATE SKIP LOCKED or a lease-expiry processing_state/lease column) to prevent missed renewals and duplicate dispatch. It does not use custom sharded queues or bespoke token-bucket infrastructure; it uses standard DB-backed job queues or a standard worker pool. Concurrency is limited by Stripe's documented rate limits (100 requests/sec live mode) and per-worker max in-flight. Charge initiation happens only on due rows; no early billing. The scheduler is disabled during dual-write before cutover; it is enabled only for rows where billing_engine='ledger'. Each renewal uses an idempotency key derived from (subscription_id, current_period_start, current_period_end, attempt_number); the billing period start and end are immutable for a given renewal attempt, and no successful charge may share the same subscription and billing period. Retries transient failures with exponential backoff, records a pending event before charging, and stores invoice/payment IDs after the Stripe PaymentIntent succeeds. Add an integration test proving a failed requires_action payment is not retried before next_retry_at.
• Implement off-session SCA/3DS and dunning handling. When a PaymentIntent returns requires_action, mark the subscription past_due with last_payment_error, increment payment_failure_count, set next_retry_at, trigger the customer action flow (SCA/3DS email or hosted confirmation), and do not retry the charge until the PaymentIntent is confirmed or the dunning schedule advances. Default dunning schedule: first retry 3 days after failure, second retry 7 days after the previous retry, final retry 14 days after that; after the final failed retry mark the subscription canceled and send the cancellation event. Document the policy in the ledger.

Phase 2 — Dual-write and backfill (1 week)

• Billing API writes every subscription creation, renewal, cancellation, status change, plan/price or quantity change, proration, payment method update, and trial conversion event to Stripe and the ledger using a transactional outbox, not fire-and-forget: the event is written in the same database transaction as the local change, and a background worker publishes it to the ledger with retries, dead-letter handling, and alerting. The outbox producer routes by ownership: billing_engine='stripe' rows are written to Stripe and the ledger; billing_engine='ledger' rows are written only to the ledger, and Stripe is touched only for PaymentIntents/payment method identifiers, never for Subscription objects. The ledger rejects duplicate event keys from Phase 1. During the dual-write window billing_engine remains 'stripe' for existing and new subscriptions, and the ledger scheduler is disabled; the outbox worker publishes events to the ledger but does not create PaymentIntents or charge. If the ledger is unavailable, the worker retries with exponential backoff and dead-letters the event; local billing API remains available, but no new ledger-owned renewals are charged until the event is durably recorded. Add an executable test asserting that during dual-write no two successful charges exist for the same subscription_id and billing period across Stripe and ledger.
• Enable the transactional outbox dual-write path in the billing API first. Then run a backfill migration for all non-deleted Stripe subscriptions. Extract every active, trialing, past_due, unpaid, paused, incomplete, and canceled Stripe Subscription object, payment method IDs, current_period_start, current_period_end, cancel_at_period_end, trial end where present, latest invoice/payment IDs, and pending event identifiers from Stripe/webhook event log where available, and upsert them into the ledger subscriptions table preserving stripe_subscription_id. Map Stripe statuses as active->active, trialing->trialing, past_due->past_due, unpaid->unpaid, paused->paused, incomplete->incomplete, canceled->canceled, and incomplete_expired->canceled. Set billing_engine='stripe' for all backfilled rows. Use updated_at/version checks so a newer outbox event is never overwritten by the backfill. Validate row count, plan IDs, period timestamps, status distributions, and missing-row check against Stripe before Phase 3.
• Nightly reconciliation compares subscription state according to ownership. For non-migrated rows (billing_engine='stripe') compare full subscription state between Stripe and the ledger: customer_id, plan_id, status, current_period_end, cancel_at_period_end, latest invoice/payment ID, and pending event IDs. Pending event IDs are the idempotent event-log/outbox keys and Stripe webhook event IDs expected for the row; webhook ingestion must be an idempotent event log with an explicit TTL. During dual-write a transient missing outbox event or in-flight payment ID is classified as expected only if the idempotent event-log entry exists and clears the next reconciliation; any unresolved lifecycle or payment/id mismatch is unresolved. Cutover is blocked until all non-migrated rows have zero unresolved full-state diffs for seven consecutive nights before cutover morning. For migrated rows (billing_engine='ledger') do not compare against Stripe Subscription objects except to assert the corresponding Stripe Subscription is paused or canceled; compare the ledger record against Stripe PaymentIntents, customer records, and payment method identifiers only. An active Stripe Subscription for a ledger-owned row is an unresolved ownership diff, not an expected pause/cancel state. Write the diff to the S3 CSV report and emit near-real-time alerts to a billing drift channel.

Phase 3 — Canary cutover with rollback protocol

• Do not rely on a one-step global flag flip. Move customer cohorts through ledger_billing in stages on cutover day. Cohort assignment is deterministic by customer_id hash, so all subscriptions for a customer move together. Start at 1% of customers, verify for two hours, then expand to 25%, 50%, and 100% with the phase-0 metrics as gates. Before each expansion verify exactly one billing engine owns each subscription: Stripe subscription is paused/canceled when the ledger owns it, billing_engine matches the assigned cohort, and no mixed ownership exists. Alert on any subscription where both Stripe and ledger are active for the same billing period.
• As each cohort moves to ledger-owned renewals, call Stripe pause_collection on the corresponding Stripe Subscription objects wherever the API supports it. Do not cancel Stripe subscriptions during canary cutover unless pause_collection is unavailable for that state and the failure class explicitly requires cancellation. Record the Stripe pause/cancel status and event id on the ledger subscription row (stripe_pause_status, stripe_pause_event_id). Verify the Stripe subscription is paused or canceled before setting billing_engine='ledger'; if the pause/cancel cannot be verified, leave billing_engine='stripe' and alert. After verification set billing_engine='ledger', and ensure exactly one billing engine owns each subscription.
• Stripe webhooks are handled by the new handler in services/billing-webhooks and deduplicated by both Stripe event id and ledger-side subscription event id; state changes already present in the ledger are ignored. Webhook handling includes plan/price/quantity changes, proration, payment method updates, and trial conversion events.
• Rollback protocol: before cutover, snapshot active Stripe subscriptions for the affected cohort. Defined failure classes and triggers:
  - duplicate charge: any subscription with two successful charges for the same subscription and billing period across Stripe and ledger.
  - more than one failed renewal in 15 minutes.
  - p95 renewal latency above 2s for 10 minutes.
  - any unresolved non-migrated full-state diff above zero.
  - ledger outage or outbox publication lag greater than 5 minutes.
  - stuck SCA/3DS PaymentIntent still requires_action after 24 hours.
  On trigger, pause expansion and follow the runbook:
  - check the ledger subscription status before any rollback step; if the ledger subscription is canceled, do not recreate or resume Stripe billing for that subscription, and settle it only via the forward-fix/credit path;
  - stop the ledger scheduler before any rollback;
  - cancel incomplete Stripe PaymentIntent charges and mark ledger pending events as failed;
  - for paused Stripe subscriptions, end pause_collection and reverse-sync only the fields Stripe can safely update (current_period_end, status, cancel_at_period_end, and payment method state) from the ledger back to Stripe before resuming Stripe-owned billing;
  - for canceled Stripe subscriptions, create a new Stripe Subscription with the same customer, plan/quantity, and billing_cycle_anchor aligned to the ledger's current period, proration disabled, map the new stripe_subscription_id in the ledger, and reconcile/credit any overlap before resuming Stripe-owned billing;
  - otherwise use forward-fix-only where ledger remains the source of truth, and the ownership-aware nightly reconciliation resolves every affected subscription before the next cycle.
  The runbook must specify which protocol applies to each failure class.
• Post-cutover monitoring: zero hard payment failures on the cutover dashboard (legitimate pending SCA requires_action states are not counted as hard failures), p95 renewal latency under 2s, zero duplicate charges from idempotency violations, and no dropped renewals for a cohort before expanding. During the 1% canary window run a synthetic SCA/3DS end-to-end verification using a test card and confirm the off-session confirmation and webhook resolution path before expanding.

Phase 4 — Cleanup (later)

• Remove the dual-write path and the Stripe Subscriptions integration only after 30 consecutive days of zero unresolved ownership-aware reconciliation diffs (for non-migrated rows vs Stripe Subscriptions, for migrated rows vs Stripe PaymentIntents/customers and with no active Stripe Subscription for ledger-owned rows) and zero failed renewals; keep Stripe PaymentIntents as the payment processor.

Success criteria

• Zero failed renewals on the cutover dashboard.
• Ownership-aware reconciliation: non-migrated full-state diff is zero for 7 days before cutover and trends to zero during the dual-write week; migrated rows reconcile against Stripe PaymentIntents and customer records, not Stripe Subscription objects, and have no active Stripe Subscription for ledger-owned rows.
• Renewal latency p95 under 2s.
• No duplicate charge events due to missing idempotency/deduplication.

Rollout

Run the Phase 0 staging cutover drill first. Start the transactional outbox dual-write path on Monday of week 1, then run the backfill migration, and run dual-write for at least seven full calendar days with nightly ownership-aware reconciliation. Begin canary cutover on Friday of week 2 only after seven consecutive zero-diff reports for non-migrated subscriptions and all Phase 0 metrics passing. Use the customer-cohort gates from Phase 3: 1% of customers for two hours, then 25%, 50%, 100% after passing the Phase 0 metrics and exactly-one-engine verification. Ping #billing in Slack at each expansion and when done. If any cohort shows a failed renewal or duplicate charge, pause expansion and follow the rollback runbook.

Generated by Plan Court