Orchestrator (Dynamo store): step record lost under concurrent duplicate execution — whole-doc read-modify-write has no write protection

[bjagg]

Found during the 2026-07-31 backend redeploy verification (images @ c18a2f0).

Symptom

Execution exec_d0b51eb6979f (live, skill_mastered → wallet, completed 11-step plan) is missing step 7 (execute_issuer_payload_translation) from its persisted read model — steps holds keys 1–6, 8–11. The step demonstrably ran: the orchestrator log has step 7 ran degraded: execution_id=exec_d0b51eb6979f … at 15:59:40, and per execute_plan()'s ordering, save_step is called before that log line. A sibling execution one minute earlier (exec_1d3c3c8f10c3, same images, same plan shape) persisted all 11 steps — so it's a race, not deterministic.

Likely mechanism

DynamoExecutionStore persists the whole execution as ONE document with unguarded read-modify-write (_load → mutate → put_item, store_dynamo.py). That design explicitly assumes a single sequential writer ("safe: one run writes sequentially, admin only reads"). But the known event-consumer rough edge (its internal httpx read-timeout fires while the orchestrator keeps working — an EC timeout traceback at 15:59:40 coincides exactly with this execution's step 7) can produce duplicate/concurrent processing of the same execution, at which point last-writer-wins on the whole document silently drops the other writer's step. The Lambda /tmp per-instance dedup gap (known, post-freeze) feeds the same failure mode.

Impact

Cosmetic-to-moderate for the POC: the execution completed and delivered correctly; the audit trail just lost one step record (ironically the degraded one). But it undermines the audit-trail completeness guarantee (FR-OR-21) precisely when things get interesting (concurrent duplicates + degraded steps).

Fix directions (pick at implementation time)

1. Idempotent start guard — run_workflow refuses (or short-circuits) when the execution_id already exists in a non-terminal state; cheapest, addresses the duplicate at the source.
2. Optimistic locking — conditional put_item on updated_at (retry-on-conflict); keeps the single-doc shape.
3. Per-step items — store steps as separate Dynamo items under the execution pk; removes the shared-document contention entirely (matches FR-OR-31's out-of-line direction).

Related: the DynamoDB-backed event dedup follow-up (post-freeze note from the first AWS bring-up), #139 (result summary), ADR-0014/0015.