Confidence-carrying replay. One Harper component. Run it in 60 seconds.
Ishan Shah's article Building Event-Driven Systems That Can Recover With Confidence (HackerNoon, June 27, 2026) articulates something most event-driven system designs quietly get wrong: reliability is not replayability. You can have a Kafka topic you can replay and still have no architectural confidence that your projections reflect your history. Shah's answer is a Recovery Contract — seven fields (H, O, I, F, S, Q, E) that together make replay confidence-carrying, producing verifiable evidence that your projections match your authoritative history. It is one of the clearest frameworks for this problem we have seen written down.
This repo is a Harper implementation of that contract idea — not a claim that Harper replaces his stack, but a demonstration of how naturally the contract maps onto a single component.
Reading Shah's article, one thing kept jumping out: every field of his Recovery Contract maps almost one-to-one onto a primitive Harper already has. His seven-field contract — H, O, I, F, S, Q, E — is not a set of aspirational properties that require six independent systems to satisfy. It is a set of specific design decisions, and those decisions have natural homes in a unified runtime.
This repo makes that mapping concrete. Each contract field is wired to a Harper primitive, and the chaos/ scripts exercise each one — most by inducing the failure mode the field absorbs (duplicate delivery, out-of-order arrival, silent projection drift, connector backlog), and one (crash) as an architectural contrast rather than a recovery test. Each scenario runs a recovery, asserts its invariant, and prints a RecoveryEvidence record with confidence_status: "trusted".
Harper is a unified Node.js runtime: audit-logged ACID storage (RocksDB), application layer, pub/sub (MQTT), and REST in a single process. There is no WAL exporter, no replication slot, no separate stream-processing cluster. The contract fields are cheap to wire because the infrastructure surface is small. The thinking the contract requires is still yours.
This runs on the open-source Harper core (harper, Apache-2.0) — Harper Pro is not required. The audit log, Resource API, subscribe(), MQTT, and REST are all core features; Pro only adds the multi-node replication referenced under Honest Boundaries.
| Field | Contract Field | Kafka/Debezium assembly | Harper primitive | In this repo |
|---|---|---|---|---|
| H | Authoritative history | Postgres WAL → Debezium → Kafka topic | @table(audit:"30d") — every commit appended to monotonic audit log |
InventoryTransaction in schema.graphql |
| O | Ordering boundary | Custom SkuPartitioner; per-partition order only |
Per-entity audit log; fold sorted by event_time then event_id |
byEventTimeThenId comparator in resources.js |
| I | Idempotency key | Consumer-side dedup logic or Kafka transactions | event_id: ID @primaryKey; append-only log — identical re-delivery is a no-op, a conflicting one is rejected (409) |
duplicate-delivery.mjs |
| F | Deterministic projection function | Kafka Streams topology | computeSellableAvailability() in resources.js, same process as data |
out-of-order.mjs, projection-drift.mjs |
| S | Replay scope | Re-set consumer group offset; replay whole topic or partition | POST /Recover {sku, from_event_time, to_event_time} — bounded, rebuilds from full history |
connector-lag.mjs |
| Q | Reconciliation invariant | Cross-system query (Kafka + DB + downstream) | Local query in same store: stock_on_hand == Σ delta_quantity; checks read from recovery-contract.yaml |
RecoveryEvidence.reconciliation field |
| E | Recovery evidence | Ad hoc scripts or manual audit | RecoveryEvidence table; POST /Recover returns structured JSON with confidence_status |
All chaos scripts |
Requires Node ^22.18.0 || >=24 (matching Harper 5.1).
npm install # installs harper (open-source core, Apache-2.0) + yaml
npm start # boots an ISOLATED local instance on :9956 (data in ./harper-home)
# ...then, in a second terminal:
npm run demonpm start runs run-local.sh, which pins its own ports (HTTP 9956) and data root (./harper-home) so the demo never collides with another Harper instance on the machine — the most common "works on my laptop, 404s on yours" trap. It also enables AUTHENTICATION_AUTHORIZELOCAL, so localhost needs no credentials.
npm run demo runs all seven scenarios, asserts each one, and prints the final RecoveryEvidence. To reset to a clean run, stop the server and rm -rf harper-home. Against a secured (non-local) instance, set HARPER_URL, HARPER_USER, and HARPER_PASS. See DEMO-OUTPUT.md for an annotated run transcript.
| Script | Scenario | Recovery Contract field |
|---|---|---|
npm run seed |
Establishes 5-event baseline history for sku 1231241 (96 units on a fresh instance) | H — Authoritative History |
npm run chaos:duplicate |
Identical re-delivery is a no-op; a conflicting duplicate (same id, different payload) is rejected 409 — history can't be rewritten | I — Idempotency Key |
npm run chaos:reorder |
Posts events with inverted arrival order; fold corrects to logical time | O — Ordering Boundary |
npm run chaos:drift |
Corrupts StockOnHand projection; recovery detects, heals, and emits evidence | Q — Reconciliation / E — Evidence |
npm run chaos:lag |
Posts a 10-event backlog; bounded replay drains it (rebuilding from full history) | S — Replay Scope |
npm run chaos:crash |
Architectural contrast (not a recovery test): why "crash before offset commit" is structurally absent in Harper | — |
npm run chaos:concurrent |
50 concurrent same-SKU writes; the live cache may under-count (non-atomic RMW), then /Recover reconciles to the true total and proves it |
Honest boundary: cache drift → recovery |
-
The polyglot bus — Harper does this too. Harper speaks MQTT, WebSocket, SSE, and REST natively, so heterogeneous consumers (any language, any team) subscribe over open protocols with no Harper-specific client. Consolidating the bus is often faster, not just simpler: Harper's benchmark of Kafka-centered stacks vs. a single Harper cluster measured ~13× lower median latency on filtered fan-out and ~56× on durable write-then-notify versus Kafka + Redis + routing. Kafka still leads at the extremes — very high-throughput log ingestion, large fleets of independent consumer groups, sub-millisecond point reads, and the Connect/ksqlDB/Flink ecosystem — and the study is explicit that it is an architecture-fit comparison run on a laptop VM. But the bus role alone is a weaker reason than it used to be to reach for Kafka by default.
-
"History is forever" is a retention decision.
audit: "30d"inschema.graphqlkeeps the audit log for 30 days. For infinite history, size the window explicitly or archive to cold storage. The contract field H requires you to define what "authoritative" means and for how long. -
Idempotency is structural here, not magical exactly-once. The
event_idprimary key ensures re-delivery is a no-op at the database layer. This aligns semantically with the Recovery Contract's I field. It is not Kafka's transactional exactly-once across producers and consumers — it is a simpler, more local guarantee that happens to be sufficient for this pattern. -
The Recovery Contract is still a design discipline you author. Harper makes each field cheap to implement; it does not make the contract unnecessary. You still define the idempotency key, the ordering boundary, the projection function, the reconciliation invariant, and the recovery scope. The infrastructure surface shrinks; the thinking does not.
-
The live projection is a cache; the log is the source of truth.
StockOnHand/Availabilityare maintained by an incremental read-modify-write on ingest. It is not atomic — Harper commits the request's writes at transaction end — so under concurrent writes to the same SKU the live view can momentarily under-count.npm run chaos:concurrentdemonstrates it: 50 concurrent same-SKU writes may leave the cache below the true total, then/Recoverreconciles to it and proves it (confidence_status: "trusted"). For a strictly-correct live view, give each SKU a single writer (partition by SKU — Shah's ordering boundary) or use a database-native atomic increment; the audit log stays authoritative and/Recoveris the backstop. Derived state is temporary; history is authoritative.
Recovery Contract concept and the H/O/I/F/S/Q/E vocabulary: Ishan Shah, Building Event-Driven Systems That Can Recover With Confidence, HackerNoon, June 27, 2026. This repository is one implementation of the idea.
Apache-2.0 — matching the open-source Harper core this runs on.