Breaking suggestion: add support for transactions

[alecgibson]

MongoDB 3.x was end-of-lifed in April 2021, which means that all currently-supported versions of MongoDB now support transactions.

sharedb-mongo currently uses an optimistic write mechanism to deal with the fact that it has to write both an op and a snapshot as part of a single "commit". This results in some non-canonical ops being committed to the database, and means we need to do some painful work to get a range of canonical ops.

If we moved to using MongoDB transactions, we would know that all committed ops are canonical (since a conflicting transaction can be made to roll back), which means we could vastly simplify op range fetches.

This would be a breaking change, since it would require MongoDB >= 4.X.

We'd also want to do some thinking about how to enable this feature on already-established databases. For example, if you turn on transactions in sharedb-mongo, any documents that were created after the change can be considered to only have canonical ops, but any "legacy" documents created before that time would not. We could potentially set a flag on the snapshots?

[alecgibson]

According to the MongoDB docs:

In most cases, multi-document transaction incurs a greater performance cost over single document writes, and the availability of multi-document transactions should not be a replacement for effective schema design.

We probably don't want to incur a performance penalty in the mainline case (committing to the database) in order to get a performance gain for a lesser used case (fetching ops).

It's not all necessarily doom and gloom. There are in theory some benefits to latency if committing several ops in the same transaction (although I'm not sure we'd be able to leverage this?).

[ericyhwang]

Other official resources to look at, from doing a bit of searching:

https://www.mongodb.com/docs/manual/core/transactions/
https://www.mongodb.com/docs/manual/core/transactions-production-consideration/
https://www.mongodb.com/docs/manual/core/write-operations-atomicity/

I haven't personally worked with Mongo transactions before, so I don't know what the performance implications of them would be, especially for high-volume writes.

[alecgibson]

One thing that would be nice if we add this would perhaps be to add some Transaction API to ShareDB core, which would let us use this at the op level, for example if we wanted to write to two separate collections as a single atomic transaction. Not entirely sure what that API would look like. Maybe we'd do it at the Connection level?

// Perhaps `Session` extends `Connection` and just sends a unique session id with every message
// so that the server knows to create/continue a transaction (if supported), and when the session
// sends a special `commit` message, the transaction is committed to DB and propagated to other
// clients over pub/sub, etc.
const session = connection.createSession();
await session.get('docs', '1').submitOp(...);
await session.get('docs', '2').submitOp(...);
await session.commit();
await session.close();