fix: restore 7.4.1 behaviour for existing features (8.0.1)

CHANGELOG.md

@@ -1,5 +1,50 @@
 # Changelog
 
+## 8.0.1
+
+8.0.1 restores the 7.4.1 behaviour that 8.0.0 changed for callers of features 7.x already had. New 8.0 features are
+unchanged.
+
+### Fixed
+
+- Query validation reports what 7.x reported: the children of `and()`, `or()` and `elemMatch()` are validated before
+  the query itself, the first invalid query in reading order sets the message even when a later one does not parse,
+  a method 7.x could not parse that no validator takes reads `Invalid query: Invalid query method: <method>`, and a
+  raw query from a string reads `Invalid query method: raw`.
+- `select('$tenant')` passes query validation again without shared tables, and the read refuses it with `Cannot
+  select attributes: $tenant`, as in 7.x.
+- The `UID` validator description is the 7.x text again, also inside `Invalid cursor: …`.
+- On PostgreSQL an exact search (`"foo bar"`) matches every word in any order again, not the adjacent phrase.
+- `containsAll()` on an attribute that is not an array matches each value as a whole-value `LIKE` (`ILIKE` on
+  PostgreSQL) pattern, any of them, on MariaDB, MySQL and PostgreSQL, as in 7.x.
+- A read whose orders include `orderRandom()` ignores its cursor instead of failing with `Order attribute '' is
+  empty`.
+- `document_purge` fires inside the write's outermost transaction, just before it commits, so a listener that throws
+  rolls the write back and its failure reaches the caller, as in 7.x.
+- `increaseDocumentAttribute()` and `decreaseDocumentAttribute()` pass a fractional change, `max` or `min` on an
+  integer attribute to the engine again and return the exact sum, and refuse a change that is not above zero with
+  `\InvalidArgumentException`, as in 7.x.
+- Numeric operators on integer and double attributes are checked as in 7.x: a fractional change or limit, or a
+  fractional item appended to an integer array, is no longer refused.
+- A mismatched attribute default reads `Default value x does not match given type <type>` (and `is not a valid
+  integer string for type bigint`) without JSON quoting, as in 7.x.
+- `Index::fromArray()` refuses an unknown index type with `Exception\Index` (`Unknown index type: …`) instead of
+  creating a key index.
+- `Exception\Unique::MESSAGE` is `Unique index violation` again.
+- A many-to-many relationship created by 7.x can be renamed again: its junction index, which 7.x stored with `key`
+  `index_<key>`, is read under its `$id` `_index_<key>`.
+- The SQL adapters bind floats as 7.x did: fixed-point only in `find()`, as PHP writes them elsewhere, so a write
+  no longer stores `1e-300` or `-2.5e-20` as `0`.
+- Two-level relationship selects (`author.profile.*`, `tags.books.name`) return the related documents' attributes
+  as in 7.x, and `select(['*', '*.*'])` reads every column instead of failing on MariaDB, MySQL and PostgreSQL.
+- A single SQL read returns `$sequence` ahead of `$id`, in 7.x's key order.
+- An update validates the stored values it leaves unchanged again, as 7.x did; a stored object value is held to
+  7.x's object rules.
+- On MongoDB a transaction that lost a write conflict runs again up to 20 times, so concurrent increments of one
+  document all land.
+- With dates preserved, linking existing children through their one-to-many parent keeps each child's stored
+  `$updatedAt`, as 7.x did; unlinking still stamps the child.
+
 ## 8.0.0 (unreleased)
 
 8.0 is a major release. Read [UPGRADE.md](UPGRADE.md) before you upgrade from 7.x: it lists every change you may

UPGRADE.md

@@ -223,9 +223,12 @@ $permissions = [Permission::read(Role::user(Id::unique()))];
 - A filter on a path into an object attribute (`meta.address.city`) takes keys of `a-z`, `A-Z`, `0-9`, `_` and `-`
   only. The query validator refuses any other key with `Utopia\Database\Exception\Query` on every adapter (7.x
   refused such keys on PostgreSQL only), and on PostgreSQL the query is refused also when validation is skipped.
-- On PostgreSQL, an exact search (`search('title', '"foo bar"')`) matches the words as an adjacent phrase, as on
-  MariaDB, MySQL and SQLite, and `notSearch()` with an exact term excludes only that phrase. In 7.4.0 PostgreSQL
-  matched both words in any order. To match both words in any order, pass a `search()` for each word.
+- On PostgreSQL, an exact search (`search('title', '"foo bar"')`) matches documents holding every word in any order,
+  as in 7.x, where MariaDB, MySQL and SQLite match the adjacent phrase.
+- `containsAll()` on an attribute that is not an array matches as in 7.x: on MariaDB, MySQL and PostgreSQL each value
+  is a `LIKE` (`ILIKE`) pattern on the whole value and a document matching any of them is returned; on MongoDB it is
+  `$all`. Use `containsString()` for substrings.
+- A read whose orders include `orderRandom()` ignores its cursor, as in 7.x.
 
 ## Schema: value objects
 
@@ -696,13 +699,14 @@ It fires once per written document from `updateDocument()` (for both the old and
 `Event\Document\Purged` with the document's `collection` and `id`. As in 7.x, `createDocument()` and `createDocuments()` do
 not fire it. Attribute schema changes fire it for the collection's metadata document (`$collection` = `_metadata`).
 
-A write fires it after the outermost transaction commits: inside `withTransaction()` the events of every write wait
-for the outer commit, a rollback drops them, and a retried attempt announces once. Each event runs under the tenant
-and the `silent()` scope in force when its document was written. If the cache invalidation after the commit fails,
-the write throws with its data committed, after `document_purge` has fired. `purgeCachedDocument()` fires it at once.
-On an adapter without savepoints (MongoDB), a nested `withTransaction()` that fails is not rolled back on its own:
-when the caller catches the failure, the nested writes commit with the caller and their events fire after that
-commit.
+As in 7.x, a write fires it inside its transaction, so a listener that throws rolls the write back and its failure
+reaches the caller. The events wait for the end of the outermost transaction and fire just before it commits: inside
+`withTransaction()` the events of every write fire together before the outer commit, a rollback before that point
+drops them, and every attempt of a retried transaction that reaches its commit announces them. Each event runs under
+the tenant and the `silent()` scope in force when its document was written. If the cache invalidation after the
+commit fails, the write throws with its data committed. `purgeCachedDocument()` fires it at once. On an adapter
+without savepoints (MongoDB), a nested `withTransaction()` that fails is not rolled back on its own: when the caller
+catches the failure, the nested writes commit with the caller and their events fire before that commit.
 
 ### `document_update` for related documents a delete changed
 
@@ -865,27 +869,20 @@ method as a no-op, so a hook overrides only what it needs:
   counted and emitted it). Of a batch that repeats an id, only the first copy is written. Permissions are written
   only for inserted documents: in 7.x the skipped copy's permissions were added to the stored document on MariaDB,
   MySQL and SQLite, so `find()`, `count()` and `sum()` could return it to roles its own permissions do not grant.
-- **Fractional numbers on integer attributes.** `increaseDocumentAttribute()` and `decreaseDocumentAttribute()`
-  throw `Utopia\Database\Exception\Type` before anything is written, on every adapter and also on schemaless
-  collections that declare the attribute as an integer:
-  - for a fractional change value (`Change value must be an integer.`). 7.x passed it to the engine, which rounded
-    it on MariaDB and MySQL, failed on PostgreSQL, and stored a float in the integer attribute on SQLite, MongoDB,
-    Memory and Redis. Pass an integer change value, or use a float attribute for fractional counters.
-  - for a fractional `max` or `min` (`Max must be an integer.`, `Min must be an integer.`). Integer bounds are
-    compared with exact integer arithmetic, so 64-bit and unsigned values never pass through a float. The bound
-    accepts the same whole numbers as an operator limit: an integer, an integer string, a string with only zero
-    decimals such as `'102.0'`, or a float without a fractional part such as `102.0`, each converted exactly. Pass
-    a whole bound, for example `floor($max)` or `ceil($min)`, which admits the same integer values.
-
-  Change values and bounds on float and double attributes may still be fractional.
-- **Operator limits on integer attributes.** The `max` or `min` limit of `Operator::increment()`, `decrement()`,
-  `multiply()`, `divide()` and `power()` on an integer or big integer attribute has to be a whole number: an
-  integer, an integer string, or a float without a fractional part such as `9.0e18`. A fractional limit such as
-  `102.4` is refused before anything is written with `Utopia\Database\Exception\Structure`
-  (`Cannot apply <operator> operator: max/min limit must be a whole number for integer attribute '<key>', got
-  <limit>`). With validation skipped that check does not run, and Memory and Redis refuse such a limit with
-  `Utopia\Database\Exception\Operator` when the result leaves PHP's integer range. Limits on float and double
-  attributes are unchanged.
+- **Fractional numbers on integer attributes.** As in 7.x, `increaseDocumentAttribute()` and
+  `decreaseDocumentAttribute()` pass a fractional change value, `max` or `min` on an integer attribute to the
+  engine and return the document with the exact sum: MariaDB and MySQL store it rounded, MongoDB stores it and reads
+  it back truncated, and PostgreSQL fails. A whole float such as `2.0` is bound as the integer. A whole change value
+  and whole bounds are compared with exact integer arithmetic, so 64-bit and unsigned values never pass through a
+  float. A change value that is not a number greater than 0 throws `\InvalidArgumentException`
+  (`Value must be numeric and greater than 0`), as in 7.x.
+- **Operator limits on integer attributes.** As in 7.x, the change and the `max` or `min` limit of
+  `Operator::increment()`, `decrement()`, `multiply()`, `divide()` and `power()` on an integer or double attribute
+  only have to be numeric and finite: a fractional change or limit on an integer reaches the engine, which stores
+  the result as for a fractional counter. Without a limit, a result predicted outside the attribute's range is
+  refused with `Utopia\Database\Exception\Structure` (`Cannot apply <operator> operator: would overflow maximum value
+  of <max>`). On a big integer attribute the limit has to be a whole number in the attribute's range (`Cannot apply
+  <operator> operator: max/min limit must be a whole number for integer attribute '<key>', got <limit>`).
 - **Operators on upserts that create a document.** An upsert that creates a document applies every operator to the
   attribute's default, as it does for an existing document: `dateAddDays()` and `dateSubDays()` shift the date,
   `arrayFilter()` filters the array, and the maximum or minimum of increment, decrement, multiply, divide and power
@@ -1091,14 +1088,13 @@ coroutine that opened it and the coroutines it starts; see [Pools and profiling]
   `public readonly ?string $state`; a numeric string is also the integer code. `Exception\Order::__construct(string
   $message, ?string $attribute = null, int|string $code = 0, ?\Throwable $previous = null)` takes the attribute
   second.
-- **Unique index violations.** Every adapter now reports a unique index violation as
-  `Utopia\Database\Exception\Unique` with the message `Document with the requested unique attributes already exists`
-  (7.x: `Unique index violation`). The class and its hierarchy are unchanged: `Unique` extends `Duplicate`, and a
-  conflicting document `$id` still throws a plain `Duplicate` with `Document already exists`. Match on the class,
-  not the message: catch `Unique` before `Duplicate` to tell the two apart. `Exception\Unique` has no constructor of
-  its own and never rewrites the message it is given. The message is `Exception\Unique::MESSAGE`, also when
-  `createIndex()` refuses a unique index over documents that already share a value (7.4.1: `Unique index violation`,
-  and `Cannot create unique index: existing rows already contain duplicate values` on Redis).
+- **Unique index violations.** Every adapter reports a unique index violation as
+  `Utopia\Database\Exception\Unique` with the 7.x message `Unique index violation`. The class and its hierarchy are
+  unchanged: `Unique` extends `Duplicate`, and a conflicting document `$id` still throws a plain `Duplicate` with
+  `Document already exists`. Match on the class, not the message: catch `Unique` before `Duplicate` to tell the two
+  apart. `Exception\Unique` has no constructor of its own and never rewrites the message it is given. The message is
+  `Exception\Unique::MESSAGE`, also when `createIndex()` refuses a unique index over documents that already share a
+  value (7.4.1 Redis: `Cannot create unique index: existing rows already contain duplicate values`).
 - **`ignoreDuplicates()` on PostgreSQL** skips only a document whose id is stored, as in 7.x: a new id that collides
   on another unique index throws `Utopia\Database\Exception\Unique`. MariaDB, MySQL and SQLite cannot name the
   index to ignore and, as in 7.x, skip such a row without error.
@@ -1141,7 +1137,8 @@ coroutine that opened it and the coroutines it starts; see [Pools and profiling]
   below). On a sharded cluster (`mongos`) the adapter runs without transactions, as on a standalone server.
 - **Commits the server reports aborted.** On MongoDB, a commit that the server reports aborted (`NoSuchTransaction`
   (251) or `WriteConflict` (112)) stored nothing, so `withTransaction()` runs the callback again, within its usual 2
-  retries, whether it was the first commit or a retry; when the retries run out it throws `Utopia\Database\Exception`
+  retries (20 for a write conflict, after a short randomised wait, so concurrent writes to one document all land),
+  whether it was the first commit or a retry; when the retries run out it throws `Utopia\Database\Exception`
   with an `Exception\Transaction` cause. 7.x reported a first commit the server had aborted as a success, so the
   callback's writes were lost while the call returned normally. MongoDB aborts the whole transaction on a failed
   write in it, so a callback that catches a failed write (a `Duplicate`, for example) and carries on now runs again
@@ -2009,8 +2006,10 @@ $validator = new IndexDefinition($attributes, $indexes, $database->profile());
   it starts; `setStatus()`, `enable()`, `disable()` and `reset()` change the shared status unless called inside
   such a scope (see [Coroutines](#coroutines)). `setDefaultStatus()` is a constructor argument,
   `new Authorization(bool $defaultStatus = true)`. `restore()` is internal.
-- `Validator\Structure` takes `array $storedAttributes = []`: the attributes whose values are the stored ones, which
-  it does not validate again. `Database::updateDocument()` passes it.
+- `Validator\Structure` takes `array $storedAttributes = []`: the attributes whose values are the stored ones. They
+  are validated as in 7.x, so an update fails on a stored value a narrowed definition no longer admits, but a stored
+  object value is held to what 7.x accepted (any JSON string or empty value). `Database::updateDocument()` passes
+  it.
 - `Database::convertQueries()` takes an optional `array $joinedCollections` (join alias => collection). With it,
   filters on `alias.attribute`, the filters of join ON lists and `having()` conditions in the list are converted
   too; aggregates and selects in the list are left as they are. Without it the method converts as before.
@@ -2021,7 +2020,7 @@ Nothing in the library, Appwrite, Appwrite Cloud or utopia-php/migration calls t
 
 | Removed | Replacement |
 |---|---|
-| `Adapter\SQL::setFloatPrecision(int $precision)` | Floats are bound with 17 digits. A subclass can set the protected `$floatPrecision` property |
+| `Adapter\SQL::setFloatPrecision(int $precision)` | As in 7.x, a `find()` binds floats in fixed-point with 17 decimals and every other statement binds them as PHP writes them. A subclass can set the protected `$floatPrecision` property |
 | `Adapter\SQLite::setEmulateMySQL()`, `getEmulateMySQL()` | A subclass sets the protected `$emulateMySQL` property to `true` |
 | `Database::getInstanceFilters()` | The codecs given to the constructor, or `getFilters()` |
 | `Mirror::getWriteFilters()` | The `$filters` given to the constructor. A subclass reads the protected `$writeFilters` property |
@@ -2186,7 +2185,8 @@ $database->find('reviews', [
   `Utopia\Database\Exception\Query` with `Join queries are not supported for bulk updates` or
   `Join queries are not supported for bulk deletes`.
 - On adapters without joins or aggregations (Memory, MongoDB, Redis), `find()`, `aggregate()`, `count()` and
-  `sum()` reject those queries during validation with `Invalid query method: <method>`.
+  `sum()` reject those queries during validation with `Invalid query: Invalid query method: <method>`, the message
+  7.x gave for a method it could not parse.
 - MariaDB, MySQL and SQLite run a full outer join as two queries joined by `UNION ALL`. They accept one full outer
   join per query, and a right join after it has to join on a table joined before the full outer join or on the full
   outer joined table (directly or through other joins); other chains throw `Utopia\Database\Exception\Query`.

src/Database/Adapter/Memory.php

@@ -1884,8 +1884,10 @@ public function increaseDocumentAttribute(Document $collection, string $id, stri
         $previousValue = $this->data[$key]['documents'][$docKey][$column] ?? null;
         $previousUpdatedAt = $this->data[$key]['documents'][$docKey][Storage::UPDATED_AT] ?? null;
         $current = $previousValue ?? 0;
-        $exact = (\is_int($current) || (\is_string($current) && BigInt::isIntegerString($current)))
-            && (\is_int($value) || (\is_string($value) && BigInt::isIntegerString($value)));
+        $exact = BigInt::isIntegerValue($current)
+            && BigInt::isIntegerValue($value)
+            && ($min === null || BigInt::isIntegerValue($min))
+            && ($max === null || BigInt::isIntegerValue($max));
 
         // MariaDB encodes the bound check as part of the WHERE clause against
         // the current column value (`attr <= :max` / `attr >= :min`); when the