diff --git a/CHANGELOG.md b/CHANGELOG.md index f51922491..66a431432 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,50 @@ # Changelog +## 8.0.1 + +8.0.1 restores the 7.4.1 behaviour that 8.0.0 changed for callers of features 7.x already had. New 8.0 features are +unchanged. + +### Fixed + +- Query validation reports what 7.x reported: the children of `and()`, `or()` and `elemMatch()` are validated before + the query itself, the first invalid query in reading order sets the message even when a later one does not parse, + a method 7.x could not parse that no validator takes reads `Invalid query: Invalid query method: `, and a + raw query from a string reads `Invalid query method: raw`. +- `select('$tenant')` passes query validation again without shared tables, and the read refuses it with `Cannot + select attributes: $tenant`, as in 7.x. +- The `UID` validator description is the 7.x text again, also inside `Invalid cursor: …`. +- On PostgreSQL an exact search (`"foo bar"`) matches every word in any order again, not the adjacent phrase. +- `containsAll()` on an attribute that is not an array matches each value as a whole-value `LIKE` (`ILIKE` on + PostgreSQL) pattern, any of them, on MariaDB, MySQL and PostgreSQL, as in 7.x. +- A read whose orders include `orderRandom()` ignores its cursor instead of failing with `Order attribute '' is + empty`. +- `document_purge` fires inside the write's outermost transaction, just before it commits, so a listener that throws + rolls the write back and its failure reaches the caller, as in 7.x. +- `increaseDocumentAttribute()` and `decreaseDocumentAttribute()` pass a fractional change, `max` or `min` on an + integer attribute to the engine again and return the exact sum, and refuse a change that is not above zero with + `\InvalidArgumentException`, as in 7.x. +- Numeric operators on integer and double attributes are checked as in 7.x: a fractional change or limit, or a + fractional item appended to an integer array, is no longer refused. +- A mismatched attribute default reads `Default value x does not match given type ` (and `is not a valid + integer string for type bigint`) without JSON quoting, as in 7.x. +- `Index::fromArray()` refuses an unknown index type with `Exception\Index` (`Unknown index type: …`) instead of + creating a key index. +- `Exception\Unique::MESSAGE` is `Unique index violation` again. +- A many-to-many relationship created by 7.x can be renamed again: its junction index, which 7.x stored with `key` + `index_`, is read under its `$id` `_index_`. +- The SQL adapters bind floats as 7.x did: fixed-point only in `find()`, as PHP writes them elsewhere, so a write + no longer stores `1e-300` or `-2.5e-20` as `0`. +- Two-level relationship selects (`author.profile.*`, `tags.books.name`) return the related documents' attributes + as in 7.x, and `select(['*', '*.*'])` reads every column instead of failing on MariaDB, MySQL and PostgreSQL. +- A single SQL read returns `$sequence` ahead of `$id`, in 7.x's key order. +- An update validates the stored values it leaves unchanged again, as 7.x did; a stored object value is held to + 7.x's object rules. +- On MongoDB a transaction that lost a write conflict runs again up to 20 times, so concurrent increments of one + document all land. +- With dates preserved, linking existing children through their one-to-many parent keeps each child's stored + `$updatedAt`, as 7.x did; unlinking still stamps the child. + ## 8.0.0 (unreleased) 8.0 is a major release. Read [UPGRADE.md](UPGRADE.md) before you upgrade from 7.x: it lists every change you may diff --git a/UPGRADE.md b/UPGRADE.md index 5871c5e34..6f97db412 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -223,9 +223,12 @@ $permissions = [Permission::read(Role::user(Id::unique()))]; - A filter on a path into an object attribute (`meta.address.city`) takes keys of `a-z`, `A-Z`, `0-9`, `_` and `-` only. The query validator refuses any other key with `Utopia\Database\Exception\Query` on every adapter (7.x refused such keys on PostgreSQL only), and on PostgreSQL the query is refused also when validation is skipped. -- On PostgreSQL, an exact search (`search('title', '"foo bar"')`) matches the words as an adjacent phrase, as on - MariaDB, MySQL and SQLite, and `notSearch()` with an exact term excludes only that phrase. In 7.4.0 PostgreSQL - matched both words in any order. To match both words in any order, pass a `search()` for each word. +- On PostgreSQL, an exact search (`search('title', '"foo bar"')`) matches documents holding every word in any order, + as in 7.x, where MariaDB, MySQL and SQLite match the adjacent phrase. +- `containsAll()` on an attribute that is not an array matches as in 7.x: on MariaDB, MySQL and PostgreSQL each value + is a `LIKE` (`ILIKE`) pattern on the whole value and a document matching any of them is returned; on MongoDB it is + `$all`. Use `containsString()` for substrings. +- A read whose orders include `orderRandom()` ignores its cursor, as in 7.x. ## Schema: value objects @@ -696,13 +699,14 @@ It fires once per written document from `updateDocument()` (for both the old and `Event\Document\Purged` with the document's `collection` and `id`. As in 7.x, `createDocument()` and `createDocuments()` do not fire it. Attribute schema changes fire it for the collection's metadata document (`$collection` = `_metadata`). -A write fires it after the outermost transaction commits: inside `withTransaction()` the events of every write wait -for the outer commit, a rollback drops them, and a retried attempt announces once. Each event runs under the tenant -and the `silent()` scope in force when its document was written. If the cache invalidation after the commit fails, -the write throws with its data committed, after `document_purge` has fired. `purgeCachedDocument()` fires it at once. -On an adapter without savepoints (MongoDB), a nested `withTransaction()` that fails is not rolled back on its own: -when the caller catches the failure, the nested writes commit with the caller and their events fire after that -commit. +As in 7.x, a write fires it inside its transaction, so a listener that throws rolls the write back and its failure +reaches the caller. The events wait for the end of the outermost transaction and fire just before it commits: inside +`withTransaction()` the events of every write fire together before the outer commit, a rollback before that point +drops them, and every attempt of a retried transaction that reaches its commit announces them. Each event runs under +the tenant and the `silent()` scope in force when its document was written. If the cache invalidation after the +commit fails, the write throws with its data committed. `purgeCachedDocument()` fires it at once. On an adapter +without savepoints (MongoDB), a nested `withTransaction()` that fails is not rolled back on its own: when the caller +catches the failure, the nested writes commit with the caller and their events fire before that commit. ### `document_update` for related documents a delete changed @@ -865,27 +869,20 @@ method as a no-op, so a hook overrides only what it needs: counted and emitted it). Of a batch that repeats an id, only the first copy is written. Permissions are written only for inserted documents: in 7.x the skipped copy's permissions were added to the stored document on MariaDB, MySQL and SQLite, so `find()`, `count()` and `sum()` could return it to roles its own permissions do not grant. -- **Fractional numbers on integer attributes.** `increaseDocumentAttribute()` and `decreaseDocumentAttribute()` - throw `Utopia\Database\Exception\Type` before anything is written, on every adapter and also on schemaless - collections that declare the attribute as an integer: - - for a fractional change value (`Change value must be an integer.`). 7.x passed it to the engine, which rounded - it on MariaDB and MySQL, failed on PostgreSQL, and stored a float in the integer attribute on SQLite, MongoDB, - Memory and Redis. Pass an integer change value, or use a float attribute for fractional counters. - - for a fractional `max` or `min` (`Max must be an integer.`, `Min must be an integer.`). Integer bounds are - compared with exact integer arithmetic, so 64-bit and unsigned values never pass through a float. The bound - accepts the same whole numbers as an operator limit: an integer, an integer string, a string with only zero - decimals such as `'102.0'`, or a float without a fractional part such as `102.0`, each converted exactly. Pass - a whole bound, for example `floor($max)` or `ceil($min)`, which admits the same integer values. - - Change values and bounds on float and double attributes may still be fractional. -- **Operator limits on integer attributes.** The `max` or `min` limit of `Operator::increment()`, `decrement()`, - `multiply()`, `divide()` and `power()` on an integer or big integer attribute has to be a whole number: an - integer, an integer string, or a float without a fractional part such as `9.0e18`. A fractional limit such as - `102.4` is refused before anything is written with `Utopia\Database\Exception\Structure` - (`Cannot apply operator: max/min limit must be a whole number for integer attribute '', got - `). With validation skipped that check does not run, and Memory and Redis refuse such a limit with - `Utopia\Database\Exception\Operator` when the result leaves PHP's integer range. Limits on float and double - attributes are unchanged. +- **Fractional numbers on integer attributes.** As in 7.x, `increaseDocumentAttribute()` and + `decreaseDocumentAttribute()` pass a fractional change value, `max` or `min` on an integer attribute to the + engine and return the document with the exact sum: MariaDB and MySQL store it rounded, MongoDB stores it and reads + it back truncated, and PostgreSQL fails. A whole float such as `2.0` is bound as the integer. A whole change value + and whole bounds are compared with exact integer arithmetic, so 64-bit and unsigned values never pass through a + float. A change value that is not a number greater than 0 throws `\InvalidArgumentException` + (`Value must be numeric and greater than 0`), as in 7.x. +- **Operator limits on integer attributes.** As in 7.x, the change and the `max` or `min` limit of + `Operator::increment()`, `decrement()`, `multiply()`, `divide()` and `power()` on an integer or double attribute + only have to be numeric and finite: a fractional change or limit on an integer reaches the engine, which stores + the result as for a fractional counter. Without a limit, a result predicted outside the attribute's range is + refused with `Utopia\Database\Exception\Structure` (`Cannot apply operator: would overflow maximum value + of `). On a big integer attribute the limit has to be a whole number in the attribute's range (`Cannot apply + operator: max/min limit must be a whole number for integer attribute '', got `). - **Operators on upserts that create a document.** An upsert that creates a document applies every operator to the attribute's default, as it does for an existing document: `dateAddDays()` and `dateSubDays()` shift the date, `arrayFilter()` filters the array, and the maximum or minimum of increment, decrement, multiply, divide and power @@ -1091,14 +1088,13 @@ coroutine that opened it and the coroutines it starts; see [Pools and profiling] `public readonly ?string $state`; a numeric string is also the integer code. `Exception\Order::__construct(string $message, ?string $attribute = null, int|string $code = 0, ?\Throwable $previous = null)` takes the attribute second. -- **Unique index violations.** Every adapter now reports a unique index violation as - `Utopia\Database\Exception\Unique` with the message `Document with the requested unique attributes already exists` - (7.x: `Unique index violation`). The class and its hierarchy are unchanged: `Unique` extends `Duplicate`, and a - conflicting document `$id` still throws a plain `Duplicate` with `Document already exists`. Match on the class, - not the message: catch `Unique` before `Duplicate` to tell the two apart. `Exception\Unique` has no constructor of - its own and never rewrites the message it is given. The message is `Exception\Unique::MESSAGE`, also when - `createIndex()` refuses a unique index over documents that already share a value (7.4.1: `Unique index violation`, - and `Cannot create unique index: existing rows already contain duplicate values` on Redis). +- **Unique index violations.** Every adapter reports a unique index violation as + `Utopia\Database\Exception\Unique` with the 7.x message `Unique index violation`. The class and its hierarchy are + unchanged: `Unique` extends `Duplicate`, and a conflicting document `$id` still throws a plain `Duplicate` with + `Document already exists`. Match on the class, not the message: catch `Unique` before `Duplicate` to tell the two + apart. `Exception\Unique` has no constructor of its own and never rewrites the message it is given. The message is + `Exception\Unique::MESSAGE`, also when `createIndex()` refuses a unique index over documents that already share a + value (7.4.1 Redis: `Cannot create unique index: existing rows already contain duplicate values`). - **`ignoreDuplicates()` on PostgreSQL** skips only a document whose id is stored, as in 7.x: a new id that collides on another unique index throws `Utopia\Database\Exception\Unique`. MariaDB, MySQL and SQLite cannot name the index to ignore and, as in 7.x, skip such a row without error. @@ -1141,7 +1137,8 @@ coroutine that opened it and the coroutines it starts; see [Pools and profiling] below). On a sharded cluster (`mongos`) the adapter runs without transactions, as on a standalone server. - **Commits the server reports aborted.** On MongoDB, a commit that the server reports aborted (`NoSuchTransaction` (251) or `WriteConflict` (112)) stored nothing, so `withTransaction()` runs the callback again, within its usual 2 - retries, whether it was the first commit or a retry; when the retries run out it throws `Utopia\Database\Exception` + retries (20 for a write conflict, after a short randomised wait, so concurrent writes to one document all land), + whether it was the first commit or a retry; when the retries run out it throws `Utopia\Database\Exception` with an `Exception\Transaction` cause. 7.x reported a first commit the server had aborted as a success, so the callback's writes were lost while the call returned normally. MongoDB aborts the whole transaction on a failed write in it, so a callback that catches a failed write (a `Duplicate`, for example) and carries on now runs again @@ -2009,8 +2006,10 @@ $validator = new IndexDefinition($attributes, $indexes, $database->profile()); it starts; `setStatus()`, `enable()`, `disable()` and `reset()` change the shared status unless called inside such a scope (see [Coroutines](#coroutines)). `setDefaultStatus()` is a constructor argument, `new Authorization(bool $defaultStatus = true)`. `restore()` is internal. -- `Validator\Structure` takes `array $storedAttributes = []`: the attributes whose values are the stored ones, which - it does not validate again. `Database::updateDocument()` passes it. +- `Validator\Structure` takes `array $storedAttributes = []`: the attributes whose values are the stored ones. They + are validated as in 7.x, so an update fails on a stored value a narrowed definition no longer admits, but a stored + object value is held to what 7.x accepted (any JSON string or empty value). `Database::updateDocument()` passes + it. - `Database::convertQueries()` takes an optional `array $joinedCollections` (join alias => collection). With it, filters on `alias.attribute`, the filters of join ON lists and `having()` conditions in the list are converted too; aggregates and selects in the list are left as they are. Without it the method converts as before. @@ -2021,7 +2020,7 @@ Nothing in the library, Appwrite, Appwrite Cloud or utopia-php/migration calls t | Removed | Replacement | |---|---| -| `Adapter\SQL::setFloatPrecision(int $precision)` | Floats are bound with 17 digits. A subclass can set the protected `$floatPrecision` property | +| `Adapter\SQL::setFloatPrecision(int $precision)` | As in 7.x, a `find()` binds floats in fixed-point with 17 decimals and every other statement binds them as PHP writes them. A subclass can set the protected `$floatPrecision` property | | `Adapter\SQLite::setEmulateMySQL()`, `getEmulateMySQL()` | A subclass sets the protected `$emulateMySQL` property to `true` | | `Database::getInstanceFilters()` | The codecs given to the constructor, or `getFilters()` | | `Mirror::getWriteFilters()` | The `$filters` given to the constructor. A subclass reads the protected `$writeFilters` property | @@ -2186,7 +2185,8 @@ $database->find('reviews', [ `Utopia\Database\Exception\Query` with `Join queries are not supported for bulk updates` or `Join queries are not supported for bulk deletes`. - On adapters without joins or aggregations (Memory, MongoDB, Redis), `find()`, `aggregate()`, `count()` and - `sum()` reject those queries during validation with `Invalid query method: `. + `sum()` reject those queries during validation with `Invalid query: Invalid query method: `, the message + 7.x gave for a method it could not parse. - MariaDB, MySQL and SQLite run a full outer join as two queries joined by `UNION ALL`. They accept one full outer join per query, and a right join after it has to join on a table joined before the full outer join or on the full outer joined table (directly or through other joins); other chains throw `Utopia\Database\Exception\Query`. diff --git a/src/Database/Adapter/Memory.php b/src/Database/Adapter/Memory.php index 071f1fd2c..e5fc1dab6 100644 --- a/src/Database/Adapter/Memory.php +++ b/src/Database/Adapter/Memory.php @@ -1884,8 +1884,10 @@ public function increaseDocumentAttribute(Document $collection, string $id, stri $previousValue = $this->data[$key]['documents'][$docKey][$column] ?? null; $previousUpdatedAt = $this->data[$key]['documents'][$docKey][Storage::UPDATED_AT] ?? null; $current = $previousValue ?? 0; - $exact = (\is_int($current) || (\is_string($current) && BigInt::isIntegerString($current))) - && (\is_int($value) || (\is_string($value) && BigInt::isIntegerString($value))); + $exact = BigInt::isIntegerValue($current) + && BigInt::isIntegerValue($value) + && ($min === null || BigInt::isIntegerValue($min)) + && ($max === null || BigInt::isIntegerValue($max)); // MariaDB encodes the bound check as part of the WHERE clause against // the current column value (`attr <= :max` / `attr >= :min`); when the diff --git a/src/Database/Adapter/Mongo.php b/src/Database/Adapter/Mongo.php index ffec5e843..1f87731d6 100644 --- a/src/Database/Adapter/Mongo.php +++ b/src/Database/Adapter/Mongo.php @@ -92,6 +92,18 @@ class Mongo extends Adapter implements Feature\Casting, Feature\Connection, Feat private const int DEFAULT_BATCH_SIZE = 1000; + /** + * How many times a transaction that lost a write conflict (112) runs again. Concurrent writes to one document, + * such as counters, conflict on every attempt but one, and 7.x's increments did not fail under that load. + */ + private const int WRITE_CONFLICT_RETRIES = 20; + + /** + * Microseconds of the shortest wait before a transaction that lost a write conflict runs again; the wait is + * randomised so the writers that conflicted do not retry in step. + */ + private const int WRITE_CONFLICT_SLEEP = 5_000; + /** * The collation of the `_uid` index: a lookup or upsert by id must use it to match what the * index treats as the same id. @@ -561,7 +573,7 @@ public function withTransaction(callable $callback): mixed $sleep = 50_000; $retries = 2; - for ($attempts = 0; $attempts <= $retries; $attempts++) { + for ($attempts = 0; ; $attempts++) { try { $this->startTransaction(); $result = $callback(); @@ -582,6 +594,12 @@ public function withTransaction(callable $callback): mixed throw $action; } + if (self::isWriteConflict($action) && $attempts < self::WRITE_CONFLICT_RETRIES) { + \usleep(\random_int(self::WRITE_CONFLICT_SLEEP, 4 * self::WRITE_CONFLICT_SLEEP) * \min($attempts + 1, 5)); + + continue; + } + if ($attempts < $retries) { \usleep($sleep * ($attempts + 1)); @@ -591,8 +609,17 @@ public function withTransaction(callable $callback): mixed throw $action; } } + } + + private static function isWriteConflict(Throwable $failure): bool + { + for ($error = $failure; $error !== null; $error = $error->getPrevious()) { + if ($error->getCode() === 112) { + return true; + } + } - throw new TransactionException('Transaction retry loop exited unexpectedly'); + return false; } /** diff --git a/src/Database/Adapter/Redis.php b/src/Database/Adapter/Redis.php index 3935445bd..2ebd43fd0 100644 --- a/src/Database/Adapter/Redis.php +++ b/src/Database/Adapter/Redis.php @@ -1589,8 +1589,10 @@ public function increaseDocumentAttribute(Document $collection, string $id, stri $document = $this->decode($payload); $current = $document->getAttribute($attribute); - $exact = (\is_int($current) || (\is_string($current) && BigInt::isIntegerString($current))) - && (\is_int($value) || (\is_string($value) && BigInt::isIntegerString($value))); + $exact = BigInt::isIntegerValue($current) + && BigInt::isIntegerValue($value) + && ($min === null || BigInt::isIntegerValue($min)) + && ($max === null || BigInt::isIntegerValue($max)); if ($exact) { $current = BigInt::toNative($current); $value = BigInt::toNative($value); diff --git a/src/Database/Adapter/SQL.php b/src/Database/Adapter/SQL.php index 058d6d4f8..154f7a064 100644 --- a/src/Database/Adapter/SQL.php +++ b/src/Database/Adapter/SQL.php @@ -646,7 +646,7 @@ public function getDocument(Document $collection, string $id, array $queries = [ $this->remapRow($row); - return Document::fromRow($row); + return Document::fromRow(self::sequenceBeforeId($row)); } if ($this->queriesHaveJoins($queries)) { @@ -705,7 +705,33 @@ public function getDocument(Document $collection, string $id, array $queries = [ $this->remapRow($document); - return Document::fromRow($document); + return Document::fromRow(self::sequenceBeforeId($document)); + } + + /** + * A single read returns `$sequence` ahead of `$id`, in the key order 7.x returned it. + * + * @param array $row + * @return array + */ + private static function sequenceBeforeId(array $row): array + { + if (! \array_key_exists(Document::SEQUENCE, $row) || ! \array_key_exists(Document::ID, $row)) { + return $row; + } + + $ordered = []; + foreach ($row as $key => $value) { + if ($key === Document::SEQUENCE) { + continue; + } + if ($key === Document::ID) { + $ordered[Document::SEQUENCE] = $row[Document::SEQUENCE]; + } + $ordered[$key] = $value; + } + + return $ordered; } /** @@ -3498,6 +3524,9 @@ protected function writeContext(array $skipPermissions = []): WriteContext * Prepares the SQL statement and binds positional parameters from the Statement. * Does NOT call execute() - the caller is responsible for that. * + * Floats are bound as 7.x bound them: a find() binds them in fixed-point notation, every other statement as + * PHP writes them, so a write keeps a magnitude fixed point would round to zero. + * * @param string $collection The collection the statement reads or writes, for the profiler * @return PDOStatement|DatabasePDOStatement|PDOStatementProxy */ @@ -3509,8 +3538,10 @@ protected function executeResult(Statement $result, ?Event $event = null, string if (\is_bool($value) && $this->supports(Capability::IntegerBooleans)) { $value = (int) $value; } - if (\is_float($value)) { + if (\is_float($value) && $event === Event::DocumentFind) { $prepared->bindValue($i + 1, $this->getFloatPrecision($value), PDO::PARAM_STR); + } elseif (\is_float($value)) { + $prepared->bindValue($i + 1, $value, PDO::PARAM_STR); } else { $prepared->bindValue($i + 1, $value, $this->getPdoType($value)); } @@ -4628,6 +4659,9 @@ private function configureFindBuilder( $hasDistinct ? $selections : [...$selections, ...$orderAttributes], ); $hasSelectionProjection = true; + } elseif (! empty($selections)) { + // As in 7.x, `*` selects every column and the terms next to it, such as `*.*`, add nothing. + $queries = \array_values(\array_filter($queries, static fn (BaseQuery $query): bool => $query->getMethod() !== Method::Select)); } } diff --git a/src/Database/Builder/MariaDB.php b/src/Database/Builder/MariaDB.php index 8381a0692..3ed662874 100644 --- a/src/Database/Builder/MariaDB.php +++ b/src/Database/Builder/MariaDB.php @@ -5,11 +5,13 @@ use Utopia\Query\Builder\MariaDB as Base; /** - * The MariaDB builder, which also compiles filters on their own and prepares search terms as 7.x did. + * The MariaDB builder, which also compiles filters on their own and prepares search terms and matches containsAll() + * as 7.x did. */ class MariaDB extends Base implements Filtering, Scoping { use CompilesFilters; + use MatchesContainsAllPatterns; use PreparesSearchTerms; use RefusesJoinedWrites; use ScopesCollections; diff --git a/src/Database/Builder/MatchesContainsAllPatterns.php b/src/Database/Builder/MatchesContainsAllPatterns.php new file mode 100644 index 000000000..e561ed61f --- /dev/null +++ b/src/Database/Builder/MatchesContainsAllPatterns.php @@ -0,0 +1,26 @@ + $values + */ + #[\Override] + protected function compileContainsAll(string $attribute, array $values, ?string $column = null): string + { + $like = $this->getLikeKeyword(); + $parts = []; + foreach ($values as $value) { + $this->addBinding($value, $column); + $parts[] = $attribute.' '.$like.' ?'; + } + + return '('.\implode(' OR ', $parts).')'; + } +} diff --git a/src/Database/Builder/MySQL.php b/src/Database/Builder/MySQL.php index 1e2b78322..3a4dee005 100644 --- a/src/Database/Builder/MySQL.php +++ b/src/Database/Builder/MySQL.php @@ -5,11 +5,13 @@ use Utopia\Query\Builder\MySQL as Base; /** - * The MySQL builder, which also compiles filters on their own and prepares search terms as 7.x did. + * The MySQL builder, which also compiles filters on their own and prepares search terms and matches containsAll() + * as 7.x did. */ class MySQL extends Base implements Filtering, Scoping { use CompilesFilters; + use MatchesContainsAllPatterns; use PreparesSearchTerms; use RefusesJoinedWrites; use ScopesCollections; diff --git a/src/Database/Builder/Postgres.php b/src/Database/Builder/Postgres.php index 495046306..f5dec25f5 100644 --- a/src/Database/Builder/Postgres.php +++ b/src/Database/Builder/Postgres.php @@ -12,13 +12,14 @@ use Utopia\Query\Schema\ColumnType; /** - * The PostgreSQL builder, which also compiles filters on their own, prepares search terms as 7.x did, writes a - * filter on a path into an object attribute only when every key of the path is a plain key, and lets a - * distanceLessThan() filter use the spatial index. + * The PostgreSQL builder, which also compiles filters on their own, prepares and matches search terms and containsAll() + * as 7.x did, writes a filter on a path into an object attribute only when every key of the path is a plain key, and + * lets a distanceLessThan() filter use the spatial index. */ class Postgres extends Base implements Filtering, Scoping { use CompilesFilters; + use MatchesContainsAllPatterns; use PreparesSearchTerms; use ScopesCollections; @@ -48,6 +49,29 @@ public function deleteUsing(string $table, string $condition, mixed ...$bindings return parent::deleteUsing($table, $condition, ...$bindings); } + /** + * 7.x handed websearch_to_tsquery() an exact term in single quotes, which match every word in any order where + * double quotes would match the adjacent phrase, and any other term with its words joined by `or`. + * + * @param array $values + */ + #[\Override] + protected function compileSearchExpression(string $attribute, array $values, bool $not): string + { + $term = $values[0] ?? ''; + $term = \is_string($term) ? $term : ''; + $words = $this->searchWords($term); + + if ($words === '') { + return $not ? '1 = 1' : '1 = 0'; + } + + $this->addBinding("'".($this->isExactSearch($term) ? $words : \str_replace(' ', ' or ', $words))."'"); + $match = "to_tsvector(regexp_replace({$attribute}, '[^\\w]+', ' ', 'g')) @@ websearch_to_tsquery(?)"; + + return $not ? "NOT ({$match})" : $match; + } + /** * @throws QueryException */ diff --git a/src/Database/Builder/PreparesSearchTerms.php b/src/Database/Builder/PreparesSearchTerms.php index f8d0909db..c1b06c944 100644 --- a/src/Database/Builder/PreparesSearchTerms.php +++ b/src/Database/Builder/PreparesSearchTerms.php @@ -28,15 +28,24 @@ protected function compileSearchExpression(string $attribute, array $values, boo private function prepareSearchTerm(string $term): string { - $exact = \str_starts_with($term, self::SEARCH_QUOTE) && \str_ends_with($term, self::SEARCH_QUOTE); + $words = $this->searchWords($term); - $words = \preg_replace(self::SEARCH_SEPARATORS, ' ', $term) ?? ''; - $words = \trim(\preg_replace(self::SEARCH_WHITESPACE, ' ', $words) ?? ''); - - if ($exact && $words !== '') { + if ($this->isExactSearch($term) && $words !== '') { return self::SEARCH_QUOTE.$words.self::SEARCH_QUOTE; } return $words; } + + private function isExactSearch(string $term): bool + { + return \str_starts_with($term, self::SEARCH_QUOTE) && \str_ends_with($term, self::SEARCH_QUOTE); + } + + private function searchWords(string $term): string + { + $words = \preg_replace(self::SEARCH_SEPARATORS, ' ', $term) ?? ''; + + return \trim(\preg_replace(self::SEARCH_WHITESPACE, ' ', $words) ?? ''); + } } diff --git a/src/Database/Exception/Unique.php b/src/Database/Exception/Unique.php index 603e847da..ee72cc2c2 100644 --- a/src/Database/Exception/Unique.php +++ b/src/Database/Exception/Unique.php @@ -7,5 +7,5 @@ */ class Unique extends Duplicate { - public const string MESSAGE = 'Document with the requested unique attributes already exists'; + public const string MESSAGE = 'Unique index violation'; } diff --git a/src/Database/Hook/Relationships.php b/src/Database/Hook/Relationships.php index e2b657f74..b73007138 100644 --- a/src/Database/Hook/Relationships.php +++ b/src/Database/Hook/Relationships.php @@ -1704,6 +1704,22 @@ public function populateDocuments(array $documents, Document $collection, int $f }); } + /** + * The attributes a related document keeps for the selects passed down to it, as 7.x read them: a nested path + * through one of its relationships keeps that relationship, or nothing more, and a select left with nothing kept + * keeps every attribute. The selects themselves are left as they are for the next level. + * + * @param array $selectQueries + * @return array + */ + private function selectsOfRelated(Document $relatedCollection, array $selectQueries): array + { + $selects = \array_map(static fn (Query $query): Query => clone $query, $selectQueries); + $this->processQueries(self::relationships($relatedCollection), $selects); + + return $selects; + } + /** * @param array $relationships The relationship attributes of the collection the queries read * @param array $queries @@ -2310,7 +2326,7 @@ private function populateOneToOneRelationshipsBatch(array $documents, string $ke $relatedById[$related->getId()] = $related; } - $this->database->applySelectFiltersToDocuments($relatedDocuments, $selectQueries); + $this->database->applySelectFiltersToDocuments($relatedDocuments, $this->selectsOfRelated($relatedCollection, $selectQueries)); foreach ($documentsByRelatedId as $relatedId => $docs) { if (isset($relatedById[$relatedId])) { @@ -2398,7 +2414,7 @@ private function populateOneToManyRelationshipsBatch(array $documents, string $k $relatedByParentId[$parentKey][] = $related; } - $this->database->applySelectFiltersToDocuments($relatedDocuments, $selectQueries); + $this->database->applySelectFiltersToDocuments($relatedDocuments, $this->selectsOfRelated($relatedCollection, $selectQueries)); foreach ($documents as $document) { $parentId = $document->getId(); @@ -2480,7 +2496,7 @@ private function populateManyToOneRelationshipsBatch(array $documents, string $k $relatedByChildId[$childKey][] = $related; } - $this->database->applySelectFiltersToDocuments($relatedDocuments, $selectQueries); + $this->database->applySelectFiltersToDocuments($relatedDocuments, $this->selectsOfRelated($relatedCollection, $selectQueries)); foreach ($documents as $document) { $childId = $document->getId(); @@ -2580,7 +2596,7 @@ private function populateManyToManyRelationshipsBatch(array $documents, Document $relatedById[$doc->getId()] = $doc; } - $this->database->applySelectFiltersToDocuments($allRelatedDocs, $selectQueries); + $this->database->applySelectFiltersToDocuments($allRelatedDocs, $this->selectsOfRelated($relatedCollection, $selectQueries)); foreach ($junctionsByDocumentId as $documentId => $relatedDocIds) { $documentRelated = []; @@ -2989,6 +3005,15 @@ private function deleteRelatedDocuments(string $collection, array $ids): void */ private function linkRelatedDocuments(Document $collection, string $twoWayKey, string $documentId, array $ids): void { + // 7.x linked each document on its own, so with dates preserved each kept its stored $updatedAt. + if ($this->database->isPreservingDates()) { + foreach (\array_values(\array_unique($ids)) as $id) { + $this->linkRelatedDocument($collection, $id, $twoWayKey, $documentId); + } + + return; + } + foreach (\array_chunk(\array_values(\array_unique($ids)), $this->relationQueryChunkSize()) as $chunk) { $linked = $this->database->skipRelationships(fn () => $this->database->updateDocuments( $collection->getId(), @@ -3033,10 +3058,15 @@ private function linkRelatedDocument(Document $collection, string $id, string $t $this->authorizeLink($collection, $related); + $link = [$twoWayKey => $documentId]; + if ($this->database->isPreservingDates()) { + $link[Document::UPDATED_AT] = $related->getUpdatedAt(); + } + $this->database->skipRelationships(fn () => $this->database->updateDocument( $collection->getId(), $id, - new Document([$twoWayKey => $documentId]), + new Document($link), )); } diff --git a/src/Database/Index.php b/src/Database/Index.php index d2479210e..de4012e00 100644 --- a/src/Database/Index.php +++ b/src/Database/Index.php @@ -133,7 +133,7 @@ public static function ttl(string $key, string $attribute, int $ttl): self public static function fromDocument(Document $document): self { return self::hydrate( - $document->getAttribute(self::KEY, $document->getId()), + self::storedKey($document), $document->getAttribute(self::TYPE, IndexType::Key->value), $document->getAttribute(self::ATTRIBUTES, []), $document->getAttribute(self::LENGTHS, []), @@ -149,6 +149,11 @@ public static function fromDocument(Document $document): self */ public static function fromArray(array $data): self { + $type = $data[self::TYPE] ?? null; + if (\is_string($type) && IndexType::tryFrom($type) === null) { + throw new IndexException(self::unknownTypeMessage($type)); + } + return self::hydrate( $data[self::KEY] ?? $data[Document::ID] ?? '', $data[self::TYPE] ?? IndexType::Key->value, @@ -159,6 +164,26 @@ public static function fromArray(array $data): self ); } + /** + * The key an index is stored under. 7.x wrote a many-to-many junction's first index with `$id` `_index_`, + * the name the engine holds, but `key` `index_`; that index is read by its `$id`. + */ + private static function storedKey(Document $document): mixed + { + $key = $document->getAttribute(self::KEY, $document->getId()); + $id = $document->getId(); + + return \is_string($key) && $id === '_'.$key && \str_starts_with($id, '_index_') ? $id : $key; + } + + /** + * @internal + */ + public static function unknownTypeMessage(string $type): string + { + return 'Unknown index type: '.$type.'. Must be one of '.IndexType::Key->value.', '.IndexType::Unique->value.', '.IndexType::Fulltext->value.', '.IndexType::Spatial->value.', '.IndexType::Object->value.', '.IndexType::HnswEuclidean->value.', '.IndexType::HnswCosine->value.', '.IndexType::HnswDot->value.', '.IndexType::Trigram->value.', '.IndexType::Ttl->value; + } + public function toDocument(): Document { $data = [ diff --git a/src/Database/Query.php b/src/Database/Query.php index 1d1990c97..9107f07fe 100644 --- a/src/Database/Query.php +++ b/src/Database/Query.php @@ -82,6 +82,10 @@ public static function parse(string $query, bool $allowRaw = false): static #[\Override] public static function parseQuery(array $query, bool $allowRaw = false): static { + if (! $allowRaw && ($query['method'] ?? null) === Method::Raw->value) { + throw new QueryException('Invalid query method: '.Method::Raw->value); + } + try { $parsed = parent::parseQuery(self::decodeNestedValues($query, $allowRaw), $allowRaw); diff --git a/src/Database/Trait/Attributes.php b/src/Database/Trait/Attributes.php index 130a0dec9..61c41a37c 100644 --- a/src/Database/Trait/Attributes.php +++ b/src/Database/Trait/Attributes.php @@ -780,6 +780,10 @@ protected function validateDefaultTypes(ColumnType $type, mixed $default, bool $ $value = \is_scalar($default) ? (string) $default : '[non-scalar]'; + if ($type === ColumnType::BigInteger && \is_string($default) && ! BigInt::isIntegerString($default, $signed)) { + throw new DatabaseException('Default value '.$value.' is not a valid integer string for type bigint'); + } + throw new DatabaseException('Default value '.$value.' does not match given type '.Attribute::storedType($type)); } diff --git a/src/Database/Trait/Documents.php b/src/Database/Trait/Documents.php index 827538f44..d2d3b4964 100644 --- a/src/Database/Trait/Documents.php +++ b/src/Database/Trait/Documents.php @@ -269,11 +269,35 @@ private function isDeclaredInteger(?Attribute $attribute): bool return $attribute !== null && ! $attribute->array && $attribute->isInteger(); } - private function assertIntegerChange(int|float|string $value): void + /** + * A change or bound that is not a whole number on an integer attribute: 7.x applied it as given, so the + * engine stored the sum its own way (MariaDB and MySQL round it) and the returned document held the sum. + */ + private function isFractionalChange(int|float|string $value, int|float|string|null $bound): bool + { + return \is_float($value) + || (\is_string($value) && ! BigInt::isIntegerString($value)) + || ($bound !== null && BigInt::integralValue($bound) === null); + } + + /** + * A whole float as the integer 7.x bound it as, which PostgreSQL takes for an integer column. + */ + private function wholeOrNative(int|float $value): int|float + { + return \is_float($value) && \is_finite($value) && \floor($value) === $value && \abs($value) < \PHP_INT_MAX + ? (int) $value + : $value; + } + + private function getFractionalResult(mixed $current, int|float|string $value, bool $increase): int|float { - if (! \is_int($value) && (! \is_string($value) || ! BigInt::isIntegerString($value))) { - throw new TypeException('Change value must be an integer.'); + if ($current !== null && ! \is_numeric($current)) { + throw new TypeException('Attribute value must be numeric.'); } + $current = $current === null ? 0 : $this->getNativeNumber($current); + + return $increase ? $current + $this->getNativeNumber($value) : $current - $this->getNativeNumber($value); } private function integerBound(int|float|string $bound, string $name): int|string @@ -2459,18 +2483,18 @@ public function increaseDocumentAttribute( $collection = $this->silent(fn () => $this->getCollection($collection)); $numericAttribute = $this->numericAttribute($collection, $attribute); + $declaredInteger = $this->isDeclaredInteger($numericAttribute ?? $this->declaredAttribute($collection, $attribute)); + $fractional = $declaredInteger && $this->isFractionalChange($value, $max); - if ($this->isDeclaredInteger($numericAttribute ?? $this->declaredAttribute($collection, $attribute))) { - $this->assertIntegerChange($value); - if ($max !== null) { - $max = $this->integerBound($max, 'Max'); - } + if ($declaredInteger && ! $fractional && $max !== null) { + $max = $this->integerBound($max, 'Max'); } + $integerArithmetic = $numericAttribute?->isInteger() === true && ! $fractional; $cacheTarget = $collection->getId() === self::METADATA ? new Document([Document::ID => $id, Document::COLLECTION => self::METADATA]) : $collection->getId(); - $document = $this->withMutation(Event::DocumentIncrease, $cacheTarget, function () use ($collection, $id, $attribute, $value, $max, $numericAttribute) { + $document = $this->withMutation(Event::DocumentIncrease, $cacheTarget, function () use ($collection, $id, $attribute, $value, $max, $numericAttribute, $fractional, $integerArithmetic) { /** @var Document $document */ $document = $this->authorization->skip(fn () => $this->silent(fn () => $this->readDocument($collection, $id, forUpdate: true))); // Skip ensures user does not need read permission for this @@ -2491,7 +2515,9 @@ public function increaseDocumentAttribute( $attributeExists = $document->offsetExists($attribute); $currentVal = $document->getAttribute($attribute); - if ($numericAttribute instanceof Attribute) { + if ($fractional) { + $result = $this->getFractionalResult($currentVal, $value, true); + } elseif ($numericAttribute instanceof Attribute) { $result = $this->getNumericResult($numericAttribute, $currentVal, $value, true); } else { if (! $attributeExists) { @@ -2503,7 +2529,7 @@ public function increaseDocumentAttribute( $result = $currentVal + $this->getNativeNumber($value); } $exceedsMaximum = ! \is_null($max) && ( - $numericAttribute?->isInteger() === true + $integerArithmetic ? BigInt::compare($result, $max) > 0 : $result > $max ); @@ -2515,18 +2541,22 @@ public function increaseDocumentAttribute( $updatedAt = $document->getUpdatedAt(); $updatedAt = (empty($updatedAt) || ! $this->datePreservation()->get()) ? $time : DateTime::setTimezone($updatedAt); if ($max !== null) { - $max = $numericAttribute?->isInteger() === true - ? BigInt::subtract($max, $value) - : $this->getNativeNumber($max) - $this->getNativeNumber($value); + $max = match (true) { + $integerArithmetic => BigInt::subtract($max, $value), + $fractional => $this->wholeOrNative($this->getNativeNumber($max) - $this->getNativeNumber($value)), + default => $this->getNativeNumber($max) - $this->getNativeNumber($value), + }; } $this->adapter->increaseDocumentAttribute( $collection, $id, $attribute, - $numericAttribute?->isInteger() === true - ? BigInt::toNative($value) - : $this->getNativeNumber($value), + match (true) { + $integerArithmetic => BigInt::toNative($value), + $fractional => $this->wholeOrNative($this->getNativeNumber($value)), + default => $this->getNativeNumber($value), + }, $updatedAt, max: $max ); @@ -2546,14 +2576,14 @@ public function increaseDocumentAttribute( } /** - * @throws TypeException + * @throws \InvalidArgumentException */ private function assertPositiveChange(int|float|string $value): void { if (! \is_numeric($value) || (\is_string($value) && BigInt::isIntegerString($value) ? BigInt::compare($value, 0) <= 0 : (float) $value <= 0)) { - throw new TypeException('Value must be numeric and greater than 0'); + throw new \InvalidArgumentException('Value must be numeric and greater than 0'); } } @@ -2569,7 +2599,7 @@ private function assertPositiveChange(int|float|string $value): void * * @throws AuthorizationException * @throws DatabaseException - * @throws TypeException When $value is not a number greater than 0 + * @throws \InvalidArgumentException When $value is not a number greater than 0 */ public function decreaseDocumentAttribute( string $collection, @@ -2583,18 +2613,18 @@ public function decreaseDocumentAttribute( $collection = $this->silent(fn () => $this->getCollection($collection)); $numericAttribute = $this->numericAttribute($collection, $attribute); + $declaredInteger = $this->isDeclaredInteger($numericAttribute ?? $this->declaredAttribute($collection, $attribute)); + $fractional = $declaredInteger && $this->isFractionalChange($value, $min); - if ($this->isDeclaredInteger($numericAttribute ?? $this->declaredAttribute($collection, $attribute))) { - $this->assertIntegerChange($value); - if ($min !== null) { - $min = $this->integerBound($min, 'Min'); - } + if ($declaredInteger && ! $fractional && $min !== null) { + $min = $this->integerBound($min, 'Min'); } + $integerArithmetic = $numericAttribute?->isInteger() === true && ! $fractional; $cacheTarget = $collection->getId() === self::METADATA ? new Document([Document::ID => $id, Document::COLLECTION => self::METADATA]) : $collection->getId(); - $document = $this->withMutation(Event::DocumentDecrease, $cacheTarget, function () use ($collection, $id, $attribute, $value, $min, $numericAttribute) { + $document = $this->withMutation(Event::DocumentDecrease, $cacheTarget, function () use ($collection, $id, $attribute, $value, $min, $numericAttribute, $fractional, $integerArithmetic) { /** @var Document $document */ $document = $this->authorization->skip(fn () => $this->silent(fn () => $this->readDocument($collection, $id, forUpdate: true))); // Skip ensures user does not need read permission for this @@ -2615,7 +2645,9 @@ public function decreaseDocumentAttribute( $attributeExists = $document->offsetExists($attribute); $currentDecVal = $document->getAttribute($attribute); - if ($numericAttribute instanceof Attribute) { + if ($fractional) { + $result = $this->getFractionalResult($currentDecVal, $value, false); + } elseif ($numericAttribute instanceof Attribute) { $result = $this->getNumericResult($numericAttribute, $currentDecVal, $value, false); } else { if (! $attributeExists) { @@ -2627,7 +2659,7 @@ public function decreaseDocumentAttribute( $result = $currentDecVal - $this->getNativeNumber($value); } $belowMinimum = ! \is_null($min) && ( - $numericAttribute?->isInteger() === true + $integerArithmetic ? BigInt::compare($result, $min) < 0 : $result < $min ); @@ -2639,18 +2671,22 @@ public function decreaseDocumentAttribute( $updatedAt = $document->getUpdatedAt(); $updatedAt = (empty($updatedAt) || ! $this->datePreservation()->get()) ? $time : DateTime::setTimezone($updatedAt); if ($min !== null) { - $min = $numericAttribute?->isInteger() === true - ? BigInt::add($min, $value) - : $this->getNativeNumber($min) + $this->getNativeNumber($value); + $min = match (true) { + $integerArithmetic => BigInt::add($min, $value), + $fractional => $this->wholeOrNative($this->getNativeNumber($min) + $this->getNativeNumber($value)), + default => $this->getNativeNumber($min) + $this->getNativeNumber($value), + }; } $this->adapter->increaseDocumentAttribute( $collection, $id, $attribute, - $numericAttribute?->isInteger() === true - ? BigInt::negate($value) - : $this->getNativeNumber($value) * -1, + match (true) { + $integerArithmetic => BigInt::negate($value), + $fractional => $this->wholeOrNative($this->getNativeNumber($value) * -1), + default => $this->getNativeNumber($value) * -1, + }, $updatedAt, min: $min ); @@ -3807,6 +3843,11 @@ private function fetch(Collection $collection, array $queries, PermissionType $f $joinedByAlias = $this->joinedCollectionsByAlias($joins, $joinedCollectionsById); $joinedCollections = $isAggregation ? [] : $joinedByAlias; + // 7.x read a random order with a cursor as a random order without one, so the cursor is dropped as it was. + if (\in_array(OrderDirection::Random, $orderTypes, true)) { + $cursor = null; + } + if ($joinedCollections !== [] && $cursor !== null) { [$orderAttributes, $cursor] = $this->qualifyJoinedOrders($collection, $orderAttributes, $cursor, $joinedCollections); } diff --git a/src/Database/Trait/Transactions.php b/src/Database/Trait/Transactions.php index da8eefdea..7c218eb66 100644 --- a/src/Database/Trait/Transactions.php +++ b/src/Database/Trait/Transactions.php @@ -166,13 +166,17 @@ private function withAdapterTransaction(callable $callback): mixed $discard = function () use ($context, $queued): void { \array_splice($this->documentPurgeEvents[$context], $queued); }; + $outermost = ! $this->adapter->inTransaction(); $returned = false; try { - return $this->adapter->withTransaction(function () use ($callback, $discard, &$returned): mixed { + return $this->adapter->withTransaction(function () use ($callback, $discard, $outermost, $context, &$returned): mixed { $returned = false; $discard(); $result = $callback(); + if ($outermost) { + $this->announceBeforeCommit($context); + } $returned = true; return $result; @@ -192,6 +196,24 @@ private function withAdapterTransaction(callable $callback): mixed } } + /** + * Fire the document purge events queued in the outermost transaction before it commits, as 7.x fired them + * from inside the write's transaction: a listener that fails rolls the write back and its failure is thrown, + * rather than the write committing and the request failing after it. + * + * @throws Throwable The first listener failure + */ + private function announceBeforeCommit(int $context): void + { + $events = $this->documentPurgeEvents[$context]; + $this->documentPurgeEvents[$context] = []; + + $failure = $this->announceDocumentPurges($events); + if ($failure !== null) { + throw $failure; + } + } + /** * Fire every queued document purge event, even after one of them fails. * @@ -214,10 +236,11 @@ private function announceDocumentPurges(array $events): ?Throwable /** * Keep all nested mutation tombstones blocked, and purge every written document - * again, once the outer transaction has committed or rolled back. Document purge - * events queued in the scope fire after a commit, even when the invalidation after it - * fails, and after a commit that could not be confirmed, whose failure is still the - * one thrown. A rollback drops them, also when its callback threw Exception\Unconfirmed. + * again, once the outer transaction has committed or rolled back. The outermost adapter + * transaction fires the document purge events queued in it before it commits; any queued + * outside one fire after the commit, even when the invalidation after it fails, and after a + * commit that could not be confirmed, whose failure is still the one thrown. A rollback + * drops them, also when its callback threw Exception\Unconfirmed. * * @template T * diff --git a/src/Database/Validator/AttributeDefinition.php b/src/Database/Validator/AttributeDefinition.php index 729bedc6c..f93509c9d 100644 --- a/src/Database/Validator/AttributeDefinition.php +++ b/src/Database/Validator/AttributeDefinition.php @@ -504,6 +504,14 @@ private function checkJsonEncodable(Attribute $attribute): void } } + /** + * A default as 7.x wrote it into the message: a scalar as PHP prints it, so a string default is not quoted. + */ + private static function defaultText(mixed $default): string + { + return \is_scalar($default) ? (string) $default : (string) \json_encode($default); + } + /** * @throws DatabaseException */ @@ -543,33 +551,35 @@ protected function validateDefaultTypes(ColumnType $type, mixed $default, bool $ case ColumnType::MediumText: case ColumnType::LongText: if ($defaultType !== 'string') { - $this->message = 'Default value '.json_encode($default).' does not match given type '.Attribute::storedType($type); + $this->message = 'Default value '.self::defaultText($default).' does not match given type '.Attribute::storedType($type); throw new DatabaseException($this->message); } break; case ColumnType::Integer: case ColumnType::Boolean: if ($type->value !== $defaultType) { - $this->message = 'Default value '.json_encode($default).' does not match given type '.Attribute::storedType($type); + $this->message = 'Default value '.self::defaultText($default).' does not match given type '.Attribute::storedType($type); throw new DatabaseException($this->message); } break; case ColumnType::BigInteger: if (! (new BigInt($signed, $this->profile->supports(Capability::UnsignedBigInt)))->isValid($default)) { - $this->message = 'Default value '.json_encode($default).' does not match given type '.Attribute::storedType($type); + $this->message = \is_string($default) && ! BigInt::isIntegerString($default, $signed) + ? 'Default value '.$default.' is not a valid integer string for type bigint' + : 'Default value '.self::defaultText($default).' does not match given type '.Attribute::storedType($type); throw new DatabaseException($this->message); } break; case ColumnType::Float: case ColumnType::Double: if ($defaultType !== 'double') { - $this->message = 'Default value '.json_encode($default).' does not match given type '.Attribute::storedType($type); + $this->message = 'Default value '.self::defaultText($default).' does not match given type '.Attribute::storedType($type); throw new DatabaseException($this->message); } break; case ColumnType::Datetime: if ($defaultType !== 'string') { - $this->message = 'Default value '.json_encode($default).' does not match given type '.Attribute::storedType($type); + $this->message = 'Default value '.self::defaultText($default).' does not match given type '.Attribute::storedType($type); throw new DatabaseException($this->message); } break; diff --git a/src/Database/Validator/IndexDefinition.php b/src/Database/Validator/IndexDefinition.php index 48264c711..2287138f0 100644 --- a/src/Database/Validator/IndexDefinition.php +++ b/src/Database/Validator/IndexDefinition.php @@ -264,7 +264,7 @@ public function checkValidIndex(Index $index): bool break; default: - $this->message = self::unknownTypeMessage($type->value); + $this->message = Index::unknownTypeMessage($type->value); return false; } @@ -280,7 +280,7 @@ private function checkStoredDefinition(Document $index): bool } if (! \is_string($type) || IndexType::tryFrom($type) === null) { - $this->message = self::unknownTypeMessage(\is_string($type) ? $type : ''); + $this->message = Index::unknownTypeMessage(\is_string($type) ? $type : ''); return false; } @@ -294,11 +294,6 @@ private function checkStoredDefinition(Document $index): bool return true; } - private static function unknownTypeMessage(string $type): string - { - return 'Unknown index type: '.$type.'. Must be one of '.IndexType::Key->value.', '.IndexType::Unique->value.', '.IndexType::Fulltext->value.', '.IndexType::Spatial->value.', '.IndexType::Object->value.', '.IndexType::HnswEuclidean->value.', '.IndexType::HnswCosine->value.', '.IndexType::HnswDot->value.', '.IndexType::Trigram->value.', '.IndexType::Ttl->value; - } - /** * Check that all index attributes exist in the collection schema. * diff --git a/src/Database/Validator/ObjectValue.php b/src/Database/Validator/ObjectValue.php index 3c335c07d..fbfb6edfa 100644 --- a/src/Database/Validator/ObjectValue.php +++ b/src/Database/Validator/ObjectValue.php @@ -6,6 +6,14 @@ class ObjectValue extends Validator { + /** + * @param bool $stored Whether the value is one already stored, which 7.x accepted as any JSON string or any + * empty value too + */ + public function __construct(private readonly bool $stored = false) + { + } + #[\Override] public function getDescription(): string { @@ -15,6 +23,10 @@ public function getDescription(): string #[\Override] public function isValid(mixed $value): bool { + if ($this->stored && (empty($value) || (\is_string($value) && \json_validate($value)))) { + return true; + } + if (is_string($value)) { $decoded = json_decode($value); diff --git a/src/Database/Validator/Operator.php b/src/Database/Validator/Operator.php index 082895af4..ec565483c 100644 --- a/src/Database/Validator/Operator.php +++ b/src/Database/Validator/Operator.php @@ -129,6 +129,93 @@ private function isValidLimit(mixed $limit, Attribute $attribute, DatabaseOperat return true; } + /** + * A numeric operator on an integer or double attribute is checked as 7.x checked it: the change and the limit + * only have to be numeric, so a fractional change or limit on an integer reaches the engine, which stores the + * result its own way. Without a limit, the result predicted from the current document has to fit the attribute. + * + * @param array $values + */ + private function isValidNumericChange(DatabaseOperator $operator, Attribute $attribute, array $values): bool + { + $method = $operator->getMethod(); + $methodName = $method->value; + + if (! isset($values[0]) || ! \is_numeric($values[0]) || ! \is_finite((float) $values[0])) { + $this->message = "Cannot apply {$methodName} operator: value must be numeric, got ".\gettype($operator->getValue()); + + return false; + } + + if (($method === OperatorType::Divide || $method === OperatorType::Modulo) && (float) $values[0] === 0.0) { + $this->message = "Cannot apply {$methodName} operator: ".($method === OperatorType::Divide ? 'division' : 'modulo').' by zero'; + + return false; + } + + if (\count($values) > 1 && $values[1] !== null && (! \is_numeric($values[1]) || ! \is_finite((float) $values[1]))) { + $this->message = "Cannot apply {$methodName} operator: max/min limit must be numeric, got ".\gettype($values[1]); + + return false; + } + + if ($this->currentDocument === null || ! $attribute->isInteger() || isset($values[1])) { + return true; + } + + $stored = $this->getIntegerValue($this->currentDocument->getAttribute($operator->getAttribute()) ?? 0); + $bounds = $attribute->bounds(); + if ($stored === null || $bounds === null || ! $this->isNumericValueInBounds($stored, $attribute)) { + $this->message = "Cannot apply {$methodName} operator: current value is outside the attribute range"; + + return false; + } + + $current = $this->getNumericValue($stored); + $change = $this->getNumericValue($values[0]); + if ($current === null || $change === null) { + return true; + } + $predicted = match ($method) { + OperatorType::Increment => $current + $change, + OperatorType::Decrement => $current - $change, + OperatorType::Multiply => $current * $change, + OperatorType::Divide => $current / $change, + OperatorType::Modulo => (int) $change === 0 ? $current : (int) $current % (int) $change, + default => $current ** $change, + }; + + if ($predicted > (float) $bounds->max) { + $this->message = "Cannot apply {$methodName} operator: would overflow maximum value of {$bounds->max}"; + + return false; + } + + if ($predicted < (float) $bounds->min) { + $this->message = "Cannot apply {$methodName} operator: would underflow minimum value of {$bounds->min}"; + + return false; + } + + return true; + } + + /** + * 7.x checked a numeric item appended to an integer array against the range only, so a fractional item passes + * here; a bigint array, new in 8.0, checks the item is a whole number in range. + */ + private function isArrayItemInBounds(int|float|string $item, Attribute $attribute): bool + { + if ($attribute->type === ColumnType::BigInteger) { + return $this->isNumericValueInBounds($item, $attribute); + } + + $bounds = $attribute->bounds(); + $numeric = $this->getNumericValue($item); + + return $bounds !== null && $numeric !== null && $numeric <= (float) $bounds->max && $numeric >= (float) $bounds->min; + } + private function getIntegerValue(mixed $value): int|string|null { if (\is_int($value)) { @@ -274,6 +361,14 @@ private function validateOperatorForAttribute( return false; } + if ($type !== ColumnType::BigInteger) { + if (! $this->isValidNumericChange($operator, $attribute, $values)) { + return false; + } + + break; + } + if (! isset($values[0]) || ! $this->isNumericValueInBounds($values[0], $attribute)) { $this->message = "Cannot apply {$methodName} operator: value must be numeric, got ".gettype($operator->getValue()); @@ -345,7 +440,7 @@ private function validateOperatorForAttribute( if (! empty($values) && $attribute->isInteger()) { $newItems = \is_array($values[0]) ? $values[0] : $values; foreach ($newItems as $item) { - if (\is_numeric($item) && ! $this->isNumericValueInBounds($item, $attribute)) { + if (\is_numeric($item) && ! $this->isArrayItemInBounds($item, $attribute)) { $bounds = $attribute->bounds(); if ($bounds === null) { return false; diff --git a/src/Database/Validator/Queries/Base.php b/src/Database/Validator/Queries/Base.php index c48c1c488..ac837e53d 100644 --- a/src/Database/Validator/Queries/Base.php +++ b/src/Database/Validator/Queries/Base.php @@ -15,10 +15,66 @@ use Utopia\Database\Validator\Query\Order; use Utopia\Database\Validator\Query\Select; use Utopia\Query\Method; +use Utopia\Query\Query as BaseQuery; use Utopia\Validator; class Base extends Validator { + /** + * The methods 7.x parsed. Any other method was a parse failure there, so when no validator takes it, + * its message keeps the "Invalid query: " prefix clients saw for it. + */ + private const array LEGACY_METHODS = [ + Method::Equal, + Method::NotEqual, + Method::LessThan, + Method::LessThanEqual, + Method::GreaterThan, + Method::GreaterThanEqual, + Method::Contains, + Method::ContainsAny, + Method::NotContains, + Method::Search, + Method::NotSearch, + Method::IsNull, + Method::IsNotNull, + Method::Between, + Method::NotBetween, + Method::StartsWith, + Method::NotStartsWith, + Method::EndsWith, + Method::NotEndsWith, + Method::Crosses, + Method::NotCrosses, + Method::DistanceEqual, + Method::DistanceNotEqual, + Method::DistanceGreaterThan, + Method::DistanceLessThan, + Method::Intersects, + Method::NotIntersects, + Method::Overlaps, + Method::NotOverlaps, + Method::Touches, + Method::NotTouches, + Method::VectorDot, + Method::VectorCosine, + Method::VectorEuclidean, + Method::Exists, + Method::NotExists, + Method::Select, + Method::OrderDesc, + Method::OrderAsc, + Method::OrderRandom, + Method::Limit, + Method::Offset, + Method::CursorAfter, + Method::CursorBefore, + Method::And, + Method::Or, + Method::ContainsAll, + Method::ElemMatch, + ]; + protected string $message = 'Invalid queries'; /** @@ -123,14 +179,15 @@ public function isValid(mixed $value): bool $aggregationAliases = []; $joinAliases = []; $hasJoins = false; + $parseFailure = null; foreach ($value as $q) { if (! $q instanceof Query) { try { $q = Query::parse($q); } catch (Throwable $e) { - $this->message = 'Invalid query: '.$e->getMessage(); + $parseFailure = 'Invalid query: '.$e->getMessage(); - return false; + break; } } @@ -191,134 +248,153 @@ public function isValid(mixed $value): bool $this->prepareAggregations($parsedQueries); - // Same pass: nested and/or children must keep the join aliases collected above. - $pending = $parsedQueries; - while ($pending !== []) { - $query = \array_shift($pending); - - if ($query->isNested() && ! $this->isValidLength($query->getValues(), $query->getMethod())) { + foreach ($parsedQueries as $query) { + if (! $this->isValidQuery($query, $hasFilterValidator)) { return false; } + } - if (\in_array($query->getMethod(), Query::LOGICAL_TYPES, true)) { - foreach ($query->getValues() as $nested) { - if (! $nested instanceof Query) { - if (! \is_string($nested)) { - $this->message = 'Invalid query: nested query must be a string'; - - return false; - } - try { - $nested = Query::parse($nested); - } catch (Throwable $e) { - $this->message = 'Invalid query: '.$e->getMessage(); - - return false; - } + if ($parseFailure !== null) { + $this->message = $parseFailure; + + return false; + } + + return true; + } + + /** + * Validate the children of a logical query before the query itself, as 7.x did, so a child's error is + * the one reported; a join's ON conditions come after the join. + */ + private function isValidQuery(BaseQuery $query, bool $hasFilterValidator): bool + { + if ($query->isNested() && ! $this->isValidLength($query->getValues(), $query->getMethod())) { + return false; + } + + if (\in_array($query->getMethod(), Query::LOGICAL_TYPES, true)) { + foreach ($query->getValues() as $nested) { + if (! $nested instanceof Query) { + if (! \is_string($nested)) { + $this->message = 'Invalid query: nested query must be a string'; + + return false; } - $pending[] = $nested; - } - } + try { + $nested = Query::parse($nested); + } catch (Throwable $e) { + $this->message = 'Invalid query: '.$e->getMessage(); - if ($hasFilterValidator && $query->getMethod()->isJoin() && $query->isNestedJoin()) { - foreach ($query->getJoinOnQueries() as $onQuery) { - if ($onQuery->getMethod() === Method::On) { - continue; + return false; } - $pending[] = $onQuery; + } + if (! $this->isValidQuery($nested, $hasFilterValidator)) { + return false; } } + } - $method = $query->getMethod(); + $method = $query->getMethod(); + + if ($method->isAggregate()) { + $methodType = QueryBase::METHOD_TYPE_AGGREGATE; + } else { + $methodType = match ($method) { + Method::Select => QueryBase::METHOD_TYPE_SELECT, + Method::Limit => QueryBase::METHOD_TYPE_LIMIT, + Method::Offset => QueryBase::METHOD_TYPE_OFFSET, + Method::CursorAfter, + Method::CursorBefore => QueryBase::METHOD_TYPE_CURSOR, + Method::OrderAsc, + Method::OrderDesc, + Method::OrderRandom => QueryBase::METHOD_TYPE_ORDER, + Method::Equal, + Method::NotEqual, + Method::LessThan, + Method::LessThanEqual, + Method::GreaterThan, + Method::GreaterThanEqual, + Method::Search, + Method::NotSearch, + Method::IsNull, + Method::IsNotNull, + Method::Between, + Method::NotBetween, + Method::StartsWith, + Method::NotStartsWith, + Method::EndsWith, + Method::NotEndsWith, + Method::Contains, + Method::ContainsAny, + Method::NotContains, + Method::And, + Method::Or, + Method::ContainsAll, + Method::ElemMatch, + Method::Crosses, + Method::NotCrosses, + Method::DistanceEqual, + Method::DistanceNotEqual, + Method::DistanceGreaterThan, + Method::DistanceLessThan, + Method::Intersects, + Method::NotIntersects, + Method::Overlaps, + Method::NotOverlaps, + Method::Touches, + Method::NotTouches, + Method::Covers, + Method::NotCovers, + Method::SpatialEquals, + Method::NotSpatialEquals, + Method::VectorDot, + Method::VectorCosine, + Method::VectorEuclidean, + Method::Regex, + Method::Exists, + Method::NotExists => QueryBase::METHOD_TYPE_FILTER, + Method::Distinct => QueryBase::METHOD_TYPE_DISTINCT, + Method::GroupBy => QueryBase::METHOD_TYPE_GROUP_BY, + Method::Having => QueryBase::METHOD_TYPE_HAVING, + Method::Join, + Method::LeftJoin, + Method::RightJoin, + Method::CrossJoin, + Method::FullOuterJoin, + Method::NaturalJoin => QueryBase::METHOD_TYPE_JOIN, + default => '', + }; + } - if ($method->isAggregate()) { - $methodType = QueryBase::METHOD_TYPE_AGGREGATE; - } else { - $methodType = match ($method) { - Method::Select => QueryBase::METHOD_TYPE_SELECT, - Method::Limit => QueryBase::METHOD_TYPE_LIMIT, - Method::Offset => QueryBase::METHOD_TYPE_OFFSET, - Method::CursorAfter, - Method::CursorBefore => QueryBase::METHOD_TYPE_CURSOR, - Method::OrderAsc, - Method::OrderDesc, - Method::OrderRandom => QueryBase::METHOD_TYPE_ORDER, - Method::Equal, - Method::NotEqual, - Method::LessThan, - Method::LessThanEqual, - Method::GreaterThan, - Method::GreaterThanEqual, - Method::Search, - Method::NotSearch, - Method::IsNull, - Method::IsNotNull, - Method::Between, - Method::NotBetween, - Method::StartsWith, - Method::NotStartsWith, - Method::EndsWith, - Method::NotEndsWith, - Method::Contains, - Method::ContainsAny, - Method::NotContains, - Method::And, - Method::Or, - Method::ContainsAll, - Method::ElemMatch, - Method::Crosses, - Method::NotCrosses, - Method::DistanceEqual, - Method::DistanceNotEqual, - Method::DistanceGreaterThan, - Method::DistanceLessThan, - Method::Intersects, - Method::NotIntersects, - Method::Overlaps, - Method::NotOverlaps, - Method::Touches, - Method::NotTouches, - Method::Covers, - Method::NotCovers, - Method::SpatialEquals, - Method::NotSpatialEquals, - Method::VectorDot, - Method::VectorCosine, - Method::VectorEuclidean, - Method::Regex, - Method::Exists, - Method::NotExists => QueryBase::METHOD_TYPE_FILTER, - Method::Distinct => QueryBase::METHOD_TYPE_DISTINCT, - Method::GroupBy => QueryBase::METHOD_TYPE_GROUP_BY, - Method::Having => QueryBase::METHOD_TYPE_HAVING, - Method::Join, - Method::LeftJoin, - Method::RightJoin, - Method::CrossJoin, - Method::FullOuterJoin, - Method::NaturalJoin => QueryBase::METHOD_TYPE_JOIN, - default => '', - }; + $methodIsValid = false; + foreach ($this->validators as $validator) { + if ($validator->getMethodType() !== $methodType) { + continue; } + if (! $validator->isValid($query)) { + $this->message = 'Invalid query: '.$validator->getDescription(); - $methodIsValid = false; - foreach ($this->validators as $validator) { - if ($validator->getMethodType() !== $methodType) { - continue; - } - if (! $validator->isValid($query)) { - $this->message = 'Invalid query: '.$validator->getDescription(); + return false; + } - return false; - } + $methodIsValid = true; + } - $methodIsValid = true; - } + if (! $methodIsValid) { + $this->message = (\in_array($method, self::LEGACY_METHODS, true) ? '' : 'Invalid query: ').'Invalid query method: '.$method->value; - if (! $methodIsValid) { - $this->message = 'Invalid query method: '.$method->value; + return false; + } - return false; + if ($hasFilterValidator && $method->isJoin() && $query->isNestedJoin()) { + foreach ($query->getJoinOnQueries() as $onQuery) { + if ($onQuery->getMethod() === Method::On) { + continue; + } + if (! $this->isValidQuery($onQuery, $hasFilterValidator)) { + return false; + } } } diff --git a/src/Database/Validator/Query/Select.php b/src/Database/Validator/Query/Select.php index 551ec1dc9..d9f75a338 100644 --- a/src/Database/Validator/Query/Select.php +++ b/src/Database/Validator/Query/Select.php @@ -161,7 +161,8 @@ public function isValid(mixed $value): bool $attribute = $alias; } - if (\in_array($attribute, $internalKeys)) { + // 7.x let `$tenant` through without shared tables too, and the read refused it with "Cannot select attributes". + if (\in_array($attribute, $internalKeys) || $attribute === Document::TENANT) { continue; } diff --git a/src/Database/Validator/Structure.php b/src/Database/Validator/Structure.php index c249ce8c6..b65375877 100644 --- a/src/Database/Validator/Structure.php +++ b/src/Database/Validator/Structure.php @@ -60,8 +60,9 @@ class Structure extends Validator /** * @param list $storedAttributes Attributes whose values are the stored ones, unchanged by the - * write: they are not validated again, as the rules may have - * tightened since those values were stored. + * write. They are validated again, as 7.x did, so a value a + * narrowed definition no longer admits fails the update, but an + * object value is held to what 7.x accepted when it was stored. */ public function __construct( protected readonly Document $collection, @@ -275,10 +276,6 @@ protected function checkForInvalidAttributeValues(Document $document, array $str continue; } - if (isset($this->storedAttributes[$key])) { - continue; - } - $attribute = $definitions[$key] ?? null; if ($attribute === null) { continue; @@ -364,7 +361,7 @@ protected function checkForInvalidAttributeValues(Document $document, array $str break; case ColumnType::Object: - $validators[] = new ObjectValue(); + $validators[] = new ObjectValue(stored: isset($this->storedAttributes[$key])); break; case ColumnType::Point: diff --git a/src/Database/Validator/UID.php b/src/Database/Validator/UID.php index 7ee9d6e8d..e55d55137 100644 --- a/src/Database/Validator/UID.php +++ b/src/Database/Validator/UID.php @@ -14,6 +14,6 @@ public function __construct(int $maxLength = Database::MAX_UID_DEFAULT_LENGTH) #[\Override] public function getDescription(): string { - return 'UID must contain at most '.$this->maxLength.' chars. Valid chars are a-z, A-Z, 0-9, period, hyphen, and underscore. Can\'t start with a leading period, hyphen, or underscore'; + return 'UID must contain at most '.$this->maxLength.' chars. Valid chars are a-z, A-Z, 0-9, and underscore. Can\'t start with a leading underscore'; } } diff --git a/tests/e2e/Adapter/Scopes/DocumentTests.php b/tests/e2e/Adapter/Scopes/DocumentTests.php index 1603bc0e9..4f930cadf 100644 --- a/tests/e2e/Adapter/Scopes/DocumentTests.php +++ b/tests/e2e/Adapter/Scopes/DocumentTests.php @@ -9,8 +9,13 @@ use Utopia\Cache\Adapter\None as NoneCacheAdapter; use Utopia\Cache\Cache; use Utopia\Database\Adapter\Feature; +use Utopia\Database\Adapter\MariaDB; +use Utopia\Database\Adapter\Memory; use Utopia\Database\Adapter\Mongo; +use Utopia\Database\Adapter\Postgres; +use Utopia\Database\Adapter\Redis; use Utopia\Database\Adapter\SQL; +use Utopia\Database\Adapter\SQLite; use Utopia\Database\Attribute; use Utopia\Database\AttributeUpdate; use Utopia\Database\Capability; @@ -1981,6 +1986,85 @@ public function testFindFulltextSeparatorsSplitWords(): void $this->assertNotContains('comet', $found); } + public function testContainsAllOnAStringMatchesAnyWholeValueAs7xDid(): void + { + /** @var Database $database */ + $database = $this->getDatabase(); + $adapter = $database->getAdapter(); + + if ($adapter instanceof SQLite || $adapter instanceof Memory || $adapter instanceof Redis) { + $this->expectNotToPerformAssertions(); + + return; + } + + $collection = 'contains_all_string'; + + $database->createCollection(Collection::create(id: $collection, permissions: [ + Permission::create(Role::any()), + Permission::read(Role::any()), + ])); + $database->createAttribute($collection, Attribute::string(key: 'name', size: 64, required: true)); + + foreach (['lower' => 'alpha', 'longer' => 'alphabet', 'other' => 'beta', 'unrelated' => 'gamma'] as $id => $name) { + $database->createDocument($collection, new Document([ + '$id' => $id, + '$permissions' => [Permission::read(Role::any())], + 'name' => $name, + ])); + } + + $ids = function (array $values) use ($database, $collection): array { + $ids = \array_map( + static fn (Document $document): string => $document->getId(), + $database->find($collection, [Query::containsAll('name', $values)]), + ); + \sort($ids); + + return $ids; + }; + + $this->assertSame([], $ids(['lph'])); + $this->assertSame(['lower'], $ids(['alpha'])); + $this->assertSame($adapter instanceof Mongo ? [] : ['lower', 'other'], $ids(['alpha', 'beta'])); + } + + public function testFindFulltextExactTermOnPostgresMatchesEveryWordInAnyOrder(): void + { + /** @var Database $database */ + $database = $this->getDatabase(); + + if (! $database->getAdapter() instanceof Postgres) { + $this->expectNotToPerformAssertions(); + + return; + } + + $collection = 'full_text_exact_words'; + $database->createCollection(Collection::create(id: $collection, permissions: [ + Permission::create(Role::any()), + Permission::read(Role::any()), + ])); + $database->createAttribute($collection, Attribute::string(key: 'text', size: 128, required: true)); + $database->createIndex($collection, Index::fulltext(key: 'text-ft', attributes: ['text'])); + + foreach (['phrase' => 'donald trump', 'reversed' => 'trump donald', 'apart' => 'donald j trump', 'one' => 'donald duck'] as $id => $text) { + $database->createDocument($collection, new Document([ + '$id' => $id, + '$permissions' => [Permission::read(Role::any())], + 'text' => $text, + ])); + } + + $this->assertSame(['apart', 'phrase', 'reversed'], $this->searchedIds($database, $collection, '"donald trump"')); + + $excluded = \array_map( + static fn (Document $document): string => $document->getId(), + $database->find($collection, [Query::notSearch('text', '"donald trump"')]), + ); + $this->assertSame(['one'], $excluded); + } + /** * @return list */ @@ -2542,6 +2626,126 @@ public function testFindOrderRandom(): void $this->assertLessThanOrEqual(25, count($documents)); // Default limit is 25 } + public function testConcurrentIncrementsOnMongoDBAllLand(): void + { + /** @var Database $database */ + $database = $this->getDatabase(); + $adapter = $database->getAdapter(); + + if (! $adapter instanceof Mongo || $database->hasSharedTables() || ! $adapter->supports(Capability::DefinedAttributes)) { + $this->expectNotToPerformAssertions(); + + return; + } + + $collection = 'concurrent_counters'; + $database->createCollection(Collection::create(id: $collection, permissions: [ + Permission::create(Role::any()), + Permission::read(Role::any()), + Permission::update(Role::any()), + ], documentSecurity: false)); + $database->createAttribute($collection, Attribute::integer(key: 'count')); + $database->createDocument($collection, new Document(['$id' => 'counter', 'count' => 5])); + + $failures = []; + try { + \Swoole\Coroutine\run(function () use ($database, $collection, &$failures): void { + for ($writer = 0; $writer < 8; $writer++) { + \Swoole\Coroutine::create(function () use ($database, $collection, &$failures): void { + $client = new \Utopia\Mongo\Client($database->getDatabase(), 'mongo', 27017, 'root', 'password', true); + $writerDatabase = (new Database(new Mongo($client), new Cache(new NoneCacheAdapter()))) + ->setAuthorization($database->getAuthorization()) + ->setDatabase($database->getDatabase()) + ->setNamespace($database->getNamespace()); + $writerDatabase->setSchemaless(false); + for ($increment = 0; $increment < 10; $increment++) { + try { + $writerDatabase->increaseDocumentAttribute($collection, 'counter', 'count', 1); + } catch (Throwable $error) { + $failures[] = $error->getMessage(); + } + } + }); + } + }); + + $this->assertSame([], $failures); + $this->assertSame(85, $database->getDocument($collection, 'counter')->getAttribute('count')); + } finally { + $database->deleteCollection($collection); + } + } + + public function testFloatsAreStoredAs7xBoundThem(): void + { + /** @var Database $database */ + $database = $this->getDatabase(); + $family = $this->engineFamily($database); + if ($family === null) { + $this->expectNotToPerformAssertions(); + + return; + } + + $collection = 'float_binding'; + $database->createCollection(Collection::create(id: $collection, permissions: [ + Permission::create(Role::any()), + Permission::read(Role::any()), + Permission::update(Role::any()), + ], documentSecurity: false)); + $database->createAttribute($collection, Attribute::double(key: 'f')); + $database->createAttribute($collection, Attribute::double(key: 'fa', array: true)); + + $values = [ + 'tiny' => [1e-300, 1e-300], + 'negative' => [-2.5e-20, -2.5e-20], + 'digits' => [1.2345678901234567e-10, $family === 'mongo' ? 1.2345678901234567e-10 : 1.2345678901235e-10], + 'avogadro' => [6.02214076e23, 6.02214076e23], + 'sum' => [0.1 + 0.2, $family === 'mongo' ? 0.1 + 0.2 : 0.3], + 'max' => [1.7976931348623157e308, $family === 'mongo' ? 1.7976931348623157e308 : 1.7976931348623e308], + ]; + foreach ($values as $id => [$written]) { + $database->createDocument($collection, new Document(['$id' => $id, 'f' => $written, 'fa' => [$written]])); + } + + foreach ($values as $id => [$written, $stored]) { + $document = $database->getDocument($collection, $id); + $this->assertSame($stored, $document->getAttribute('f'), $id); + $this->assertSame([$written], $document->getAttribute('fa'), $id); + } + + $database->updateDocument($collection, 'tiny', new Document(['f' => -1e-300])); + $this->assertSame(-1e-300, $database->getDocument($collection, 'tiny')->getAttribute('f')); + + $small = \array_map( + static fn (Document $document): string => $document->getId(), + $database->find($collection, [Query::lessThan('f', 1e-5), Query::greaterThan('f', -1), Query::orderAsc('$id')]), + ); + $this->assertSame(['digits', 'negative', 'tiny'], $small); + } + + public function testFindOrderRandomIgnoresACursorAs7xDid(): void + { + $this->initMoviesFixture(); + /** @var Database $database */ + $database = $this->getDatabase(); + + if (! $database->getAdapter()->supports(Capability::OrderRandom)) { + $this->expectNotToPerformAssertions(); + + return; + } + + $all = $database->find($this->getMoviesCollection(), [Query::limit(100)]); + $cursor = $all[0]; + + $after = $database->find($this->getMoviesCollection(), [Query::orderRandom(), Query::cursorAfter($cursor), Query::limit(100)]); + $before = $database->find($this->getMoviesCollection(), [Query::orderRandom(), Query::cursorBefore($cursor), Query::limit(100)]); + + $this->assertCount(\count($all), $after); + $this->assertCount(\count($all), $before); + } + public function testSum(): void { $this->initMoviesFixture(); @@ -7817,7 +8021,7 @@ public function testDuplicateExceptionMessages(): void $this->fail('Expected UniqueException for unique index violation'); } catch (DuplicateException $e) { $this->assertInstanceOf(UniqueException::class, $e); - $this->assertStringContainsString('unique attributes', $e->getMessage()); + $this->assertSame('Unique index violation', $e->getMessage()); } // '_uid' is the document ID column, so a conflicting value containing it must not be read as an ID conflict. @@ -7840,7 +8044,7 @@ public function testDuplicateExceptionMessages(): void $this->fail('Expected UniqueException for unique index violation on a value containing _uid'); } catch (DuplicateException $e) { $this->assertInstanceOf(UniqueException::class, $e); - $this->assertStringContainsString('unique attributes', $e->getMessage()); + $this->assertSame('Unique index violation', $e->getMessage()); } $database->deleteCollection('duplicateMessages'); @@ -10663,10 +10867,11 @@ private function statementsOn(Database $database, string $table, callable $read) return [$result, $statements]; } - public function testIncreaseAndDecreaseRefuseAFractionalBoundOnAnInteger(): void + public function testIncreaseAndDecreaseCompareAFractionalBoundOnAnIntegerAs7xDid(): void { /** @var Database $database */ $database = $this->getDatabase(); + $adapter = $database->getAdapter(); $collection = 'fractional_bound_'.uniqid(); $database->createCollection(Collection::create(id: $collection, permissions: [ @@ -10675,35 +10880,42 @@ public function testIncreaseAndDecreaseRefuseAFractionalBoundOnAnInteger(): void Permission::update(Role::any()), ], documentSecurity: false)); $database->createAttribute($collection, Attribute::integer(key: 'count', required: true)); - $database->createDocument($collection, new Document(['$id' => 'counter', 'count' => 100])); + foreach (['up', 'past', 'down', 'whole'] as $id) { + $database->createDocument($collection, new Document(['$id' => $id, 'count' => 5])); + } try { try { - $database->increaseDocumentAttribute($collection, 'counter', 'count', 1, 102.4); - $this->fail('A fractional maximum on an integer attribute was accepted'); - } catch (TypeException $error) { - $this->assertSame('Max must be an integer.', $error->getMessage()); + $database->increaseDocumentAttribute($collection, 'past', 'count', 2, 6.4); + $this->fail('An increase past a fractional maximum was accepted'); + } catch (LimitException $error) { + $this->assertSame('Attribute value exceeds maximum limit: 6.4', $error->getMessage()); } + $this->assertSame(5, $database->getDocument($collection, 'past')->getAttribute('count')); - try { - $database->decreaseDocumentAttribute($collection, 'counter', 'count', 1, 0.5); - $this->fail('A fractional minimum on an integer attribute was accepted'); - } catch (TypeException $error) { - $this->assertSame('Min must be an integer.', $error->getMessage()); + if ($adapter instanceof Postgres) { + $this->assertInstanceOf(\PDOException::class, $this->failureOf(fn () => $database->increaseDocumentAttribute($collection, 'up', 'count', 1, 6.4))); + $this->assertInstanceOf(\PDOException::class, $this->failureOf(fn () => $database->decreaseDocumentAttribute($collection, 'down', 'count', 1, 0.5))); + $this->assertSame(5, $database->getDocument($collection, 'up')->getAttribute('count')); + } else { + $this->assertSame(6, $database->increaseDocumentAttribute($collection, 'up', 'count', 1, 6.4)->getAttribute('count')); + $this->assertSame(4, $database->decreaseDocumentAttribute($collection, 'down', 'count', 1, 0.5)->getAttribute('count')); + $this->assertSame(6, $database->getDocument($collection, 'up')->getAttribute('count')); + $this->assertSame(4, $database->getDocument($collection, 'down')->getAttribute('count')); } - $this->assertSame(100, $database->getDocument($collection, 'counter')->getAttribute('count')); - $this->assertSame(101, $database->increaseDocumentAttribute($collection, 'counter', 'count', 1, 102.0)->getAttribute('count')); - $this->assertSame(100, $database->decreaseDocumentAttribute($collection, 'counter', 'count', 1, 99.0)->getAttribute('count')); + $this->assertSame(6, $database->increaseDocumentAttribute($collection, 'whole', 'count', 1, 6.0)->getAttribute('count')); + $this->assertSame(5, $database->decreaseDocumentAttribute($collection, 'whole', 'count', 1, 5.0)->getAttribute('count')); } finally { $database->deleteCollection($collection); } } - public function testIncreaseAndDecreaseRefuseAFractionalChangeValueOnAnInteger(): void + public function testIncreaseAndDecreaseApplyAFractionalChangeOnAnIntegerAs7xDid(): void { /** @var Database $database */ $database = $this->getDatabase(); + $adapter = $database->getAdapter(); $collection = 'fractional_change_'.uniqid(); $database->createCollection(Collection::create(id: $collection, permissions: [ @@ -10712,30 +10924,43 @@ public function testIncreaseAndDecreaseRefuseAFractionalChangeValueOnAnInteger() Permission::update(Role::any()), ], documentSecurity: false)); $database->createAttribute($collection, Attribute::integer(key: 'count', required: true)); - $database->createDocument($collection, new Document(['$id' => 'counter', 'count' => 100])); + foreach (['up', 'down', 'whole'] as $id) { + $database->createDocument($collection, new Document(['$id' => $id, 'count' => 5])); + } try { - try { - $database->increaseDocumentAttribute($collection, 'counter', 'count', 1.5); - $this->fail('A fractional increase of an integer attribute was accepted'); - } catch (TypeException $error) { - $this->assertSame('Change value must be an integer.', $error->getMessage()); - } + if ($adapter instanceof Postgres) { + $this->assertInstanceOf(\PDOException::class, $this->failureOf(fn () => $database->increaseDocumentAttribute($collection, 'up', 'count', 1.5))); + $this->assertInstanceOf(\PDOException::class, $this->failureOf(fn () => $database->decreaseDocumentAttribute($collection, 'down', 'count', 0.5))); + $this->assertSame(5, $database->getDocument($collection, 'up')->getAttribute('count')); + } else { + $this->assertSame(6.5, $database->increaseDocumentAttribute($collection, 'up', 'count', 1.5)->getAttribute('count')); + $this->assertSame(4.5, $database->decreaseDocumentAttribute($collection, 'down', 'count', 0.5)->getAttribute('count')); - try { - $database->decreaseDocumentAttribute($collection, 'counter', 'count', 0.5); - $this->fail('A fractional decrease of an integer attribute was accepted'); - } catch (TypeException $error) { - $this->assertSame('Change value must be an integer.', $error->getMessage()); + if ($adapter instanceof MariaDB || $adapter instanceof Mongo) { + $this->assertSame(6, $database->getDocument($collection, 'up')->getAttribute('count')); + $this->assertSame(4, $database->getDocument($collection, 'down')->getAttribute('count')); + } } - $this->assertSame(100, $database->getDocument($collection, 'counter')->getAttribute('count')); - $this->assertSame(102, $database->increaseDocumentAttribute($collection, 'counter', 'count', 2)->getAttribute('count')); + $this->assertSame(7.0, $database->increaseDocumentAttribute($collection, 'whole', 'count', 2.0)->getAttribute('count')); + $this->assertSame(7, $database->getDocument($collection, 'whole')->getAttribute('count')); } finally { $database->deleteCollection($collection); } } + private function failureOf(callable $call): ?Throwable + { + try { + $call(); + } catch (Throwable $error) { + return $error; + } + + return null; + } + public function testDistinctIsRefusedWhereTheAdapterCannotDeduplicate(): void { /** @var Database $database */ diff --git a/tests/e2e/Adapter/Scopes/OperatorTests.php b/tests/e2e/Adapter/Scopes/OperatorTests.php index 0bf19d41a..95b15a23a 100644 --- a/tests/e2e/Adapter/Scopes/OperatorTests.php +++ b/tests/e2e/Adapter/Scopes/OperatorTests.php @@ -5230,19 +5230,13 @@ public function testOperatorWholeFloatLimitHoldsAndAFractionalLimitIsRefused(): $this->assertSame(PHP_INT_MAX - 5, $updated->getAttribute('counter')); $this->assertSame(PHP_INT_MAX - 5, $database->getDocument($collectionId, 'doc')->getAttribute('counter')); - try { - $database->updateDocument($collectionId, 'doc', new Document([ - 'count' => Operator::increment(5, 102.4), - ])); - $this->fail('A fractional limit on an integer attribute must be refused'); - } catch (StructureException $exception) { - $this->assertSame( - "Invalid document structure: Cannot apply increment operator: max/min limit must be a whole number for integer attribute 'count', got 102.4", - $exception->getMessage(), - ); - } + $past = $database->updateDocument($collectionId, 'doc', new Document(['count' => Operator::increment(5, 102.4)])); + $this->assertSame($database->getDocument($collectionId, 'doc')->getAttribute('count'), $past->getAttribute('count')); - $this->assertSame(100, $database->getDocument($collectionId, 'doc')->getAttribute('count')); + if ($this->engineFamily($database) !== null) { + $this->assertSame(100, $past->getAttribute('count')); + $this->assertSame(102, $database->updateDocument($collectionId, 'doc', new Document(['count' => Operator::increment(2, 102.4)]))->getAttribute('count')); + } } finally { $database->deleteCollection($collectionId); } @@ -5293,4 +5287,68 @@ public function testPowerWithANumericTextExponent(): void $database->deleteCollection($collectionId); } } + + public function testFractionalNumericOperatorsOnAnIntegerReachTheEngineAs7xDid(): void + { + $database = static::getDatabase(); + $family = $this->engineFamily($database); + + if (! $database->getAdapter()->supports(Capability::Operators) || $family === null) { + $this->expectNotToPerformAssertions(); + + return; + } + + $collectionId = 'operator_fractional_changes'; + $database->createCollection(Collection::create(id: $collectionId, permissions: [Permission::create(Role::any()), Permission::read(Role::any()), Permission::update(Role::any())], documentSecurity: false)); + $database->createAttribute($collectionId, Attribute::integer(key: 'count')); + $database->createAttribute($collectionId, Attribute::integer(key: 'nums', array: true)); + + try { + $changes = [ + 'increment' => [Operator::increment(1.5), ['mariadb' => 6, 'mongo' => 6]], + 'multiply' => [Operator::multiply(1.5), ['mariadb' => 8, 'mongo' => 7]], + 'divide' => [Operator::divide(0.5), ['mariadb' => 10, 'mongo' => 10]], + ]; + foreach ($changes as $id => [$operator, $expected]) { + $database->createDocument($collectionId, new Document(['$id' => $id, 'count' => 5, 'nums' => [1]])); + + if ($family === 'postgres') { + $failure = null; + try { + $database->updateDocument($collectionId, $id, new Document(['count' => $operator])); + } catch (\Throwable $error) { + $failure = $error; + } + $this->assertInstanceOf(\PDOException::class, $failure, "PostgreSQL stored a fractional {$id} of an integer"); + $this->assertSame(5, $database->getDocument($collectionId, $id)->getAttribute('count'), $id); + + continue; + } + + $this->assertSame($expected[$family], $database->updateDocument($collectionId, $id, new Document(['count' => $operator]))->getAttribute('count'), $id); + $this->assertSame($expected[$family], $database->getDocument($collectionId, $id)->getAttribute('count'), $id); + } + + $database->createDocument($collectionId, new Document(['$id' => 'append', 'count' => 5, 'nums' => [1]])); + $this->assertSame([1, 1], $database->updateDocument($collectionId, 'append', new Document(['nums' => Operator::arrayAppend([1.5])]))->getAttribute('nums')); + } finally { + $database->deleteCollection($collectionId); + } + } + + /** + * The engine a 7.x comparison is pinned for: MariaDB and MySQL, PostgreSQL or MongoDB. + */ + private function engineFamily(Database $database): ?string + { + $adapter = $database->getAdapter(); + + return match (true) { + $adapter instanceof \Utopia\Database\Adapter\MariaDB => 'mariadb', + $adapter instanceof \Utopia\Database\Adapter\Postgres => 'postgres', + $adapter instanceof \Utopia\Database\Adapter\Mongo => 'mongo', + default => null, + }; + } } diff --git a/tests/e2e/Adapter/Scopes/RelationshipTests.php b/tests/e2e/Adapter/Scopes/RelationshipTests.php index da5ab9dcd..0b42fd8d0 100644 --- a/tests/e2e/Adapter/Scopes/RelationshipTests.php +++ b/tests/e2e/Adapter/Scopes/RelationshipTests.php @@ -506,6 +506,82 @@ public function testZoo(): void $this->assertEquals('Bronx Zoo', $animal->getDocument('zoo')->getAttribute('name')); // Check zoo is an object } + private function librarySchema(Database $database): void + { + $any = [Permission::create(Role::any()), Permission::read(Role::any()), Permission::update(Role::any()), Permission::delete(Role::any())]; + foreach (['lib_authors', 'lib_profiles', 'lib_books', 'lib_libraries', 'lib_tags'] as $collection) { + $database->createCollection(Collection::create(id: $collection, permissions: $any, documentSecurity: false)); + $database->createAttribute($collection, Attribute::string(key: 'name', size: 64, required: true)); + } + $database->createRelationship('lib_authors', Relationship::oneToOne('lib_profiles', key: 'profile', twoWay: true, twoWayKey: 'author', onDelete: RelationshipDeleteAction::Cascade)); + $database->createRelationship('lib_authors', Relationship::oneToMany('lib_books', key: 'books', twoWay: true, twoWayKey: 'author', onDelete: RelationshipDeleteAction::SetNull)); + $database->createRelationship('lib_books', Relationship::manyToOne('lib_libraries', key: 'library', twoWayKey: 'books_inverse')); + $database->createRelationship('lib_books', Relationship::manyToMany('lib_tags', key: 'tags', twoWay: true, twoWayKey: 'books', onDelete: RelationshipDeleteAction::SetNull)); + $database->createRelationship('lib_libraries', Relationship::oneToMany('lib_tags', key: 'featured', twoWayKey: 'featured_inverse', onDelete: RelationshipDeleteAction::Cascade)); + + foreach ([1, 2] as $number) { + $database->createDocument('lib_libraries', new Document(['$id' => "lib{$number}", 'name' => "Library {$number}", 'featured' => [['$id' => "feat{$number}", 'name' => "Featured {$number}"]]])); + } + foreach ([1, 2] as $number) { + $database->createDocument('lib_tags', new Document(['$id' => "tag{$number}", 'name' => "Tag {$number}"])); + } + $database->createDocument('lib_authors', new Document([ + '$id' => 'author1', + 'name' => 'Ada', + 'profile' => ['$id' => 'profile1', 'name' => 'Ada profile'], + 'books' => [['$id' => 'book1', 'name' => 'Notes', 'library' => 'lib1', 'tags' => ['tag1', 'tag2']]], + ])); + $database->createDocument('lib_books', new Document(['$id' => 'book2', 'name' => 'Orphan'])); + } + + public function testNestedRelationshipSelectsReturnWhat7xReturned(): void + { + /** @var Database $database */ + $database = static::getDatabase(); + + if (! $database->getAdapter()->hasFeature(Feature\Relationships::class)) { + $this->expectNotToPerformAssertions(); + + return; + } + + $this->librarySchema($database); + + try { + $books = $database->find('lib_books', [Query::select(['name', 'author.name', 'author.profile.*']), Query::equal('$id', ['book1'])]); + $author = $books[0]->getAttribute('author'); + $this->assertInstanceOf(Document::class, $author); + $this->assertSame('Ada', $author->getAttribute('name')); + $profile = $author->getAttribute('profile'); + $this->assertInstanceOf(Document::class, $profile); + $this->assertSame('Ada profile', $profile->getAttribute('name')); + + $books = $database->find('lib_books', [Query::select(['tags.books.name']), Query::equal('$id', ['book1'])]); + $tags = $books[0]->getAttribute('tags'); + $this->assertIsArray($tags); + $this->assertSame(['Tag 1', 'Tag 2'], \array_map(static fn (mixed $tag): mixed => $tag instanceof Document ? $tag->getAttribute('name') : null, $tags)); + + $library = $database->getDocument('lib_books', 'book1', [Query::select(['name', 'library.featured.*'])])->getAttribute('library'); + $this->assertInstanceOf(Document::class, $library); + $this->assertSame('Library 1', $library->getAttribute('name')); + $featured = $library->getAttribute('featured'); + $this->assertIsArray($featured); + $this->assertInstanceOf(Document::class, $featured[0]); + $this->assertSame('Featured 1', $featured[0]->getAttribute('name')); + + $books = $database->find('lib_books', [Query::select(['*', '*.*']), Query::orderAsc('$id')]); + $this->assertSame(['book1', 'book2'], \array_map(static fn (Document $book): string => $book->getId(), $books)); + $this->assertSame('Notes', $books[0]->getAttribute('name')); + + $keys = \array_keys($database->getDocument('lib_authors', 'author1')->getArrayCopy()); + $this->assertLessThan(\array_search('$id', $keys, true), \array_search('$sequence', $keys, true), 'a single read returns $sequence ahead of $id, as in 7.x'); + } finally { + foreach (['lib_books', 'lib_authors', 'lib_profiles', 'lib_libraries', 'lib_tags'] as $collection) { + $database->deleteCollection($collection); + } + } + } + public function testSimpleRelationshipPopulation(): void { /** @var Database $database */ diff --git a/tests/e2e/Adapter/Scopes/Relationships/ManyToManyTests.php b/tests/e2e/Adapter/Scopes/Relationships/ManyToManyTests.php index af969aa0c..904672d78 100644 --- a/tests/e2e/Adapter/Scopes/Relationships/ManyToManyTests.php +++ b/tests/e2e/Adapter/Scopes/Relationships/ManyToManyTests.php @@ -25,6 +25,49 @@ trait ManyToManyTests { + public function testRenamingAManyToManyRelationshipCreatedBy7x(): void + { + /** @var Database $database */ + $database = $this->getDatabase(); + + if (! ($database->getAdapter()->hasFeature(Feature\Relationships::class))) { + $this->expectNotToPerformAssertions(); + + return; + } + + $database->createCollection(Collection::create(id: 'legacy_books', permissions: [Permission::create(Role::any()), Permission::read(Role::any()), Permission::update(Role::any())], documentSecurity: false)); + $database->createCollection(Collection::create(id: 'legacy_tags', permissions: [Permission::create(Role::any()), Permission::read(Role::any()), Permission::update(Role::any())], documentSecurity: false)); + $database->createAttribute('legacy_books', Attribute::string(key: 'name', required: true)); + $database->createAttribute('legacy_tags', Attribute::string(key: 'name', required: true)); + $database->createRelationship('legacy_books', Relationship::manyToMany(relatedCollection: 'legacy_tags', key: 'tags', twoWay: true, twoWayKey: 'books')); + $database->createDocument('legacy_books', new Document(['$id' => 'book1', 'name' => 'Notes', 'tags' => [['$id' => 'tag1', 'name' => 'Tag 1']]])); + + $junction = '_'.$database->getCollection('legacy_books')->getSequence().'_'.$database->getCollection('legacy_tags')->getSequence(); + $database->getAuthorization()->skip(fn () => $database->silent(function () use ($database, $junction): void { + $definition = $database->getDocument(Database::METADATA, $junction); + $stored = $definition->getAttribute('indexes', []); + $indexes = \array_map( + static fn (mixed $index): mixed => $index instanceof Document && $index->getId() === '_index_tags' ? $index->setAttribute('key', 'index_tags') : $index, + \is_array($stored) ? $stored : [], + ); + $database->updateDocument(Database::METADATA, $junction, $definition->setAttribute('indexes', $indexes)); + })); + $database->purgeCachedCollection($junction); + + $database->updateRelationship('legacy_books', 'tags', new RelationshipUpdate(key: 'labels')); + + $keys = \array_map(static fn ($index): string => $index->key, $database->getCollection($junction)->indexes()); + $this->assertContains('_index_labels', $keys); + $this->assertNotContains('_index_tags', $keys); + $labels = $database->getDocument('legacy_books', 'book1')->getAttribute('labels'); + $this->assertIsArray($labels); + $this->assertSame(['tag1'], \array_map(static fn (mixed $tag): mixed => $tag instanceof Document ? $tag->getId() : null, $labels)); + + $database->deleteCollection('legacy_books'); + $database->deleteCollection('legacy_tags'); + } + public function testManyToManyOneWayRelationship(): void { /** @var Database $database */ diff --git a/tests/e2e/Adapter/Scopes/Relationships/OneToManyTests.php b/tests/e2e/Adapter/Scopes/Relationships/OneToManyTests.php index bd7d161ca..a441602c8 100644 --- a/tests/e2e/Adapter/Scopes/Relationships/OneToManyTests.php +++ b/tests/e2e/Adapter/Scopes/Relationships/OneToManyTests.php @@ -24,6 +24,47 @@ trait OneToManyTests { + public function testLinkingAChildThroughItsParentKeepsItsUpdatedAtUnderPreservedDates(): void + { + /** @var Database $database */ + $database = $this->getDatabase(); + + if (! ($database->getAdapter()->hasFeature(Feature\Relationships::class))) { + $this->expectNotToPerformAssertions(); + + return; + } + + $stored = '2020-01-01T00:00:00.000+00:00'; + $any = [Permission::create(Role::any()), Permission::read(Role::any()), Permission::update(Role::any())]; + $database->createCollection(Collection::create(id: 'dated_authors', permissions: $any, documentSecurity: false)); + $database->createCollection(Collection::create(id: 'dated_books', permissions: $any, documentSecurity: false)); + $database->createAttribute('dated_authors', Attribute::string(key: 'name', size: 64)); + $database->createAttribute('dated_books', Attribute::string(key: 'name', size: 64)); + $database->createRelationship('dated_authors', Relationship::oneToMany('dated_books', key: 'books', twoWay: true, twoWayKey: 'author', onDelete: RelationshipDeleteAction::SetNull)); + + try { + $database->withPreserveDates(true, function () use ($database, $stored): void { + $database->createDocument('dated_authors', new Document(['$id' => 'author3', 'name' => 'Linus', '$createdAt' => $stored, '$updatedAt' => $stored])); + foreach (['book4', 'book5'] as $id) { + $database->createDocument('dated_books', new Document(['$id' => $id, 'name' => $id, '$createdAt' => $stored, '$updatedAt' => $stored])); + } + $database->updateDocument('dated_authors', 'author3', new Document(['books' => ['book4', 'book5']])); + }); + + foreach (['book4', 'book5'] as $id) { + $book = $database->getDocument('dated_books', $id); + $author = $book->getAttribute('author'); + $this->assertInstanceOf(Document::class, $author, $id); + $this->assertSame('author3', $author->getId(), $id); + $this->assertSame($stored, $book->getUpdatedAt(), $id); + } + } finally { + $database->deleteCollection('dated_authors'); + $database->deleteCollection('dated_books'); + } + } + public function testOneToManyOneWayRelationship(): void { /** @var Database $database */ diff --git a/tests/unit/Adapter/BuildsAggregates.php b/tests/unit/Adapter/BuildsAggregates.php index de00a43a3..8b6337068 100644 --- a/tests/unit/Adapter/BuildsAggregates.php +++ b/tests/unit/Adapter/BuildsAggregates.php @@ -48,6 +48,6 @@ public function builtAggregate(string $operation, Document $collection, array $q */ public function boundValues(array $bindings): array { - return \array_map(fn (mixed $value): mixed => \is_float($value) ? $this->getFloatPrecision($value) : $value, $bindings); + return $bindings; } } diff --git a/tests/unit/Adapter/FloatBindingTest.php b/tests/unit/Adapter/FloatBindingTest.php new file mode 100644 index 000000000..e1c86b55b --- /dev/null +++ b/tests/unit/Adapter/FloatBindingTest.php @@ -0,0 +1,72 @@ + + */ + private array $bindings = []; + + public function testAWriteBindsAFloatAsPhpWritesIt(): void + { + $adapter = $this->adapter(); + $collection = Collection::create(id: 'numbers', attributes: [Attribute::double(key: 'f')]); + + $adapter->createDocument($collection, new Document(['$id' => 'tiny', '$permissions' => [], 'f' => 1e-300])); + + $this->assertContains(1e-300, $this->bindings); + $this->assertNotContains('0.00000000000000000', $this->bindings); + } + + public function testAFindBindsAFloatInFixedPoint(): void + { + $adapter = $this->adapter(); + $collection = Collection::create(id: 'numbers', attributes: [Attribute::double(key: 'f')]); + + $adapter->find($collection, [Query::lessThan('f', 0.5)]); + + $this->assertContains('0.50000000000000000', $this->bindings); + } + + private function adapter(): MariaDB + { + $pdo = $this->createStub(PDO::class); + $pdo->method('prepare')->willReturnCallback(function (): PDOStatement { + $statement = $this->createStub(PDOStatement::class); + $statement->method('execute')->willReturn(true); + $statement->method('bindValue')->willReturnCallback(function (int|string $parameter, mixed $value): bool { + $this->bindings[] = $value; + + return true; + }); + $statement->method('fetchAll')->willReturn([]); + $statement->method('fetch')->willReturn(false); + $statement->method('rowCount')->willReturn(1); + $statement->method('closeCursor')->willReturn(true); + + return $statement; + }); + $pdo->method('lastInsertId')->willReturn('1'); + + $adapter = new MariaDB($pdo); + $adapter->setDatabase('database'); + $adapter->setNamespace('namespace'); + $authorization = new Authorization(); + $authorization->disable(); + $adapter->setAuthorization($authorization); + + return $adapter; + } +} diff --git a/tests/unit/Adapter/MemoryAdapterTest.php b/tests/unit/Adapter/MemoryAdapterTest.php index bb5650378..bd98fc489 100644 --- a/tests/unit/Adapter/MemoryAdapterTest.php +++ b/tests/unit/Adapter/MemoryAdapterTest.php @@ -114,7 +114,7 @@ public function testRenamingKeepsItsUniqueValue(): void $database->createDocument('users', new Document(['$id' => 'copy', 'email' => 'a@example.test'])); $this->fail('The renamed document must still hold its unique value'); } catch (UniqueException $exception) { - $this->assertSame('Document with the requested unique attributes already exists', $exception->getMessage()); + $this->assertSame('Unique index violation', $exception->getMessage()); } $database->createDocument('users', new Document(['$id' => 'reuse', 'email' => 'c@example.test'])); diff --git a/tests/unit/Adapter/MemoryWritePathsTest.php b/tests/unit/Adapter/MemoryWritePathsTest.php index 503c7f4ee..7a176b2f4 100644 --- a/tests/unit/Adapter/MemoryWritePathsTest.php +++ b/tests/unit/Adapter/MemoryWritePathsTest.php @@ -246,7 +246,7 @@ private function assertDuplicate(\Closure $write, string $message): void try { $write(); } catch (DuplicateException $exception) { - $this->assertSame('Document with the requested unique attributes already exists', $exception->getMessage()); + $this->assertSame('Unique index violation', $exception->getMessage()); return; } diff --git a/tests/unit/Adapter/ProfileTest.php b/tests/unit/Adapter/ProfileTest.php index 390061283..9d7d6a3f4 100644 --- a/tests/unit/Adapter/ProfileTest.php +++ b/tests/unit/Adapter/ProfileTest.php @@ -83,12 +83,12 @@ public function testSharedTablesShrinkTheIndexKeyOnlyAfterTheyAreTurnedOn(): voi $this->assertTrue(new IndexDefinition($attributes, [], $database->profile())->isValid($index)); } - public function testTheTenantIsSelectableOnlyAfterSharedTablesAreTurnedOn(): void + public function testTheTenantPassesSelectValidationWithAndWithoutSharedTables(): void { $database = $this->database(new Memory()); $select = [Query::select(['$tenant'])]; - $this->assertFalse(new Documents([], [], $database->profile())->isValid($select)); + $this->assertTrue(new Documents([], [], $database->profile())->isValid($select)); $database->setSharedTables(true); diff --git a/tests/unit/Adapter/RedisAdapterPathsTest.php b/tests/unit/Adapter/RedisAdapterPathsTest.php index 78452fe7a..72344bbb3 100644 --- a/tests/unit/Adapter/RedisAdapterPathsTest.php +++ b/tests/unit/Adapter/RedisAdapterPathsTest.php @@ -394,7 +394,7 @@ public function testUniqueIndexComparesArrayValuesByContent(): void $adapter->createDocument($this->notes(), new Document(['$id' => 'second', '$permissions' => [], 'tags' => ['a', 'b']])); } - public function testFractionalOperatorLimitIsRefusedBeforeTheWrite(): void + public function testFractionalOperatorLimitOnABigIntegerIsRefusedBeforeTheWrite(): void { $database = $this->database(); $database->create(); @@ -406,12 +406,12 @@ public function testFractionalOperatorLimitIsRefusedBeforeTheWrite(): void $database->createDocument(self::NOTES, new Document(['$id' => 'counter', 'count' => 100, 'big' => PHP_INT_MAX - 5])); try { - $database->updateDocument(self::NOTES, 'counter', new Document(['count' => Operator::increment(5, 102.4)])); - $this->fail('A fractional limit on an integer attribute must be refused'); + $database->updateDocument(self::NOTES, 'counter', new Document(['big' => Operator::increment(5, 4.0e15 + 0.5)])); + $this->fail('A fractional limit on a bigint attribute must be refused'); } catch (StructureException $exception) { - $this->assertSame("Invalid document structure: Cannot apply increment operator: max/min limit must be a whole number for integer attribute 'count', got 102.4", $exception->getMessage()); + $this->assertStringStartsWith("Invalid document structure: Cannot apply increment operator: max/min limit must be a whole number for integer attribute 'big'", $exception->getMessage()); } - $this->assertSame(100, $database->getDocument(self::NOTES, 'counter')->getAttribute('count')); + $this->assertSame(PHP_INT_MAX - 5, $database->getDocument(self::NOTES, 'counter')->getAttribute('big')); $database->updateDocument(self::NOTES, 'counter', new Document(['big' => Operator::increment(10, 9.0e18)])); $this->assertSame(PHP_INT_MAX - 5, $database->getDocument(self::NOTES, 'counter')->getAttribute('big')); diff --git a/tests/unit/Adapter/RedisUniqueIndexTest.php b/tests/unit/Adapter/RedisUniqueIndexTest.php index 77fc4f1be..a9b72b76a 100644 --- a/tests/unit/Adapter/RedisUniqueIndexTest.php +++ b/tests/unit/Adapter/RedisUniqueIndexTest.php @@ -77,7 +77,7 @@ public function testUpdateDocumentsRejectsADuplicateUniqueValue(): void $database->updateDocuments(self::USERS, new Document(['email' => 'first@example.test']), [Query::equal('$id', ['second'])]); $this->fail('A batch update onto another document\'s unique value must be rejected'); } catch (UniqueException $exception) { - $this->assertSame('Document with the requested unique attributes already exists', $exception->getMessage()); + $this->assertSame('Unique index violation', $exception->getMessage()); } $this->assertSame(['first@example.test', 'second@example.test', 'third@example.test'], $this->emails($database)); @@ -94,7 +94,7 @@ public function testUpsertRejectsADuplicateUniqueValue(): void $database->upsertDocuments(self::USERS, [new Document(['$id' => 'second', 'email' => 'first@example.test'])]); $this->fail('An upsert that updates onto another document\'s unique value must be rejected'); } catch (UniqueException $exception) { - $this->assertSame('Document with the requested unique attributes already exists', $exception->getMessage()); + $this->assertSame('Unique index violation', $exception->getMessage()); } $this->assertSame(['first@example.test', 'second@example.test', 'third@example.test'], $this->emails($database)); @@ -110,7 +110,7 @@ public function testABatchCannotCollideWithItself(): void $database->updateDocuments(self::USERS, new Document(['email' => 'shared@example.test']), [Query::equal('$id', ['second', 'third'])]); $this->fail('A batch update that gives two documents one unique value must be rejected'); } catch (UniqueException $exception) { - $this->assertSame('Document with the requested unique attributes already exists', $exception->getMessage()); + $this->assertSame('Unique index violation', $exception->getMessage()); } try { @@ -120,7 +120,7 @@ public function testABatchCannotCollideWithItself(): void ]); $this->fail('An upsert batch that gives two documents one unique value must be rejected'); } catch (UniqueException $exception) { - $this->assertSame('Document with the requested unique attributes already exists', $exception->getMessage()); + $this->assertSame('Unique index violation', $exception->getMessage()); } $this->assertSame(['first@example.test', 'second@example.test', 'third@example.test'], $this->emails($database)); @@ -156,7 +156,7 @@ public function testAnUpsertBatchChecksEachDocumentAgainstTheOnesBeforeIt(): voi ]); $this->fail('A new document that takes a unique value before the document holding it gives it up must be rejected'); } catch (UniqueException $exception) { - $this->assertSame('Document with the requested unique attributes already exists', $exception->getMessage()); + $this->assertSame('Unique index violation', $exception->getMessage()); } $this->assertSame(['first@example.test', 'second@example.test', 'third@example.test'], $this->emails($database)); @@ -182,7 +182,7 @@ public function testTenantPerDocumentChecksTheDocumentsTenant(): void $database->createDocument(self::USERS, $this->user('second', 'taken@example.test')->setAttribute('$tenant', self::TENANT)); $this->fail('A duplicate under the document\'s own tenant must be rejected while another tenant is selected'); } catch (UniqueException $exception) { - $this->assertSame('Document with the requested unique attributes already exists', $exception->getMessage()); + $this->assertSame('Unique index violation', $exception->getMessage()); } $database->createDocument(self::USERS, $this->user('second', 'taken@example.test')->setAttribute('$tenant', self::OTHER_TENANT)); diff --git a/tests/unit/Builder/ContainsAllTest.php b/tests/unit/Builder/ContainsAllTest.php new file mode 100644 index 000000000..ffdc12494 --- /dev/null +++ b/tests/unit/Builder/ContainsAllTest.php @@ -0,0 +1,23 @@ +compileFilters([Query::containsAll('title', ['alpha', 'be%ta'])]); + + $this->assertStringContainsString(" {$like} ? OR ", $condition->expression, $builder::class); + $this->assertStringNotContainsString(' AND ', $condition->expression, $builder::class); + $this->assertSame(['alpha', 'be%ta'], $condition->bindings, $builder::class); + } + } +} diff --git a/tests/unit/Builder/SearchTermTest.php b/tests/unit/Builder/SearchTermTest.php index 3c6ab33f5..7b39973d6 100644 --- a/tests/unit/Builder/SearchTermTest.php +++ b/tests/unit/Builder/SearchTermTest.php @@ -18,23 +18,23 @@ final class SearchTermTest extends TestCase public static function postgreSQLTerms(): array { return [ - 'slash separates words' => ['foo/bar', 'foo or bar'], - 'comma separates words' => ['foo,bar', 'foo or bar'], - 'semicolon separates words' => ['foo;bar', 'foo or bar'], - 'percent separates words' => ['foo%bar', 'foo or bar'], - 'equals separates words' => ['foo=bar', 'foo or bar'], - 'question mark separates words' => ['foo?bar', 'foo or bar'], - 'hash separates words' => ['foo#bar', 'foo or bar'], - 'colon separates words' => ['foo:bar', 'foo or bar'], - 'pipe separates words' => ['foo|bar', 'foo or bar'], - 'ampersand separates words' => ['foo&bar', 'foo or bar'], - 'exclamation mark separates words' => ['foo!bar', 'foo or bar'], - 'underscore stays inside a word' => ['foo_bar', 'foo_bar'], - 'mixed separators and spaces' => ['baz, foo/bar; qux', 'baz or foo or bar or qux'], - 'operators are dropped' => ['+foo -bar* @3 ~qux (quux)', 'foo or bar or 3 or baz or qux or quux'], - 'accented words are kept' => ['@García!', 'García'], - 'exact phrase keeps its quotes' => ['"foo/bar baz"', '"foo bar baz"'], - 'unbalanced quote is not exact' => ['"foo/bar', 'foo or bar'], + 'slash separates words' => ['foo/bar', "'foo or bar'"], + 'comma separates words' => ['foo,bar', "'foo or bar'"], + 'semicolon separates words' => ['foo;bar', "'foo or bar'"], + 'percent separates words' => ['foo%bar', "'foo or bar'"], + 'equals separates words' => ['foo=bar', "'foo or bar'"], + 'question mark separates words' => ['foo?bar', "'foo or bar'"], + 'hash separates words' => ['foo#bar', "'foo or bar'"], + 'colon separates words' => ['foo:bar', "'foo or bar'"], + 'pipe separates words' => ['foo|bar', "'foo or bar'"], + 'ampersand separates words' => ['foo&bar', "'foo or bar'"], + 'exclamation mark separates words' => ['foo!bar', "'foo or bar'"], + 'underscore stays inside a word' => ['foo_bar', "'foo_bar'"], + 'mixed separators and spaces' => ['baz, foo/bar; qux', "'baz or foo or bar or qux'"], + 'operators are dropped' => ['+foo -bar* @3 ~qux (quux)', "'foo or bar or 3 or baz or qux or quux'"], + 'accented words are kept' => ['@García!', "'García'"], + 'exact term matches every word in single quotes' => ['"foo/bar baz"', "'foo bar baz'"], + 'unbalanced quote is not exact' => ['"foo/bar', "'foo or bar'"], ]; } @@ -77,6 +77,16 @@ public function testPostgreSQLSearchesEachWordOfATerm(string $term, string $boun $this->assertSame([$bound], $this->bindings(new Postgres(), Query::notSearch('title', $term))); } + public function testPostgreSQLMatchesTheTermWithWebsearchToTsquery(): void + { + $search = (new Postgres())->compileFilters([Query::search('title', '"foo bar"')]); + $notSearch = (new Postgres())->compileFilters([Query::notSearch('title', '"foo bar"')]); + + $this->assertStringContainsString("@@ websearch_to_tsquery(?)", $search->expression); + $this->assertStringStartsWith('NOT (', $notSearch->expression); + $this->assertSame(["'foo bar'"], $search->bindings); + } + #[DataProvider('mySQLTerms')] public function testMySQLSearchesEachWordOfATerm(string $term, string $bound): void { diff --git a/tests/unit/CoreMinorsTest.php b/tests/unit/CoreMinorsTest.php index ba0d3b4f1..3903d140b 100644 --- a/tests/unit/CoreMinorsTest.php +++ b/tests/unit/CoreMinorsTest.php @@ -478,7 +478,7 @@ public function testTypeMismatchMessagesSayBigint(): void $error = $this->attempt(fn (): bool => $validator->isValid(Attribute::bigInteger(key: 'total', default: 'many'))); $this->assertInstanceOf(DatabaseException::class, $error); - $this->assertSame('Default value "many" does not match given type bigint', $error->getMessage()); + $this->assertSame('Default value many is not a valid integer string for type bigint', $error->getMessage()); $database = $this->interceptingMetadataWrites(static function (): void { }); @@ -488,7 +488,7 @@ public function testTypeMismatchMessagesSayBigint(): void $error = $this->attempt(fn (): Attribute => $database->updateAttribute('logs', 'total', new AttributeUpdate(default: 'many'))); $this->assertInstanceOf(DatabaseException::class, $error); - $this->assertSame('Default value many does not match given type bigint', $error->getMessage()); + $this->assertSame('Default value many is not a valid integer string for type bigint', $error->getMessage()); $error = $this->attempt(fn (): bool => $validator->isValid(Attribute::fromArray(['key' => 'value', 'type' => ColumnType::Timestamp]))); @@ -545,7 +545,7 @@ public function testEveryUniqueViolationUsesOneMessage(Closure $adapter): void $error = $this->attempt($violation); $this->assertInstanceOf(UniqueException::class, $error, $name); - $this->assertSame('Document with the requested unique attributes already exists', $error->getMessage(), $name); + $this->assertSame('Unique index violation', $error->getMessage(), $name); } } diff --git a/tests/unit/Documents/AggregateSelectTest.php b/tests/unit/Documents/AggregateSelectTest.php index 7c3dc0272..a7a4244aa 100644 --- a/tests/unit/Documents/AggregateSelectTest.php +++ b/tests/unit/Documents/AggregateSelectTest.php @@ -306,11 +306,11 @@ public function testValidatorsWithoutAggregatesRejectTheAggregateItself(): void $documents = new DocumentsValidator($attributes, [], Profiles::of(capabilities: [Capability::DefinedAttributes, Capability::UnsignedBigInt, Capability::OrderRandom], idType: ColumnType::String)); $this->assertFalse($documents->isValid($queries)); - $this->assertSame('Invalid query method: count', $documents->getDescription()); + $this->assertSame('Invalid query: Invalid query method: count', $documents->getDescription()); $document = new DocumentValidator($attributes, Profiles::of(capabilities: [Capability::DefinedAttributes, Capability::UnsignedBigInt, Capability::Joins])); $this->assertFalse($document->isValid($queries)); - $this->assertSame('Invalid query method: count', $document->getDescription()); + $this->assertSame('Invalid query: Invalid query method: count', $document->getDescription()); } public function testSelectValidatorAcceptsOnlyGroupsAndWildcardsOfAnAggregationQuery(): void diff --git a/tests/unit/Documents/DocumentsValidatorGrammarTest.php b/tests/unit/Documents/DocumentsValidatorGrammarTest.php index adcede2a9..7aed2af70 100644 --- a/tests/unit/Documents/DocumentsValidatorGrammarTest.php +++ b/tests/unit/Documents/DocumentsValidatorGrammarTest.php @@ -42,10 +42,10 @@ public function testAdaptersWithoutJoinsOrAggregationsKeepTheFilterGrammar(): vo $validator = (new DocumentsValidatorDatabase(new Memory(), new Cache(new None())))->documentsValidator($this->orders); $this->assertFalse($validator->isValid([Query::join('customers', 'j0', [Query::on('$id', 'customerId')])])); - $this->assertSame('Invalid query method: join', $validator->getDescription()); + $this->assertSame('Invalid query: Invalid query method: join', $validator->getDescription()); $this->assertFalse($validator->isValid([Query::sum('amount', 'total')])); - $this->assertSame('Invalid query method: sum', $validator->getDescription()); + $this->assertSame('Invalid query: Invalid query method: sum', $validator->getDescription()); } public function testAdaptersWithJoinsAndAggregationsAcceptThem(): void @@ -56,12 +56,12 @@ public function testAdaptersWithJoinsAndAggregationsAcceptThem(): void $this->assertTrue($validator->isValid([Query::sum('amount', 'total')]), $validator->getDescription()); } - public function testTheCachedValidatorFollowsTheProfile(): void + public function testTheCachedValidatorLetsTheTenantThroughWithAndWithoutSharedTables(): void { $database = new DocumentsValidatorDatabase(new Memory(), new Cache(new None())); $queries = [Query::select(['$tenant'])]; - $this->assertFalse($database->documentsValidator($this->orders)->isValid($queries)); + $this->assertTrue($database->documentsValidator($this->orders)->isValid($queries)); $database->setSharedTables(true); diff --git a/tests/unit/Documents/FractionalBoundTest.php b/tests/unit/Documents/FractionalBoundTest.php index 7b454a485..1e14d929d 100644 --- a/tests/unit/Documents/FractionalBoundTest.php +++ b/tests/unit/Documents/FractionalBoundTest.php @@ -13,7 +13,6 @@ use Utopia\Database\Database; use Utopia\Database\Document; use Utopia\Database\Exception\Limit as LimitException; -use Utopia\Database\Exception\Type as TypeException; use Utopia\Database\Operator; use Utopia\Database\Permission; use Utopia\Database\Role; @@ -35,44 +34,38 @@ public static function lanes(): iterable } #[DataProvider('lanes')] - public function testIncreaseWithAFractionalMaximumOnAnIntegerIsRefused(bool $definedAttributes): void + public function testAFractionalMaximumOnAnIntegerIsComparedAs7xDid(bool $definedAttributes): void { $database = $this->database($definedAttributes); + $this->assertSame(101, $database->increaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', 1, 102.4)->getAttribute('count')); + $this->assertSame(102, $database->increaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', 1, '102.5')->getAttribute('count')); + try { - $database->increaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', 1, 102.4); - $this->fail('A fractional maximum on an integer attribute was accepted'); - } catch (TypeException $error) { - $this->assertSame('Max must be an integer.', $error->getMessage()); + $database->increaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', 1, 102.5); + $this->fail('An increase past a fractional maximum was accepted'); + } catch (LimitException $error) { + $this->assertSame('Attribute value exceeds maximum limit: 102.5', $error->getMessage()); } - $this->assertSame(100, $this->stored($database, 'count')); + $this->assertSame(102, $this->stored($database, 'count')); } #[DataProvider('lanes')] - public function testDecreaseWithAFractionalMinimumOnAnIntegerIsRefused(bool $definedAttributes): void + public function testAFractionalMinimumOnAnIntegerIsComparedAs7xDid(bool $definedAttributes): void { $database = $this->database($definedAttributes); + $this->assertSame(99, $database->decreaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', 1, 0.5)->getAttribute('count')); + try { - $database->decreaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', 1, 0.5); - $this->fail('A fractional minimum on an integer attribute was accepted'); - } catch (TypeException $error) { - $this->assertSame('Min must be an integer.', $error->getMessage()); + $database->decreaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', 99, 0.5); + $this->fail('A decrease past a fractional minimum was accepted'); + } catch (LimitException $error) { + $this->assertSame('Attribute value exceeds minimum limit: 0.5', $error->getMessage()); } - $this->assertSame(100, $this->stored($database, 'count')); - } - - #[DataProvider('lanes')] - public function testANonNumericBoundOnAnIntegerIsRefused(bool $definedAttributes): void - { - $database = $this->database($definedAttributes); - - $this->expectException(TypeException::class); - $this->expectExceptionMessage('Max must be an integer.'); - - $database->increaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', 1, '102.5'); + $this->assertSame(99, $this->stored($database, 'count')); } #[DataProvider('lanes')] @@ -118,24 +111,14 @@ public function testWholeNumberStringBoundsOnAnIntegerAreAcceptedAsOperatorLimit } #[DataProvider('lanes')] - public function testAFractionalChangeValueOnAnIntegerIsRefused(bool $definedAttributes): void + public function testAFractionalChangeOnAnIntegerReturnsTheSumAs7xDid(bool $definedAttributes): void { - $database = $this->database($definedAttributes); + $this->assertSame(101.5, $this->database($definedAttributes)->increaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', 1.5)->getAttribute('count')); + $this->assertSame(99.5, $this->database($definedAttributes)->decreaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', 0.5)->getAttribute('count')); + $this->assertSame(101.5, $this->database($definedAttributes)->increaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', '1.5')->getAttribute('count')); + $this->assertSame(102.0, $this->database($definedAttributes)->increaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', 2.0)->getAttribute('count')); - foreach ([ - 'increase' => static fn (): Document => $database->increaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', 1.5), - 'decrease' => static fn (): Document => $database->decreaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', 0.5), - 'increase by a numeric string' => static fn (): Document => $database->increaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', '1.5'), - ] as $case => $change) { - try { - $change(); - $this->fail("A fractional change value on an integer attribute was accepted ({$case})"); - } catch (TypeException $error) { - $this->assertSame('Change value must be an integer.', $error->getMessage(), $case); - } - } - - $this->assertSame(100, $this->stored($database, 'count')); + $database = $this->database($definedAttributes); $this->assertSame(102, $database->increaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'count', 2)->getAttribute('count')); $this->assertSame(3.0, $database->increaseDocumentAttribute(self::COLLECTION, self::DOCUMENT, 'ratio', 1.5)->getAttribute('ratio')); } diff --git a/tests/unit/Documents/FractionalOperatorLimitTest.php b/tests/unit/Documents/FractionalOperatorLimitTest.php index 621e83a4c..d13cf367b 100644 --- a/tests/unit/Documents/FractionalOperatorLimitTest.php +++ b/tests/unit/Documents/FractionalOperatorLimitTest.php @@ -19,34 +19,36 @@ final class FractionalOperatorLimitTest extends TestCase { private const string COLLECTION = 'counters'; - private const string REFUSAL = "Invalid document structure: Cannot apply increment operator: max/min limit must be a whole number for integer attribute 'count', got 102.4"; - public function testUpdateDocumentRefusesAFractionalLimitBeforeTheWrite(): void + public function testUpdateDocumentAppliesAFractionalLimitOnAnIntegerAs7xDid(): void { $database = $this->database(); - try { - $database->updateDocument(self::COLLECTION, 'counter', new Document(['count' => Operator::increment(5, 102.4)])); - $this->fail('A fractional limit on an integer attribute must be refused'); - } catch (StructureException $exception) { - $this->assertSame(self::REFUSAL, $exception->getMessage()); - } + $updated = $database->updateDocument(self::COLLECTION, 'counter', new Document(['count' => Operator::increment(5, 102.4)])); + + $this->assertSame($database->getDocument(self::COLLECTION, 'counter')->getAttribute('count'), $updated->getAttribute('count')); + } + + public function testUpdateDocumentsAppliesAFractionalLimitOnAnIntegerAs7xDid(): void + { + $database = $this->database(); - $this->assertSame(100, $database->getDocument(self::COLLECTION, 'counter')->getAttribute('count')); + $this->assertSame(1, $database->updateDocuments(self::COLLECTION, new Document(['count' => Operator::increment(1, 102.4)]))); + $this->assertSame(101, $database->getDocument(self::COLLECTION, 'counter')->getAttribute('count')); } - public function testUpdateDocumentsRefusesAFractionalLimitBeforeTheWrite(): void + public function testUpdateDocumentRefusesAFractionalLimitOnABigIntegerBeforeTheWrite(): void { $database = $this->database(); try { - $database->updateDocuments(self::COLLECTION, new Document(['count' => Operator::increment(1, 102.4)])); - $this->fail('A fractional limit on an integer attribute must be refused'); + $database->updateDocument(self::COLLECTION, 'counter', new Document(['big' => Operator::increment(5, 4.0e15 + 0.5)])); + $this->fail('A fractional limit on a bigint attribute must be refused'); } catch (StructureException $exception) { - $this->assertSame(self::REFUSAL, $exception->getMessage()); + $this->assertStringStartsWith("Invalid document structure: Cannot apply increment operator: max/min limit must be a whole number for integer attribute 'big'", $exception->getMessage()); } - $this->assertSame(100, $database->getDocument(self::COLLECTION, 'counter')->getAttribute('count')); + $this->assertSame(0, $database->getDocument(self::COLLECTION, 'counter')->getAttribute('big')); } public function testAWholeFloatLimitHoldsABigIntegerAtTheSignedEdge(): void diff --git a/tests/unit/Documents/IncreaseDecreaseTest.php b/tests/unit/Documents/IncreaseDecreaseTest.php index 80b3eb746..acc45c032 100644 --- a/tests/unit/Documents/IncreaseDecreaseTest.php +++ b/tests/unit/Documents/IncreaseDecreaseTest.php @@ -432,7 +432,7 @@ public function testIncreaseDocumentAttributeExceedsMax(): void public function testIncreaseDocumentAttributeWithZeroValue(): void { - $this->expectException(TypeException::class); + $this->expectException(\InvalidArgumentException::class); $this->expectExceptionMessage('Value must be numeric and greater than 0'); $doc = new Document([ @@ -449,7 +449,7 @@ public function testIncreaseDocumentAttributeWithZeroValue(): void public function testIncreaseDocumentAttributeWithNegativeValue(): void { - $this->expectException(TypeException::class); + $this->expectException(\InvalidArgumentException::class); $this->expectExceptionMessage('Value must be numeric and greater than 0'); $doc = new Document([ @@ -526,7 +526,7 @@ public function testDecreaseDocumentAttributeExceedsMin(): void public function testDecreaseDocumentAttributeWithZeroValue(): void { - $this->expectException(TypeException::class); + $this->expectException(\InvalidArgumentException::class); $this->expectExceptionMessage('Value must be numeric and greater than 0'); $doc = new Document([ diff --git a/tests/unit/Documents/IncreaseValueTest.php b/tests/unit/Documents/IncreaseValueTest.php index 2d990ef4a..1fe750dcd 100644 --- a/tests/unit/Documents/IncreaseValueTest.php +++ b/tests/unit/Documents/IncreaseValueTest.php @@ -2,12 +2,11 @@ namespace Tests\Unit\Documents; +use InvalidArgumentException; use PHPUnit\Framework\Attributes\DataProvider; use PHPUnit\Framework\TestCase; use Tests\Unit\Event\HookFixture; use Utopia\Database\Database; -use Utopia\Database\Exception as DatabaseException; -use Utopia\Database\Exception\Type as TypeException; final class IncreaseValueTest extends TestCase { @@ -44,11 +43,11 @@ public function testDecreaseRefusesAChangeThatIsNotAPositiveNumber(int|float|str $this->assertSame(1, $database->getDocument(HookFixture::COLLECTION, 'first')->getAttribute('views')); } - public function testTheRefusalIsADatabaseException(): void + public function testTheRefusalIsAnInvalidArgumentAs7xThrew(): void { $database = $this->database(); - $this->expectException(DatabaseException::class); + $this->expectException(InvalidArgumentException::class); $database->increaseDocumentAttribute(HookFixture::COLLECTION, 'first', 'views', 0); } @@ -66,7 +65,7 @@ private function assertRefused(callable $operation): void try { $operation(); $this->fail('A change that is not a positive number was accepted'); - } catch (TypeException $error) { + } catch (InvalidArgumentException $error) { $this->assertSame('Value must be numeric and greater than 0', $error->getMessage()); } } diff --git a/tests/unit/Documents/NumericUpdateGuardsTest.php b/tests/unit/Documents/NumericUpdateGuardsTest.php index afa3e5cc3..5027971c4 100644 --- a/tests/unit/Documents/NumericUpdateGuardsTest.php +++ b/tests/unit/Documents/NumericUpdateGuardsTest.php @@ -120,8 +120,8 @@ public function testAnIntegerStringChangeIsCheckedAsAnInteger(): void $this->assertSame(12, $database->decreaseDocumentAttribute(self::COLLECTION, 'ledger', 'count', '3')->getAttribute('count')); foreach (['0', '-4'] as $change) { - $this->assertRefused(TypeException::class, 'Value must be numeric and greater than 0', fn (): Document => $database->increaseDocumentAttribute(self::COLLECTION, 'ledger', 'count', $change)); - $this->assertRefused(TypeException::class, 'Value must be numeric and greater than 0', fn (): Document => $database->decreaseDocumentAttribute(self::COLLECTION, 'ledger', 'count', $change)); + $this->assertRefused(\InvalidArgumentException::class, 'Value must be numeric and greater than 0', fn (): Document => $database->increaseDocumentAttribute(self::COLLECTION, 'ledger', 'count', $change)); + $this->assertRefused(\InvalidArgumentException::class, 'Value must be numeric and greater than 0', fn (): Document => $database->decreaseDocumentAttribute(self::COLLECTION, 'ledger', 'count', $change)); } $this->assertSame(12, $database->getDocument(self::COLLECTION, 'ledger')->getAttribute('count')); } diff --git a/tests/unit/Documents/RandomOrderCursorTest.php b/tests/unit/Documents/RandomOrderCursorTest.php new file mode 100644 index 000000000..7d9ea2ab5 --- /dev/null +++ b/tests/unit/Documents/RandomOrderCursorTest.php @@ -0,0 +1,39 @@ +getDocument(HookFixture::COLLECTION, 'second'); + + foreach ([Query::cursorAfter($cursor), Query::cursorBefore($cursor)] as $page) { + $ids = \array_map( + static fn (Document $document): string => $document->getId(), + $database->find(HookFixture::COLLECTION, [Query::orderRandom(), $page]), + ); + \sort($ids); + + $this->assertSame(['first', 'second', 'third'], $ids, $page->getMethod()->value); + } + } + + public function testARandomOrderNextToAnotherOrderReadsEveryRowWhateverTheCursor(): void + { + $database = HookFixture::memory(); + HookFixture::seed($database, ['first', 'second', 'third']); + $cursor = $database->getDocument(HookFixture::COLLECTION, 'second'); + + $documents = $database->find(HookFixture::COLLECTION, [Query::orderRandom(), Query::orderAsc('title'), Query::cursorAfter($cursor)]); + + $this->assertCount(3, $documents); + } +} diff --git a/tests/unit/Documents/StoredValueRevalidationTest.php b/tests/unit/Documents/StoredValueRevalidationTest.php new file mode 100644 index 000000000..644b2ad51 --- /dev/null +++ b/tests/unit/Documents/StoredValueRevalidationTest.php @@ -0,0 +1,86 @@ +}} $attribute */ + return new WhiteList($attribute['formatOptions']['elements'] ?? [], true); + }, ColumnType::String); + } + + public function testAnUpdateFailsOnAStoredValueANarrowedRangeNoLongerAdmits(): void + { + $database = $this->database(); + $database->updateAttribute(self::COLLECTION, 'level', new AttributeUpdate(format: new Format('storedRange', ['min' => 0, 'max' => 10]))); + + try { + $database->updateDocument(self::COLLECTION, 'row', new Document(['note' => 'after narrow'])); + $this->fail('An update kept a stored value the narrowed range refuses'); + } catch (StructureException $error) { + $this->assertSame('Invalid document structure: Attribute "level" has invalid format. Value must be a valid range between 0 and 10', $error->getMessage()); + } + + $this->assertSame('before', $database->getDocument(self::COLLECTION, 'row')->getAttribute('note')); + } + + public function testAnUpdateFailsOnAStoredValueAShrunkEnumNoLongerAdmits(): void + { + $database = $this->database(); + $database->updateAttribute(self::COLLECTION, 'kind', new AttributeUpdate(format: new Format('storedEnum', ['elements' => ['a']]))); + + $this->expectException(StructureException::class); + $this->expectExceptionMessage('Attribute "kind" has invalid format'); + + $database->updateDocument(self::COLLECTION, 'row', new Document(['note' => 'after shrink'])); + } + + private function database(): Database + { + $database = new Database(new Memory(), new Cache(new None())); + $database->setDatabase('stored')->setNamespace('stored_'.\uniqid()); + $database->getAuthorization()->addRole(Role::any()->toString()); + $database->create(); + $database->createCollection(Collection::create( + id: self::COLLECTION, + attributes: [ + Attribute::integer(key: 'level', format: new Format('storedRange', ['min' => 0, 'max' => 100])), + Attribute::string(key: 'kind', size: 8, format: new Format('storedEnum', ['elements' => ['a', 'b']])), + Attribute::string(key: 'note', size: 32), + ], + permissions: [Permission::create(Role::any()), Permission::read(Role::any()), Permission::update(Role::any())], + documentSecurity: false, + )); + $database->createDocument(self::COLLECTION, new Document(['$id' => 'row', 'level' => 50, 'kind' => 'b', 'note' => 'before'])); + + return $database; + } +} diff --git a/tests/unit/Event/DocumentPurgeTest.php b/tests/unit/Event/DocumentPurgeTest.php index 68fc8a5a5..d8f135e42 100644 --- a/tests/unit/Event/DocumentPurgeTest.php +++ b/tests/unit/Event/DocumentPurgeTest.php @@ -233,7 +233,7 @@ public function testPurgeCachedDocumentPurgesTheDocument(Closure $database): voi $this->assertSame(['posts/first'], $this->purged($recorder)); } - public function testDocumentPurgeFiresOnceWhenTheTransactionIsRetried(): void + public function testDocumentPurgeFiresForEveryAttemptThatReachesItsCommit(): void { $adapter = new class () extends Memory { public int $commitFailures = 0; @@ -258,13 +258,16 @@ public function commitTransaction(): bool $database->deleteDocuments(HookFixture::COLLECTION); $this->assertSame([ + Event::DocumentPurge, Event::DocumentPurge, Event::DocumentUpdate, Event::DocumentPurge, Event::DocumentPurge, + Event::DocumentPurge, + Event::DocumentPurge, Event::DocumentsDelete, ], $recorder->getEvents()); - $this->assertSame(['posts/first', 'posts/first', 'posts/second'], $this->purged($recorder)); + $this->assertSame(['posts/first', 'posts/first', 'posts/first', 'posts/second', 'posts/first', 'posts/second'], $this->purged($recorder)); } /** @@ -286,6 +289,26 @@ public function testDocumentPurgeHookFailureReachesTheCaller(Closure $call): voi } } + /** + * @param Closure(Database): mixed $write + */ + #[DataProvider('writes')] + public function testAPurgeListenerFailureRollsTheWriteBack(Closure $write): void + { + $database = HookFixture::sqlite(); + HookFixture::seed($database, ['first', 'second']); + $stored = static fn (): array => \array_map( + static fn (Document $document): array => [$document->getId(), $document->getAttribute('title'), $document->getAttribute('views')], + $database->find(HookFixture::COLLECTION), + ); + $before = $stored(); + $failure = new RuntimeException('region broadcast failed'); + $database->addHook(new FailingLifecycle(Event::DocumentPurge, $failure)); + + $this->assertSame($failure, $this->failureOf(static fn () => $write($database))); + $this->assertSame($before, $stored()); + } + public function testDocumentPurgeThroughMirrorReachesTheCaller(): void { $source = HookFixture::sqlite(); @@ -360,14 +383,14 @@ public static function writes(): iterable * @param Closure(Database): mixed $write */ #[DataProvider('writes')] - public function testPurgeEventsInsideACallerTransactionFireAfterTheCommit(Closure $write): void + public function testPurgeEventsInsideACallerTransactionFireBeforeItsCommit(Closure $write): void { [$database, $recorder] = $this->seeded(HookFixture::sqlite(), ['first']); $inTransaction = $this->observePurges($database, static fn (): bool => $database->getAdapter()->inTransaction()); $database->withTransaction(static fn (): mixed => $write($database)); - $this->assertSame([false], $inTransaction->getArrayCopy()); + $this->assertSame([true], $inTransaction->getArrayCopy()); $this->assertSame(['posts/first'], $this->purged($recorder)); } @@ -466,7 +489,7 @@ static function () use ($database, $abandoned): never { * @param Closure(Database): mixed $write */ #[DataProvider('writes')] - public function testPurgeEventsOfARetriedCallerTransactionFireOnce(Closure $write): void + public function testPurgeEventsOfARetriedCallerTransactionFireForEveryAttemptThatReachesItsCommit(Closure $write): void { $adapter = new class (new PDO('sqlite::memory:')) extends SQLite { public int $commitFailures = 0; @@ -489,7 +512,7 @@ public function commitTransaction(): bool $database->withTransaction(static fn (): mixed => $write($database)); $this->assertSame(0, $adapter->commitFailures); - $this->assertSame(['posts/first'], $this->purged($recorder)); + $this->assertSame(['posts/first', 'posts/first'], $this->purged($recorder)); } public function testPurgeEventsOfWritesWithoutAnAdapterTransactionFireAtOnce(): void @@ -704,7 +727,7 @@ public function testPurgeEventsOfAnUnconfirmedCommitFire(Closure $write, bool $s } #[DataProvider('savepoints')] - public function testOnlyTheUnconfirmedAttemptOfARetriedTransactionAnnouncesItsPurgeEvents(bool $savepoints): void + public function testEveryAttemptOfARetriedTransactionThatReachesItsCommitAnnouncesItsPurgeEvents(bool $savepoints): void { [$database, $recorder] = $this->unconfirmed($savepoints, ['first', 'second'], commitFailures: 1); $attempts = 0; @@ -718,7 +741,7 @@ public function testOnlyTheUnconfirmedAttemptOfARetriedTransactionAnnouncesItsPu $this->assertSame(2, $attempts); $this->assertInstanceOf(UnconfirmedException::class, $thrown); - $this->assertSame(['posts/second'], $this->purged($recorder)); + $this->assertSame(['posts/first', 'posts/second'], $this->purged($recorder)); } #[DataProvider('savepoints')] @@ -740,10 +763,11 @@ static function () use ($database, $abandoned): never { } #[DataProvider('savepoints')] - public function testAnUnconfirmedCommitStaysTheFailureWhenAPurgeListenerFails(bool $savepoints): void + public function testAPurgeListenerFailureRollsTheTransactionBackBeforeItsCommit(bool $savepoints): void { [$database, $recorder] = $this->unconfirmed($savepoints, ['first', 'second']); - $database->addHook(new FailingLifecycle(Event::DocumentPurge, new RuntimeException('region broadcast failed'))); + $failure = new RuntimeException('region broadcast failed'); + $database->addHook(new FailingLifecycle(Event::DocumentPurge, $failure)); $thrown = $this->failureOf(static function () use ($database): void { $database->withTransaction(static function () use ($database): void { @@ -752,8 +776,10 @@ public function testAnUnconfirmedCommitStaysTheFailureWhenAPurgeListenerFails(bo }); }); - $this->assertInstanceOf(UnconfirmedException::class, $thrown); + $this->assertSame($failure, $thrown); $this->assertSame(['posts/first', 'posts/second'], $this->purged($recorder)); + $this->assertSame('first', $database->getDocument(HookFixture::COLLECTION, 'first')->getAttribute('title')); + $this->assertSame('second', $database->getDocument(HookFixture::COLLECTION, 'second')->getAttribute('title')); } /** @@ -779,7 +805,7 @@ static function () use ($database, $foreign): never { } #[DataProvider('savepoints')] - public function testPurgeEventsAreDroppedWhenARetriedCallbackThrowsAnotherCommitsUnconfirmed(bool $savepoints): void + public function testOnlyTheAttemptThatReachedItsCommitAnnouncesWhenARetriedCallbackThrowsAnotherCommitsUnconfirmed(bool $savepoints): void { [$database, $recorder] = $this->unconfirmed($savepoints, ['first', 'second'], commitFailures: 1, confirmed: true); $foreign = new UnconfirmedException('Failed to commit transaction: the commit could not be confirmed'); @@ -798,7 +824,7 @@ public function testPurgeEventsAreDroppedWhenARetriedCallbackThrowsAnotherCommit $this->assertSame($foreign, $this->failureOf(static fn (): mixed => $database->withTransaction($attempt))); $this->assertSame(2, $attempts); - $this->assertSame([], $this->purged($recorder)); + $this->assertSame(['posts/first'], $this->purged($recorder)); } #[DataProvider('savepoints')] diff --git a/tests/unit/Indexes/LegacyJunctionIndexTest.php b/tests/unit/Indexes/LegacyJunctionIndexTest.php new file mode 100644 index 000000000..76e52d20e --- /dev/null +++ b/tests/unit/Indexes/LegacyJunctionIndexTest.php @@ -0,0 +1,24 @@ + '_index_tags', 'key' => 'index_tags', 'type' => 'key', 'attributes' => ['tags']])); + + $this->assertSame('_index_tags', $index->key); + } + + public function testAnyOtherIndexKeepsItsKey(): void + { + $this->assertSame('index_tags', Index::fromDocument(new Document(['$id' => 'index_tags', 'key' => 'index_tags', 'type' => 'key', 'attributes' => ['tags']]))->key); + $this->assertSame('title', Index::fromDocument(new Document(['$id' => 'renamed', 'key' => 'title', 'type' => 'key', 'attributes' => ['tags']]))->key); + $this->assertSame('_index_books', Index::fromDocument(new Document(['$id' => '_index_books', 'key' => '_index_books', 'type' => 'key', 'attributes' => ['books']]))->key); + } +} diff --git a/tests/unit/Indexes/UnknownIndexTypeTest.php b/tests/unit/Indexes/UnknownIndexTypeTest.php new file mode 100644 index 000000000..956ad28f6 --- /dev/null +++ b/tests/unit/Indexes/UnknownIndexTypeTest.php @@ -0,0 +1,23 @@ +expectException(IndexException::class); + $this->expectExceptionMessage('Unknown index type: nope. Must be one of key, unique, fulltext, spatial, object, hnsw_euclidean, hnsw_cosine, hnsw_dot, trigram, ttl'); + + Index::fromArray(['key' => 'i', 'type' => 'nope', 'attributes' => ['name']]); + } + + public function testAMissingTypeIsAKeyIndex(): void + { + $this->assertSame('key', Index::fromArray(['key' => 'i', 'attributes' => ['name']])->type->value); + } +} diff --git a/tests/unit/Joins/JoinInternalColumnsTest.php b/tests/unit/Joins/JoinInternalColumnsTest.php index df277e73d..7cf423df5 100644 --- a/tests/unit/Joins/JoinInternalColumnsTest.php +++ b/tests/unit/Joins/JoinInternalColumnsTest.php @@ -148,7 +148,6 @@ public function testTenantIsRejectedWithoutSharedTables(): void foreach ([ 'count' => [[Query::count('$tenant', 'total')], '$tenant'], 'groupBy' => [[Query::count('*', 'rows'), Query::groupBy(['$tenant'])], '$tenant'], - 'select' => [[Query::select(['name', '$tenant'])], '$tenant'], 'joined count' => [[$note, Query::count('note.$tenant', 'total')], 'note.$tenant'], 'joined groupBy' => [[$note, Query::count('*', 'rows'), Query::groupBy(['note.$tenant'])], 'note.$tenant'], 'joined select' => [[$note, Query::select(['name', 'note.$tenant'])], 'note.$tenant'], @@ -165,6 +164,12 @@ public function testTenantIsRejectedWithoutSharedTables(): void ); } + public function testSelectingTheTenantWithoutSharedTablesIsRefusedByTheRead(): void + { + $this->assertInvalidQuery('Cannot select attributes: $tenant', fn (): mixed => $this->database->find('customers', [Query::select(['name', '$tenant'])])); + $this->assertInvalidQuery('Cannot select attributes: $tenant', fn (): mixed => $this->database->getDocument('customers', 'c1', [Query::select(['name', '$tenant'])])); + } + public function testTenantIsReadUnderSharedTables(): void { $this->useDatabase(new SQLite(new PDO('sqlite::memory:')), sharedTables: true); diff --git a/tests/unit/MongoTransactionRetryTest.php b/tests/unit/MongoTransactionRetryTest.php index b1338b7b2..4f3fb456a 100644 --- a/tests/unit/MongoTransactionRetryTest.php +++ b/tests/unit/MongoTransactionRetryTest.php @@ -58,7 +58,6 @@ public static function transientFailures(): array 'contention' => [new ContentionException('Write conflict')], 'transaction' => [new TransactionException('Transaction aborted')], 'labelled transient' => [new MongoException('Transaction was aborted', 251, null, [self::TRANSIENT_TRANSACTION_ERROR])], - 'write conflict' => [new MongoException('WriteConflict', self::WRITE_CONFLICT)], 'network error' => [new MongoException('Socket error', self::SOCKET_EXCEPTION)], 'unsent' => [new UnsentException('Connection to MongoDB has been lost')], 'failure wrapping a transient error' => [new DatabaseException('Failed to commit transaction', previous: new MongoException('Transaction was aborted', 0, null, [self::TRANSIENT_TRANSACTION_ERROR]))], @@ -94,6 +93,19 @@ public function testTransientFailureRunsEveryAttempt(Throwable $failure): void $this->assertFalse($adapter->inTransaction()); } + public function testAWriteConflictRunsUntilItsRetriesForWriteConflictsRunOut(): void + { + $client = new ReplicaSetClient(); + $adapter = new Mongo($client); + $failure = new MongoException('WriteConflict', self::WRITE_CONFLICT); + + [$thrown, $attempts] = $this->attempt($adapter, $failure); + + $this->assertSame($failure, $thrown); + $this->assertSame(21, $attempts); + $this->assertFalse($adapter->inTransaction()); + } + public function testTransientFailureSucceedsWhenItRunsAgain(): void { $client = new ReplicaSetClient(); diff --git a/tests/unit/Operator/OperatorLimitTest.php b/tests/unit/Operator/OperatorLimitTest.php index 655693d19..886f1e78c 100644 --- a/tests/unit/Operator/OperatorLimitTest.php +++ b/tests/unit/Operator/OperatorLimitTest.php @@ -14,29 +14,36 @@ final class OperatorLimitTest extends TestCase { /** - * @return array + * @return array */ public static function fractionalLimits(): array { return [ - 'increment max' => [OperatorType::Increment, 'count', 102.4], - 'decrement min' => [OperatorType::Decrement, 'count', -0.5], - 'multiply max' => [OperatorType::Multiply, 'count', 99.9], - 'divide min' => [OperatorType::Divide, 'count', 1.5], - 'power max' => [OperatorType::Power, 'count', 1000.01], - 'numeric string' => [OperatorType::Increment, 'count', '102.4'], - 'big integer' => [OperatorType::Increment, 'big', 4.0e15 + 0.5], + 'increment max' => [OperatorType::Increment, 102.4], + 'decrement min' => [OperatorType::Decrement, -0.5], + 'multiply max' => [OperatorType::Multiply, 99.9], + 'divide min' => [OperatorType::Divide, 1.5], + 'power max' => [OperatorType::Power, 1000.01], + 'numeric string' => [OperatorType::Increment, '102.4'], ]; } #[DataProvider('fractionalLimits')] - public function testAFractionalLimitOnAnIntegerAttributeIsRefused(OperatorType $method, string $attribute, int|float|string $limit): void + public function testAFractionalLimitOnAnIntegerAttributeIsLeftToTheEngineAs7xDid(OperatorType $method, int|float|string $limit): void { $validator = $this->validator(); - $this->assertFalse($validator->isValid(new Operator($method, $attribute, [2, $limit]))); + $this->assertTrue($validator->isValid(new Operator($method, 'count', [2, $limit])), $validator->getDescription()); + } + + public function testAFractionalLimitOnABigIntegerAttributeIsRefused(): void + { + $validator = $this->validator(); + $limit = 4.0e15 + 0.5; + + $this->assertFalse($validator->isValid(new Operator(OperatorType::Increment, 'big', [2, $limit]))); $this->assertSame( - "Cannot apply {$method->value} operator: max/min limit must be a whole number for integer attribute '{$attribute}', got {$limit}", + "Cannot apply increment operator: max/min limit must be a whole number for integer attribute 'big', got {$limit}", $validator->getDescription(), ); } diff --git a/tests/unit/Relationships/LinkPreservedDatesTest.php b/tests/unit/Relationships/LinkPreservedDatesTest.php new file mode 100644 index 000000000..47fe0a8c0 --- /dev/null +++ b/tests/unit/Relationships/LinkPreservedDatesTest.php @@ -0,0 +1,83 @@ +database(); + + $authorization->skip(fn () => $database->withPreserveDates(true, fn () => $database->updateDocument('authors', 'author3', new Document(['books' => ['book4', 'book5']])))); + + foreach (['book4', 'book5'] as $id) { + $book = $authorization->skip(fn () => $database->getDocument('books', $id)); + $author = $book->getAttribute('author'); + $this->assertInstanceOf(Document::class, $author, $id); + $this->assertSame('author3', $author->getId(), $id); + $this->assertSame(self::STORED, $book->getUpdatedAt(), $id); + } + } + + public function testLinkingWithoutPreservedDatesStillStampsTheChildren(): void + { + [$database, $authorization] = $this->database(); + + $authorization->skip(fn () => $database->updateDocument('authors', 'author3', new Document(['books' => ['book4']]))); + + $this->assertNotSame(self::STORED, $authorization->skip(fn () => $database->getDocument('books', 'book4'))->getUpdatedAt()); + } + + public function testUnlinkingUnderPreservedDatesStillStampsTheChild(): void + { + [$database, $authorization] = $this->database(); + $authorization->skip(fn () => $database->withPreserveDates(true, fn () => $database->updateDocument('authors', 'author3', new Document(['books' => ['book4']])))); + + $authorization->skip(fn () => $database->withPreserveDates(true, fn () => $database->updateDocument('authors', 'author3', new Document(['books' => []])))); + + $this->assertNotSame(self::STORED, $authorization->skip(fn () => $database->getDocument('books', 'book4'))->getUpdatedAt()); + } + + /** + * @return array{Database, Authorization} + */ + private function database(): array + { + $authorization = new Authorization(); + $database = (new Database(new Memory(), new Cache(new None()))) + ->setAuthorization($authorization) + ->setDatabase('link_dates') + ->setNamespace('link_dates_'.\uniqid()); + $database->addHook(new Relationships()); + + $authorization->skip(function () use ($database): void { + $database->create(); + $database->createCollection(Collection::create('authors', attributes: [Attribute::string('name', size: 64)])); + $database->createCollection(Collection::create('books', attributes: [Attribute::string('name', size: 64)])); + $database->createRelationship('authors', Relationship::oneToMany('books', 'books', twoWay: true, twoWayKey: 'author', onDelete: RelationshipDeleteAction::SetNull)); + $database->withPreserveDates(true, function () use ($database): void { + $database->createDocument('authors', new Document(['$id' => 'author3', 'name' => 'Linus', '$createdAt' => self::STORED, '$updatedAt' => self::STORED])); + foreach (['book4', 'book5'] as $id) { + $database->createDocument('books', new Document(['$id' => $id, 'name' => $id, '$createdAt' => self::STORED, '$updatedAt' => self::STORED])); + } + }); + }); + + return [$database, $authorization]; + } +} diff --git a/tests/unit/UniqueViolationTest.php b/tests/unit/UniqueViolationTest.php index 6cb67516c..0d94944b1 100644 --- a/tests/unit/UniqueViolationTest.php +++ b/tests/unit/UniqueViolationTest.php @@ -435,6 +435,6 @@ private function assertDuplicate(Throwable $exception): void private function assertUnique(Throwable $exception): void { $this->assertInstanceOf(UniqueException::class, $exception, $exception->getMessage()); - $this->assertSame('Document with the requested unique attributes already exists', $exception->getMessage()); + $this->assertSame('Unique index violation', $exception->getMessage()); } } diff --git a/tests/unit/Validator/AttributeJsonDefaultTest.php b/tests/unit/Validator/AttributeJsonDefaultTest.php index 21d441956..e7faeac3c 100644 --- a/tests/unit/Validator/AttributeJsonDefaultTest.php +++ b/tests/unit/Validator/AttributeJsonDefaultTest.php @@ -33,7 +33,7 @@ public static function scalarDefaults(): array return [ 'integer' => [12345, 'Default value 12345 does not match given type string'], 'float' => [1.5, 'Default value 1.5 does not match given type string'], - 'boolean' => [true, 'Default value true does not match given type string'], + 'boolean' => [true, 'Default value 1 does not match given type string'], ]; } diff --git a/tests/unit/Validator/AttributeTest.php b/tests/unit/Validator/AttributeTest.php index b1b93ba51..f22927af3 100644 --- a/tests/unit/Validator/AttributeTest.php +++ b/tests/unit/Validator/AttributeTest.php @@ -389,7 +389,7 @@ public function test_default_value_type_mismatch(): void ]); $this->expectException(DatabaseException::class); - $this->expectExceptionMessage('Default value "not_an_integer" does not match given type integer'); + $this->expectExceptionMessage('Default value not_an_integer does not match given type integer'); $validator->isValid($attribute); } @@ -933,7 +933,7 @@ public function test_float_default_value_type_mismatch(): void ]); $this->expectException(DatabaseException::class); - $this->expectExceptionMessage('Default value "not_a_float" does not match given type double'); + $this->expectExceptionMessage('Default value not_a_float does not match given type double'); $validator->isValid($attribute); } @@ -957,7 +957,7 @@ public function test_boolean_default_value_type_mismatch(): void ]); $this->expectException(DatabaseException::class); - $this->expectExceptionMessage('Default value "not_a_boolean" does not match given type boolean'); + $this->expectExceptionMessage('Default value not_a_boolean does not match given type boolean'); $validator->isValid($attribute); } @@ -1976,7 +1976,7 @@ public function testBigIntegerDefaultRejectsNonNumericString(): void $validator = new AttributeDefinition(attributes: [], profile: Profiles::of()); $this->expectException(DatabaseException::class); - $this->expectExceptionMessage('does not match given type bigint'); + $this->expectExceptionMessage('Default value not_a_bigint is not a valid integer string for type bigint'); $validator->isValid(Attribute::bigInteger( key: 'counter', default: 'not_a_bigint', diff --git a/tests/unit/Validator/DocumentsQueriesGrammarTest.php b/tests/unit/Validator/DocumentsQueriesGrammarTest.php index bb33a0b5b..e40299522 100644 --- a/tests/unit/Validator/DocumentsQueriesGrammarTest.php +++ b/tests/unit/Validator/DocumentsQueriesGrammarTest.php @@ -74,7 +74,7 @@ public function test_default_grammar_rejects_extended_methods(Query $query, stri ); $this->assertFalse($validator->isValid([$query])); - $this->assertSame('Invalid query method: '.$method, $validator->getDescription()); + $this->assertSame('Invalid query: Invalid query method: '.$method, $validator->getDescription()); } #[DataProvider('joinQueries')] @@ -112,7 +112,7 @@ public function test_enabling_joins_does_not_enable_aggregations(Query $query, s ); $this->assertFalse($validator->isValid([$query])); - $this->assertSame('Invalid query method: '.$method, $validator->getDescription()); + $this->assertSame('Invalid query: Invalid query method: '.$method, $validator->getDescription()); } #[DataProvider('joinQueries')] @@ -125,6 +125,6 @@ public function test_enabling_aggregations_does_not_enable_joins(Query $query, s ); $this->assertFalse($validator->isValid([$query])); - $this->assertSame('Invalid query method: '.$method, $validator->getDescription()); + $this->assertSame('Invalid query: Invalid query method: '.$method, $validator->getDescription()); } } diff --git a/tests/unit/Validator/JoinConditionColumnsTest.php b/tests/unit/Validator/JoinConditionColumnsTest.php index 31f30b1f6..14635ef34 100644 --- a/tests/unit/Validator/JoinConditionColumnsTest.php +++ b/tests/unit/Validator/JoinConditionColumnsTest.php @@ -150,7 +150,6 @@ public static function internalAttributesWithoutAColumn(): iterable yield '$collection grouped' => [[Query::count('*', 'rows'), Query::groupBy(['$collection'])], '$collection', false]; yield '$tenant counted' => [[Query::count('$tenant', 'total')], '$tenant', true]; yield '$tenant grouped' => [[Query::count('*', 'rows'), Query::groupBy(['$tenant'])], '$tenant', true]; - yield '$tenant selected' => [[Query::select(['name', '$tenant'])], '$tenant', true]; yield 'a joined $tenant counted' => [[$note, Query::count('note.$tenant', 'total')], 'note.$tenant', true]; yield 'a joined $tenant grouped' => [[$note, Query::count('*', 'rows'), Query::groupBy(['note.$tenant'])], 'note.$tenant', true]; yield 'a joined $tenant selected' => [[$note, Query::select(['name', 'note.$tenant'])], 'note.$tenant', true]; @@ -190,8 +189,15 @@ public function testTenantIsRejectedByValidatorsBuiltWithoutSharedTables(): void $this->assertFalse((new Aggregate($attributes, sharedTables: true))->isValid(Query::count('$collection', 'total'))); $this->assertFalse((new GroupBy($attributes))->isValid(Query::groupBy(['$tenant']))); $this->assertTrue((new GroupBy($attributes, sharedTables: true))->isValid(Query::groupBy(['$tenant']))); - $this->assertFalse((new Select($attributes))->isValid(Query::select(['$tenant']))); + } + + public function testSelectLetsTheMainTenantThroughWithoutSharedTables(): void + { + $attributes = $this->customers(); + + $this->assertTrue((new Select($attributes))->isValid(Query::select(['$tenant']))); $this->assertTrue((new Select($attributes, sharedTables: true))->isValid(Query::select(['$tenant']))); + $this->assertTrue($this->documents(sharedTables: false)->isValid([Query::select(['name', '$tenant'])])); } public function testEncryptedJoinedAttributeCannotBeFiltered(): void diff --git a/tests/unit/Validator/JoinedColumnTest.php b/tests/unit/Validator/JoinedColumnTest.php index 19a1cd8be..b8e67ba51 100644 --- a/tests/unit/Validator/JoinedColumnTest.php +++ b/tests/unit/Validator/JoinedColumnTest.php @@ -46,6 +46,7 @@ public static function internalAttributeProvider(): array $cases[$attribute.' in '.$type] = [$attribute, $queries]; } } + unset($cases['$tenant in select']); return $cases; } @@ -64,6 +65,14 @@ public function testInternalAttributeUnderAnAliasFollowsTheRuleOfTheMainCollecti $this->assertSame($unaliased, $aliased, $validator->getDescription()); } + public function testTenantSelectPassesOnlyOnTheMainCollectionWithoutSharedTables(): void + { + $validator = $this->validator([$this->notes()]); + + $this->assertTrue($validator->isValid([Query::select(['$tenant'])]), $validator->getDescription()); + $this->assertFalse($validator->isValid([self::join(), Query::select(['note.$tenant'])])); + } + public function testCollectionOfAJoinedRowCannotBeReferenced(): void { $validator = $this->validator([$this->notes()]); diff --git a/tests/unit/Validator/OperatorValidatorTest.php b/tests/unit/Validator/OperatorValidatorTest.php index 894c38a98..c55d775e0 100644 --- a/tests/unit/Validator/OperatorValidatorTest.php +++ b/tests/unit/Validator/OperatorValidatorTest.php @@ -163,13 +163,11 @@ public function testAStoredValueInsideTheAttributeRangeIsAccepted(): void $this->assertTrue($validator->isValid(new Operator(OperatorType::Increment, 'count', [1])), $validator->getDescription()); } - public function testAResultThatCannotBePredictedIsRejected(): void + public function testAFractionalResultInRangeIsLeftToTheEngineAs7xDid(): void { $validator = $this->numericValidator(new Document(['count' => 2])); - $this->assertFalse($validator->isValid(new Operator(OperatorType::Power, 'count', [-1]))); - $this->assertSame('Cannot apply power operator: result is outside the attribute range', $validator->getDescription()); - + $this->assertTrue($validator->isValid(new Operator(OperatorType::Power, 'count', [-1])), $validator->getDescription()); $this->assertTrue($validator->isValid(new Operator(OperatorType::Power, 'count', [3])), $validator->getDescription()); } @@ -253,7 +251,8 @@ public function testAWholeFloatOperandOnAnIntegerIsAcceptedAsTheInteger(): void $this->assertFalse($validator->isValid(new Operator(OperatorType::Increment, 'count', [6.0]))); $this->assertStringContainsString('would overflow maximum value', $validator->getDescription()); $this->assertFalse($validator->isValid(new Operator(OperatorType::Increment, 'count', [5.5]))); - $this->assertStringStartsWith('Cannot apply increment operator: value must be numeric', $validator->getDescription()); + $this->assertSame('Cannot apply increment operator: would overflow maximum value of 2147483647', $validator->getDescription()); + $this->assertTrue($validator->isValid(new Operator(OperatorType::Increment, 'count', [4.5])), $validator->getDescription()); $this->assertFalse($validator->isValid(new Operator(OperatorType::Divide, 'count', [0.0]))); $this->assertSame('Cannot apply divide operator: division by zero', $validator->getDescription()); } diff --git a/tests/unit/Validator/ProfileConstructionTest.php b/tests/unit/Validator/ProfileConstructionTest.php index 8c59b6657..dbea15373 100644 --- a/tests/unit/Validator/ProfileConstructionTest.php +++ b/tests/unit/Validator/ProfileConstructionTest.php @@ -117,7 +117,7 @@ public function testADocumentsValidatorTakesItsGrammarFromTheProfile(): void $plain = new Documents($attributes, [], Profiles::of(capabilities: [Capability::DefinedAttributes])); $this->assertFalse($plain->isValid($count)); - $this->assertSame('Invalid query method: count', $plain->getDescription()); + $this->assertSame('Invalid query: Invalid query method: count', $plain->getDescription()); $this->assertFalse($plain->isValid($join)); $grammar = new Documents($attributes, [], Profiles::of(capabilities: [Capability::DefinedAttributes, Capability::Aggregations, Capability::Joins])); @@ -135,11 +135,11 @@ public function testQueryValidatorsTakeTheUidLengthFromTheLimits(): void $this->assertTrue(Narrow::of($cursor, $attributes, Profiles::of(uidLength: 255), 5000)?->isValid($cursor) ?? false); } - public function testADocumentValidatorAcceptsTheTenantOnlyUnderSharedTables(): void + public function testADocumentValidatorLetsTheTenantThroughToTheRead(): void { $select = [Query::select(['$tenant'])]; - $this->assertFalse(new DocumentValidator([], Profiles::of(capabilities: [Capability::DefinedAttributes]))->isValid($select)); + $this->assertTrue(new DocumentValidator([], Profiles::of(capabilities: [Capability::DefinedAttributes]))->isValid($select)); $this->assertTrue(new DocumentValidator([], Profiles::of(capabilities: [Capability::DefinedAttributes], sharedTables: true))->isValid($select)); } diff --git a/tests/unit/Validator/QueriesCompatibilityTest.php b/tests/unit/Validator/QueriesCompatibilityTest.php new file mode 100644 index 000000000..c6c47a2c7 --- /dev/null +++ b/tests/unit/Validator/QueriesCompatibilityTest.php @@ -0,0 +1,104 @@ + 'name', 'key' => 'name', 'type' => ColumnType::String->value, 'size' => 128, 'array' => false]), + new Document(['$id' => 'age', 'key' => 'age', 'type' => ColumnType::Integer->value, 'size' => 4, 'array' => false]), + new Document(['$id' => 'tags', 'key' => 'tags', 'type' => ColumnType::String->value, 'size' => 64, 'array' => true]), + ], + [], + Profiles::of(capabilities: [Capability::DefinedAttributes, Capability::OrderRandom], sharedTables: $sharedTables), + ); + } + + public function test_nested_child_is_validated_before_its_parent(): void + { + $validator = $this->documents(); + + $this->assertFalse($validator->isValid(['{"method":"and","values":[{"method":"equal","attribute":"nope","values":["a"]}]}'])); + $this->assertSame('Invalid query: Attribute not found in schema: nope', $validator->getDescription()); + + $this->assertFalse($validator->isValid(['{"method":"or","values":[{"method":"equal","attribute":"nope","values":["a"]},{"method":"limit","values":[1]}]}'])); + $this->assertSame('Invalid query: Attribute not found in schema: nope', $validator->getDescription()); + + $this->assertFalse($validator->isValid(['{"method":"elemMatch","attribute":"tags","values":[{"method":"equal","attribute":"x","values":["a"]}]}'])); + $this->assertSame('Invalid query: Attribute not found in schema: x', $validator->getDescription()); + } + + public function test_first_invalid_query_wins_over_a_later_parse_failure(): void + { + $validator = $this->documents(); + + $this->assertFalse($validator->isValid(['{"method":"equal","attribute":"nope","values":["a"]}', 'not json'])); + $this->assertSame('Invalid query: Attribute not found in schema: nope', $validator->getDescription()); + + $this->assertFalse($validator->isValid(['not json', '{"method":"equal","attribute":"nope","values":["a"]}'])); + $this->assertSame('Invalid query: Invalid query: Syntax error', $validator->getDescription()); + } + + public function test_method_unknown_to_7x_keeps_the_parse_prefix(): void + { + $validator = new Base([new Limit(), new Offset(), new Cursor()]); + + $this->assertFalse($validator->isValid(['{"method":"count","attribute":"name","values":[]}'])); + $this->assertSame('Invalid query: Invalid query method: count', $validator->getDescription()); + + $this->assertFalse($validator->isValid(['{"method":"jsonContains","attribute":"name","values":["a"]}'])); + $this->assertSame('Invalid query: Invalid query method: jsonContains', $validator->getDescription()); + + $this->assertFalse($validator->isValid(['{"method":"and","values":[{"method":"equal","attribute":"name","values":["a"]},{"method":"equal","attribute":"name","values":["b"]}]}'])); + $this->assertSame('Invalid query method: equal', $validator->getDescription()); + } + + public function test_raw_is_an_invalid_method(): void + { + $validator = $this->documents(); + + $this->assertFalse($validator->isValid(['{"method":"raw","values":["1=1"]}'])); + $this->assertSame('Invalid query: Invalid query method: raw', $validator->getDescription()); + + try { + Query::parse('{"method":"or","values":[{"method":"raw","values":["1=1"]},{"method":"equal","attribute":"name","values":["a"]}]}'); + $this->fail('A nested raw query parsed'); + } catch (QueryException $e) { + $this->assertSame('Invalid query method: raw', $e->getMessage()); + } + } + + public function test_tenant_passes_select_validation_without_shared_tables(): void + { + $this->assertTrue($this->documents()->isValid(['{"method":"select","values":["$tenant","name"]}'])); + $this->assertTrue($this->documents(sharedTables: true)->isValid(['{"method":"select","values":["$tenant"]}'])); + } + + public function test_uid_description(): void + { + $this->assertSame( + 'UID must contain at most 36 chars. Valid chars are a-z, A-Z, 0-9, and underscore. Can\'t start with a leading underscore', + (new UID())->getDescription(), + ); + } +} diff --git a/tests/unit/Validator/QueryValidationTest.php b/tests/unit/Validator/QueryValidationTest.php index a7842b099..837097831 100644 --- a/tests/unit/Validator/QueryValidationTest.php +++ b/tests/unit/Validator/QueryValidationTest.php @@ -389,7 +389,7 @@ public function testDocumentQueriesRejectJoinsUnlessEnabled(): void $this->assertTrue($enabled->isValid([$join]), $shape.': '.$enabled->getDescription()); $this->assertFalse($disabled->isValid([$join]), $shape); - $this->assertSame('Invalid query method: join', $disabled->getDescription(), $shape); + $this->assertSame('Invalid query: Invalid query method: join', $disabled->getDescription(), $shape); } $this->assertTrue($disabled->isValid([Query::select(['name'])]), $disabled->getDescription());