Expose anvil_file.file_path column from v6 schema (#8315) by hannes-ucsc · Pull Request #8376 · DataBiosphere/azul

hannes-ucsc · 2026-10-07T00:21:54Z

Linked issues: #8315

Checklist

Author

¹ when the issue title describes a problem, the corresponding PR
title is Fix: followed by the issue title

Author (partiality)

B01 Added p tag to titles of partial commits
B02 This PR is labeled partial _{or completely resolves all linked issues}
B03 This PR partially resolves each of the linked issues _{or does not have the partial label}

Author (reindex)

C01 Added r tag to commit title _{or the changes introduced by this PR will not require reindexing of any deployment}
C02 This PR is labeled reindex:dev _{or the changes introduced by it will not require reindexing of dev}
C03 This PR is labeled reindex:anvildev _{or the changes introduced by it will not require reindexing of anvildev}
C04 This PR is labeled reindex:anvilprod _{or the changes introduced by it will not require reindexing of anvilprod}
C05 This PR is labeled reindex:prod _{or the changes introduced by it will not require reindexing of prod}
C06 This PR is labeled reindex:partial and its description documents the specific reindexing procedure for dev, anvildev, anvilprod and prod _{or requires a full reindex or carries none of the labels reindex:dev, reindex:anvildev, reindex:anvilprod and reindex:prod}

Author (mirror)

D01 This PR is labeled mirror:dev _{or the changes introduced by it will not require mirroring of dev}
D02 This PR is labeled mirror:anvildev _{or the changes introduced by it will not require mirroring of anvildev}
D03 This PR is labeled mirror:anvilprod _{or the changes introduced by it will not require mirroring of anvilprod}
D04 This PR is labeled mirror:prod _{or the changes introduced by it will not require mirroring of prod}
D05 This PR is labeled mirror:partial and its description documents the specific mirroring procedure for dev, anvildev, anvilprod and prod _{or requires a full mirroring or carries none of the labels mirror:dev, mirror:anvildev, mirror:anvilprod and mirror:prod}

Author (API changes)

E01 This PR and its linked issues are labeled API _{or this PR does not modify a REST API}
E02 Added a (A) tag to commit title for backwards (in)compatible changes _{or this PR does not modify a REST API}
E03 Updated REST API version number in app.py _{or this PR does not modify a REST API}

Author (upgrading deployments)

F01 Ran make docker_images.json and committed the resulting changes _{or this PR does not modify azul_docker_images, or any other variables referenced in the definition of that variable}
F02 Documented upgrading of deployments in UPGRADING.rst _{or this PR does not require upgrading deployments}
F03 Added u tag to commit title _{or this PR does not require upgrading deployments}
F04 This PR is labeled upgrade _{or does not require upgrading deployments}
F05 This PR is labeled deploy:shared _{or does not modify docker_images.json, and does not require deploying the shared component for any other reason}
F06 This PR is labeled deploy:gitlab _{or does not require deploying the gitlab component}
F07 This PR is labeled deploy:runner _{or does not require deploying the runner image}

Author (hotfixes)

G01 Added F tag to main commit title _{or this PR does not include permanent fix for a temporary hotfix}
G02 Reverted the temporary hotfixes for any linked issues _{or the none of the stable branches (anvilprod and prod) have temporary hotfixes for any of the issues linked to this PR}

Author (before every review)

H01 Rebased PR branch on develop, squashed fixups from prior reviews
H02 Ran make requirements_update _{or this PR does not modify pyproject.toml}
H03 Added R tag to commit title _{or this PR does not modify uv.lock}
H04 This PR is labeled reqs _{or does not modify uv.lock}
H05 make integration_test passes in personal deployment _{or this PR does not modify functionality that could affect the IT outcome}
H06 PR is awaiting requested review from a peer
H07 Status of PR is Review requested
H08 PR is assigned to only the peer and the author

Peer reviewer (after approval)

Note that after requesting changes, the PR must be assigned to only the author.

J01 Actually approved the PR
J02 PR is not a draft
J03 PR is awaiting requested review from system administrator
J04 Status of PR is Review requested
J05 PR is assigned to only the system administrator and the author

System administrator (after approval)

K01 Actually approved the PR
K02 Labeled linked issues as demo or no demo
K03 Commented on linked issues about demo expectations _{or all linked issues are labeled no demo}
K04 Decided if PR can be labeled no sandbox
K05 A comment to this PR details the completed security design review
K06 PR title is appropriate as title of merge commit
K07 N reviews label is accurate
K08 Status of PR is Approved
K09 PR is assigned to only the operator and the author

Operator

L01 Checked reindex:… labels and r commit title tag
L02 Checked mirror:… labels
L03 Checked that demo expectations are clear _{or all linked issues are labeled no demo}
L04 Squashed PR branch and rebased onto develop
L05 Sanity-checked history
L06 Pushed PR branch to GitHub

Operator (deploy `.shared` and `.gitlab` components)

M01 Ran _select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
M02 Ran _select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply(an error from _login_docker_gitlab is benign if the instance was stopped for backup) _{or this PR is not labeled deploy:gitlab}
M03 Ran _select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
M04 Ran _select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply(an error from _login_docker_gitlab is benign if the instance was stopped for backup) _{or this PR is not labeled deploy:gitlab}
M05 Checked the items in the next section _{or this PR is labeled deploy:gitlab}
M06 PR is assigned to only the system administrator and the author _{or this PR is not labeled deploy:gitlab}

System administrator (post-deploy of `.gitlab` component)

N01 Background migrations for dev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
N02 Background migrations for anvildev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
N03 PR is assigned to only the operator and the author

Operator (deploy runner image)

P01 Ran _select dev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}
P02 Ran _select anvildev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}

Operator (sandbox build)

Operator (merge the branch)

R01 All status checks passed and the PR is mergeable
R02 The title of the merge commit starts with the title of this PR
R03 Added PR # reference to merge commit title
R04 Collected commit title tags in merge commit title _{but only included p if the PR is also labeled partial}
R05 Pushed merge commit to GitHub
R06 Status of PR is Merged lower
R07 Status of blocked issues is Triage _{or no issues are blocked on the linked issues}

Operator (main build)

Operator (reindex)

Operator (mirroring)

U01 Started mirroring in dev _{or this PR is not labelled mirror:dev}
U02 Started mirroring in anvildev _{or this PR is not labelled mirror:anvildev}
U03 Checked for, triaged and possibly requeued messages in mirror fail queue in dev _{or this PR is not labelled mirror:dev}
U04 Checked for, triaged and possibly requeued messages in mirror fail queue in anvildev _{or this PR is not labelled mirror:anvildev}
U05 Emptied mirror fail queue in dev _{or this PR is not labelled mirror:dev}
U06 Emptied mirror fail queue in anvildev _{or this PR is not labelled mirror:anvildev}

Operator

V01 Propagated the upgrade, API, deploy:shared, deploy:gitlab, deploy:runner, reindex:partial, reindex:anvilprod, reindex:prod, mirror:partial, mirror:anvilprod and mirror:prod labels to any open promotion PRs _{or this PR carries none of these labels, or is not included in an open promotion PR}
V02 Propagated any specific instructions related to those labels, from the description of this PR to that of any open promotion PRs _{or this PR carries none of those labels, or is not included in an open promotion PR}
V03 PR is assigned to no one

Shorthand for review comments

L line is too long
W line wrapping is wrong
Q bad quotes
F other formatting problem

AnVIL entity IDs are currently the `datarepo_row_id` of the row an entity originates from. That column is not stable across releases of a dataset: of the 72 datasets re-released between the `anvil14` and `anvil15` catalogs, not one of the 1353006 rows compared kept its `datarepo_row_id`. Primary keys are far more stable, but they are only unique within a table of a snapshot, so they can't serve as entity IDs on their own. `Plugin._entity_id` therefore derives a v5 UUID from the dataset, the table and the primary key. It has no caller yet; the transition away from `datarepo_row_id` follows in a later commit. The dataset is identified by the one component of the snapshot name that survives a re-release, lower-cased. The deployment configurations derive the same name in their `source` function and key their catalogs by it, so that a new release of a dataset supersedes the previous one. `TestAnvilSnapshotNames` asserts that the two agree by feeding every configured snapshot name back through the configuration that declared it. `SnapshotName` also replaces the ad-hoc regular expression the plugin used to extract the schema version from a snapshot name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Three places in the plugin derived the name of a table's primary key column by stripping the `anvil_` prefix from the table name and appending `_id`. The AnVIL schema declares that column for every table it describes, so there is no need to guess it. The lookup is per schema version, like the one for column names next to it, because a future version could rename a key. The schema has always declared exactly one primary key column per table, and always named it by that convention, so this changes no behavior today. `test_pk_column` pins the declaration against the convention, so that a future schema that departs from it fails the build instead of quietly producing queries against a column that doesn't exist. Reading the declaration through `one` likewise turns the assumption that a primary key is a single column into an assertion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Entities from tables the AnVIL schema describes are now identified by a UUID derived from the dataset, the table and the row's primary key, instead of by the row's `datarepo_row_id`. Entity IDs are therefore stable across releases of a dataset, where `datarepo_row_id` was not. They are user-visible, in `/index/{entity_type}/{entity_id}`, in file download URLs and in manifests, so they change once with this commit and are expected to survive re-releases from here on. The three places that identified a row all had to change together. `anvil_file` rows are reached both by the traversal that builds a primary bundle and by the batches that build a supplementary bundle, and `anvil_dataset` rows both by that traversal and by `_get_dataset`. Converting one site and not another would give a row two different IDs, depending on which bundle contributed it. The new `_entity_ref` is the one place that decides. Tables the schema doesn't describe declare no primary key, so there is nothing stable to derive an ID from and their rows keep their `datarepo_row_id`. Such rows only ever occur as replicas. The column itself is unaffected and remains part of every replica, which is why the canned index documents still mention the old IDs in that position, and why the verbatim manifest fixtures didn't change at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Every place that slices a table by a prefix of a column's value now uses the primary key the schema declares for that table, falling back to `datarepo_row_id` only for tables the schema doesn't describe. These had to change together: a batch prefix extends the prefix of the partition it falls in, so partitioning a table by one column while batching it by another would let rows fall through the gap between them. Batch membership therefore no longer changes when a dataset is re-released, which makes the contents of a batched bundle stable, not just its UUID. It also means that two rows sharing a primary key now land in the same batch, and hence in the same bundle, where a later commit can assert that they don't. Primary keys are suitable for this because they are UUIDs: all 19754063 of them across the 464 snapshots of `anvil15` are, so the prefixes keep the meaning they had when they were taken from `datarepo_row_id`. Since tables without a declared primary key are now expected rather than exceptional, `_pk_column` returns None for them instead of raising. The five callers that require a key wrap the call in `not_none` to say so, and the two that tolerate its absence go through `_batch_column`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A primary bundle's UUID has always been the UUID of its bundle entity with the version nibble changed. That entity was identified by its `datarepo_row_id`, so the bundle UUID wasn't stable across releases either. It is now derived from the entity's ID, preserving the relationship between the two while making both stable. Bundle UUIDs are the entity IDs of the `bundles` index, so they are user-visible. Deriving the UUID with `uuid5` can't be inverted, so `_bundle_entity`, which recovered the bundle entity's primary key from the UUID, no longer can. It doesn't need to: the FQID is constructed from that key, and carries it to `fetch_bundle`. Fetching a primary bundle therefore costs one BigQuery query less than it used to. Because the UUID is now a function of the FQID's other attributes, the FQID derives it itself, and asserts that a UUID passed to it agrees with that derivation. Deserializing an FQID is the only thing that passes one, whether from a notification or from the arguments of `can_bundle.py`, which gains a `--primary-key` option, no longer being able to identify a primary bundle by its UUID alone. The derivations moved out of the plugin to module scope, so that the FQID can reach them. With the row ID no longer involved, `datarepo_row_uuid_version` has no remaining use. What is left of `batch_uuid_version` is the version of any UUID we derive with `uuid5`, which is what the new derivation produces, too, hence `derived_uuid_version`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

An AnVIL bundle's UUID is derived from the table it is drawn from and either the batch or the bundle entity it contains, so requiring it as an argument meant deriving a v5 UUID by hand just to name the bundle one already knows. `--version` was worse: documented as ignored for AnVIL, it was in fact required, and omitting it failed with a `ValueError` that named neither the argument nor the script. Both are now optional and mutually exclusive with the AnVIL arguments, which is a constraint `argparse` can't express, hence the explicit check. Being able to give one of `--batch-prefix` and `--primary-key` and omit the other also retires the `"null"` sentinel that stood for the one that didn't apply. The script used to build an FQID by deserializing a dictionary of its arguments, but a serialized FQID is complete by definition, and `_from_json` type checks the UUID it requires. Constructing the FQID instead leaves it to the repository to derive what wasn't given. For AnVIL that includes the version, which every bundle and every entity in one shares. It was a property of the plugin, which is parameterized by the FQID class and constructs instances of it. Since the FQID now needs the version, too, in order to default it, the constant moved to module scope, where both can reach it without depending on each other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Entities from tables the schema describes are identified by their primary key, so two rows of a table that share one are indistinguishable to us. `add_entity` already rejected an entity it had seen before, but only among the entities, or only among the orphans, depending on where the row was headed. Two `anvil_file` rows that share a key but disagree on `is_supplementary` go to either side of that divide and used to slip through, leaving two replicas with the same ID but different content. Consulting both sides closes that, and makes the detection complete: partitioning, batching and the graph traversal are all keyed on the primary key now, so two rows sharing one always meet in the same bundle. Rows of `anvil_dataset` are the one exception, in that `_get_dataset` rejects a second one of those before it ever gets here. Whether such rows exist, and whether anything upstream prevents them, is not known. The most recent release has none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The relations in a verbatim PFB manifest were declared `MANY_TO_ONE` on the grounds that a primary key is unique within its table. It isn't, between snapshots: a manifest can span datasets whose tables of the same name share primary keys, in which case a relation's right-hand side matches more than one entity. Declaring the links `MANY_TO_MANY` makes the schema describe the data we actually hand over, instead of promising a cardinality it doesn't honour. It doesn't make the references unambiguous, which is what the FIXME is for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

coveralls · 2026-10-07T00:34:22Z

coverage: 85.034% (+0.04%) from 84.998% — issues/hannes-ucsc/8315-file-path into develop

codecov · 2026-10-07T00:46:05Z

Codecov Report

❌ Patch coverage is 97.16312% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.96%. Comparing base (c173ed2) to head (84b1794).
⚠️ Report is 20 commits behind head on develop.

Files with missing lines	Patch %	Lines
src/azul/plugins/repository/tdr_anvil/__init__.py	97.46%	2 Missing ⚠️
test/integration_test.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #8376      +/-   ##
===========================================
+ Coverage    84.92%   84.96%   +0.03%     
===========================================
  Files          170      170              
  Lines        25593    25681      +88     
===========================================
+ Hits         21734    21819      +85     
- Misses        3859     3862       +3

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

The new --issues option names the issues the PR links, the first of which determines its title. The option is independent of the PR type, and in its absence the issues are inferred from the name of the current branch, as before, for every type. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Version 6 adds the optional `anvil_file.file_path` column. The canned snapshot was ingested under version 5, whose `anvil_file` table lacks that column, so no unit test could exercise a populated value. One of the three canned files leaves the column null, because it is optional even in version 6. The UUID of a batched bundle is derived from the source spec, which contains the name of the snapshot, so the two canned batched bundles are renamed. The UUID of a primary bundle is derived from the primary key of its bundle entity, and is therefore unaffected. The replicas carry the column, and the PFB schema has declared it since #8242, so the verbatim manifests expose it without any change to the source. The ID of a replica document is a hash of its contents, so the two canned file replicas are re-keyed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The column is exposed in the compact manifest and in the response to the /index/files endpoint. The verbatim manifests already carried it for snapshots that were ingested under version 6 of the AnVIL schema. Rows from a snapshot that was ingested under version 5 lack the column, and are now treated as if it were null in them, in the index and the verbatim manifests alike. The column is neither faceted, sorted nor filtered on, so it is excluded from the aggregation of files into entities of other types, where it would bloat the aggregates just like the file name would. The minor version of the service API is incremented, because adding a field to a response and a column to a manifest requires no updates to clients. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

hannes-ucsc and others added 10 commits October 5, 2026 22:11

Align canned AnVIL dataset title with its snapshot name (#6844)

4d887c2

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Assert AnVIL dataset title against its snapshot name (#6844)

a7af744

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

hannes-ucsc self-assigned this Oct 7, 2026

hannes-ucsc linked an issue Oct 7, 2026 that may be closed by this pull request

Expose anvil_file.file_path column from v6 schema #8315

Open

hannes-ucsc added reindex:anvildev [process] PR requires reindexing anvildev reindex:anvilprod [process] PR requires reindexing anvilprod API API change affecting callers labels Oct 7, 2026

hannes-ucsc and others added 3 commits October 6, 2026 17:51

hannes-ucsc force-pushed the issues/hannes-ucsc/8315-file-path branch from 1fd4c4d to 84b1794 Compare October 7, 2026 00:51

Navigation Menu

Repository navigation

Expose anvil_file.file_path column from v6 schema (#8315) - #8376

Expose anvil_file.file_path column from v6 schema (#8315)#8376
hannes-ucsc wants to merge 13 commits into
developfrom
issues/hannes-ucsc/8315-file-path

hannes-ucsc commented Oct 7, 2026 •

edited

Loading

Uh oh!

coveralls commented Oct 7, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

hannes-ucsc commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Author

Author (partiality)

Author (reindex)

Author (mirror)

Author (API changes)

Author (upgrading deployments)

Author (hotfixes)

Author (before every review)

Peer reviewer (after approval)

System administrator (after approval)

Operator

Operator (deploy .shared and .gitlab components)

System administrator (post-deploy of .gitlab component)

Operator (deploy runner image)

Operator (sandbox build)

Operator (merge the branch)

Operator (main build)

Operator (reindex)

Operator (mirroring)

Operator

Shorthand for review comments

Uh oh!

coveralls commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hannes-ucsc commented Oct 7, 2026 •

edited

Loading

Operator (deploy `.shared` and `.gitlab` components)

System administrator (post-deploy of `.gitlab` component)

coveralls commented Oct 7, 2026 •

edited

Loading

codecov Bot commented Oct 7, 2026 •

edited

Loading