Skip to content

make asnmeta_updater use ReplicatedMergeTree - #185

Open
aagbsn wants to merge 2 commits into
mainfrom
add_replicated_asnmeta
Open

make asnmeta_updater use ReplicatedMergeTree#185
aagbsn wants to merge 2 commits into
mainfrom
add_replicated_asnmeta

Conversation

@aagbsn

@aagbsn aagbsn commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

This makes the airflow updater task use a replicated asnmeta table, for services connecting to other cluster nodes.

@aagbsn

aagbsn commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Manual migration needed before/alongside this PR

asnmeta currently exists as a plain, unreplicated MergeTree
independently on each of clickhouse1/2/3 (data1/2/3.htz-fsn.prod.ooni.nu):
data1 has the real, live data (325,908 rows, fed daily by the
oonipipeline Airflow DAG), data2 is empty, and data3 has a stale,
abandoned partial migration attempt already registered at the canonical
replicated path. This PR's CREATE TABLE IF NOT EXISTS is a no-op
wherever asnmeta already exists (i.e. everywhere right now) -- it will
not fix the current state on its own. The steps below need to run first
(or in the same change window).

End state must use the exact zookeeper path this PR's code uses
(/clickhouse/{cluster}/tables/ooni/asnmeta/{shard}), so a future
from-scratch bootstrap and this migration produce identical state. That
path is currently occupied by data3's stale attempt, so step 1 clears it
-- scoped to data3 alone, never ON CLUSTER, since asnmeta is also
the name of data1's real, live table and ON CLUSTER would drop that
too.

Zero-downtime by construction: the live asnmeta name on data1 (and
data2) is never dropped or left empty. Only data3's already-empty,
already-broken copy is touched directly; everything else is built and
verified under a separate name, and only the final EXCHANGE (a fast
catalog-level rename, not a data copy) touches the real asnmeta name on
data1/data2.

1. data3 only -- do not run ON CLUSTER, do not run this against data1 or data2. Connect directly to data3.htz-fsn.prod.ooni.nu:

DROP TABLE ooni.asnmeta;

CREATE TABLE ooni.asnmeta
(
    `asn` UInt32, `org_name` String, `cc` String,
    `changed` Date, `aut_name` String, `source` String
) ENGINE = MergeTree
ORDER BY (asn, changed);

The DROP removes only data3's local catalog entry and deregisters its
stale replica from the canonical zookeeper path, freeing it. The CREATE
that follows is a throwaway, non-replicated, empty local placeholder -- it
exists only so step 5's cluster-wide EXCHANGE has something to swap
against on this node too, matching what data1 (real data) and data2
(empty table) already independently have. Neither statement here uses ON CLUSTER, so both are scoped to data3's own catalog only.

2. Create the new table under a temporary name, ON CLUSTER, using the canonical path -- safe now that step 1 freed it on data3 (it was never occupied on data1/data2, since their existing asnmeta tables are plain, unreplicated, and not registered in zookeeper at all):

CREATE TABLE ooni.asnmeta_v2 ON CLUSTER oonidata_cluster
(
    `asn` UInt32, `org_name` String, `cc` String,
    `changed` Date, `aut_name` String, `source` String
)
ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/ooni/asnmeta/{shard}', '{replica}')
ORDER BY (asn, changed)

3. Backfill from the live table, connected to data1 specifically (the only node with real data):

INSERT INTO ooni.asnmeta_v2 SELECT * FROM ooni.asnmeta

4. Verify row counts agree on all three nodes before proceeding (connect to each individually):

SELECT hostName(), count() FROM ooni.asnmeta_v2

5. Atomic cutover, ON CLUSTER, all three nodes at once:

EXCHANGE TABLES ooni.asnmeta AND ooni.asnmeta_v2 ON CLUSTER oonidata_cluster

6. Drop the old copies (data1's now-redundant original, data2's empty table, data3's step-1 placeholder -- all safe, none hold anything not now in asnmeta):

DROP TABLE ooni.asnmeta_v2 ON CLUSTER oonidata_cluster

7. Deploy this PR. asnmeta_tmp doesn't exist anywhere yet, so its own
CREATE TABLE IF NOT EXISTS ... ON CLUSTER creates it cleanly on first
run; asnmeta is already fixed by steps 1-6, at the same canonical path
the code itself assumes, so that same check is a correct no-op from here
on.

this is needed to test the updated asnmeta updater, so that the ON
CLUSTER part of commands work
@codecov

codecov Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.87%. Comparing base (efb0f1d) to head (1468689).

❌ Your project check has failed because the head coverage (77.87%) is below the target coverage (80.00%). You can increase the head coverage or adjust the target coverage.

❗ There is a different number of reports uploaded between BASE (efb0f1d) and HEAD (1468689). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (efb0f1d) HEAD (1468689)
oonipipeline 1 0
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #185      +/-   ##
==========================================
- Coverage   83.65%   77.87%   -5.79%     
==========================================
  Files          89       33      -56     
  Lines        5917     1939    -3978     
==========================================
- Hits         4950     1510    -3440     
+ Misses        967      429     -538     
Flag Coverage Δ
oonidata 77.87% <ø> (ø)
oonipipeline ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant