make asnmeta_updater use ReplicatedMergeTree - #185
Conversation
Manual migration needed before/alongside this PR
End state must use the exact zookeeper path this PR's code uses Zero-downtime by construction: the live 1. DROP TABLE ooni.asnmeta;
CREATE TABLE ooni.asnmeta
(
`asn` UInt32, `org_name` String, `cc` String,
`changed` Date, `aut_name` String, `source` String
) ENGINE = MergeTree
ORDER BY (asn, changed);The 2. Create the new table under a temporary name, CREATE TABLE ooni.asnmeta_v2 ON CLUSTER oonidata_cluster
(
`asn` UInt32, `org_name` String, `cc` String,
`changed` Date, `aut_name` String, `source` String
)
ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/ooni/asnmeta/{shard}', '{replica}')
ORDER BY (asn, changed)3. Backfill from the live table, connected to INSERT INTO ooni.asnmeta_v2 SELECT * FROM ooni.asnmeta4. Verify row counts agree on all three nodes before proceeding (connect to each individually): SELECT hostName(), count() FROM ooni.asnmeta_v25. Atomic cutover, EXCHANGE TABLES ooni.asnmeta AND ooni.asnmeta_v2 ON CLUSTER oonidata_cluster6. Drop the old copies ( DROP TABLE ooni.asnmeta_v2 ON CLUSTER oonidata_cluster7. Deploy this PR. |
this is needed to test the updated asnmeta updater, so that the ON CLUSTER part of commands work
Codecov Report✅ All modified and coverable lines are covered by tests. ❌ Your project check has failed because the head coverage (77.87%) is below the target coverage (80.00%). You can increase the head coverage or adjust the target coverage.
Additional details and impacted files@@ Coverage Diff @@
## main #185 +/- ##
==========================================
- Coverage 83.65% 77.87% -5.79%
==========================================
Files 89 33 -56
Lines 5917 1939 -3978
==========================================
- Hits 4950 1510 -3440
+ Misses 967 429 -538
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This makes the airflow updater task use a replicated asnmeta table, for services connecting to other cluster nodes.