Add Ravel - #1887
Conversation
|
|
GitHub Actions
Error
0723092 to
5ca4e3d
Compare
GitHub Actions
Error
|
This makes sense. We can start with a local instance of Minio or SeaweedFS, so the setup will be comparable with other systems. Then we can extend the benchmark to support systems on real S3 (but currently it isn't supported). |
Ravel is an object-storage-native telemetry database queried over SQL; every durable byte lives in S3-compatible object storage and the local disk holds only the downloaded Parquet file and a read cache. Two results on c6a.4xlarge, both true cold: the server is restarted and the page cache dropped before the first run of every query, 3 runs per query. The stock entry runs the server with no performance flags. 42 of 43 queries complete; q33 exceeds the per-query memory pool the server derives on a 30 GB machine and is recorded as null rather than run with a raised limit. Its warm runs are close to its cold runs: the server holds no data on local disk, and its derived read cache (25% of memory, 8.2 GB) is smaller than this dataset (11.7 GB), so every warm run re-reads from object storage. The tuned entry passes four flags through RAVEL_TUNED_ARGS in ./start: latency-first fetch policy, fetch concurrency 256, a 12 GiB per-query pool, and a 26.3 GB read cache that the dataset fits in. All 43 queries complete. The cache flag changes the tuned totals by about 1%, because latency-first reads ranged blocks whose working set already fits the default cache; it is kept so the entry's cache behaviour does not depend on the dataset size. The entry is marked tuned and sits out of competition; the README states each flag, why the larger cache is safe under this benchmark's one-query-at-a-time protocol but is not the default, and the trade the entry accepts, about 5.3x the object-store GET requests of the stock entry for the shorter cold wall-clock. Load time is the shipped recipe at defaults and was measured separately from the query runs, since the load is not idempotent against an already loaded bucket. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
5ca4e3d to
6af8e3a
Compare
GitHub Actions
Error
The maintainer asked for a local S3-compatible store so the entry is set up like the other local-disk systems here, with real S3 left for a later extension of the benchmark. This commit makes the entry self-contained: ./install downloads a pinned MinIO release, verifies it against its published checksum, generates a credential pair into a file readable by the current user only, starts MinIO on loopback with its data directory on the machine's disk, creates the bucket and qualifies it. A new ./minio-start is idempotent and is also called from ./start, so a MinIO that died between queries is restarted rather than failing the rest of the run. ./stop stops the database only. Every script uses static credentials read from that file through the environment; nothing carries a key on a command line, and a non-loopback RAVEL_S3_ENDPOINT skips the MinIO setup so the same scripts run against another S3-compatible store. Both results are re-measured on the MinIO setup, on the same c6a.4xlarge and the same true-cold protocol as before (server restarted and page cache dropped before the first run of every query). Load time 1,203 s. Stock: cold 1,701 s, hot 184 s, q33 refused by the derived per-query pool as before. Tuned: cold 727 s, hot 82 s, all 43 statements, with the read cache's local-disk tier on (RAVEL_CACHE_DIR) in addition to the four flags; the tier survives the restart before each cold run, and the README states that this is part of what the tuned figures measure. Without the tier the same flags measure 1,222 s cold on this setup. The cold figures are the 500 GB gp2 volume's: the stock policy reads the whole 11.2 GB corpus per statement at the volume's 256 MB/s, and MinIO serves a ranged read by reading every 1 MiB block it touches, so the tuned policy saves wire bytes but fewer disk bytes. The README says so, and keeps the earlier real-S3 figures as a reference table, since the harness cannot reproduce them. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
GitHub Actions
Error
|
Done: the entry now runs against a local single-node MinIO, in the second commit on this branch.
Both results are re-measured on that setup, same c6a.4xlarge, same true-cold protocol (server restarted and page cache dropped before the first run of every query). Load time 1,203 s.
For context on the cold numbers: Ravel keeps no data on local disk, so every cold run reads from MinIO, which reads from the 500 GB gp2 volume at about 256 MB/s. The stock policy reads the whole 11.2 GB corpus per query, which is about 44 s per query at that rate whatever the query computes. MinIO also serves a ranged GET by reading every 1 MiB block the range touches, which is why the tuned policy's byte savings show up on the wire more than on disk without the tier. The earlier real-S3 figures stay in the README as a reference table only, not as a results file. |
The reference table described the real-S3 measurements as the same corpus. They were the same rows, but a tenant loaded a few days earlier by an earlier build: 12.2 GB rather than 11.2 GB, and its hot runs still re-fetched about 500 MB of column statistics per query. The README now says so, so the table is not read as a like-for-like comparison with the MinIO results. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
|
Results for Logs:
|
GitHub Actions
Error
The entry installed the previous release, while the figures in the README and in this pull request were taken on code that had not shipped yet. v0.15.0 is published now, so ./install pins it and the next collector run measures the released binaries. That server records query text on its audit trail tokenized under a key, and refuses to start unkeyed without one. ./install generates a key into the same private credentials file as the MinIO keys, and ravel-env.sh exports it. The tuned result file this branch carried is gone, since the collector removes hand-added results for a system it benchmarks. The README now presents the tuned flags as a configuration rather than a published result, and its reference table says which build each row was measured on: stock 1,702 s cold and 184 s hot, tuned with the disk tier 728 s and 83 s, load 1,224 s, all on the released binaries. Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
GitHub Actions
Waiting
|
Pinned the entry to Ravel v0.15.0, which is now published, in the commit above. Two smaller things in the same commit. The 0.15.0 server records query text on On our own machine the released binaries measure stock 1,702 s cold and 184 s |
4 participants
Ravel is an object-storage-native telemetry database queried over SQL (and other DSLs).
Every durable byte lives in S3-compatible object storage, no WAL.
Two results on c6a.4xlarge, both true cold: the server is restarted and the page cache dropped before the first run of every query, 3 runs per query. The local disk holds only the downloaded Parquet file and a read cache.
The stock entry runs the server with no performance flags. 42 of 43 queries complete; q33 exceeds the per-query memory pool the server derives on a 30 GB machine and is recorded as null rather than run with a raised limit.
The tuned entry passes four flags through RAVEL_TUNED_ARGS in ./start: latency-first fetch policy, fetch concurrency 256, a 12 GiB per-query pool, and a 26.3 GB read cache (80% of this machine's memory) so the dataset stays resident between the three runs of a query. All 43 queries complete. It is marked tuned and sits out of competition; the README states the flags and the trade they accept, about 5.3x the object-store GET requests of the stock entry for the shorter cold wall-clock.
We're working to improve the price-perf of that mode, as well as providing adaptive modes and dynamic query-cost budgets. But it's safe to say that there always going to be a "give me best latency, no matter what, in that architecture" mode. This is that mode right now.
Load time is the shipped recipe at defaults and was measured separately from the query runs, since the load is not idempotent against an already loaded bucket. It reliably measures in the same exact w/ <+-0.5% deviation with fresh buckets.
What's different with Ravel in the context of ClickBench is that it requires an S3 bucket to ingest to (and read from).
Is that something you can facilitate @alexey-milovidov?
Happy to follow any route you feel appropriate.