[WIP][POC] Pfor encoding - #579
Draft
prtkgaur wants to merge 5 commits into
Draft
Conversation
PFOR is an integer compression encoding (encoding number 11) for INT32 and INT64 columns. It compresses by subtracting the minimum value (FOR), selecting an optimal bit width via a histogram-based cost model, bit-packing the deltas, and storing outlier values as exceptions with their positions. Adds the full encoding specification in Encodings.md and the PFOR = 11 enum entry in parquet.thrift.
Remove stray colon from "Patched Frame of Reference: (PFOR = 11)" to be consistent with other headings like "Delta Encoding (DELTA_BINARY_PACKED = 5)".
Contributor
|
What are your thoughts about using FastLanes https://15721.courses.cs.cmu.edu/spring2024/papers/03-data2/p2132-afroozeh.pdf (which has frame of reference in addition to several other schemes, and is reported to have very good eprformance) |
Encodings.md now carries the same short entry for PFOR that the other encodings have -- the supported types, what the encoding does, and a pointer to the details -- while the page layout, the encoding and decoding procedures, the worked examples and the constants move to PforEncoding.md. This matches how ALP is split between Encodings.md and AlpEncoding.md, and it keeps Encodings.md readable as a survey of the encodings. The moved text is unchanged apart from heading depth. Two links into Encodings.md are now written relative to it, which fixes one of them along the way: the reference to DELTA_BINARY_PACKED used the anchor #DELTA, and the anchor defined in that file is #DELTAENC.
The specification said the frame of reference is the minimum value in the vector and that all deltas are therefore non-negative. That forbids what a writer should do on a tight cluster with a few low outliers, where the minimum forces a bit width wide enough to reach the far outlier and exceptions cannot help, because with the frame at the minimum only values above the packed window ever exceed it. The frame is now any value of the column's type, chosen by the writer. A value below it wraps under the modular subtraction to a delta too large for bit_width, which is the same unsigned test a value above the window fails, so it is patched like any other exception and the stored exception value is the original one. No field changes and no new mechanism: the frame already travels in the vector info at full width, and the decode steps add it before overwriting the exception positions, so they reconstruct either choice. Two writers may pick different frames for the same input and every reader decodes both pages to the same values. The text also does not prescribe a search. Costing a frame candidate is the expression the cost model already applies to the deltas that candidate produces, so the frame and the width are chosen together, but how a writer enumerates candidates is left to it. Example 3's arithmetic was inconsistent with its own conclusion: it put the frame on the null sentinel and still claimed 11 bits, which only holds for a frame on the cluster. It now names that frame and the deltas it produces.
The short entry still described PFOR as subtracting the minimum value, which the specification body no longer requires. It now names the frame and says what the freedom buys: outliers on either side of the packed window are patched, not just the ones above it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
What changes are included in this PR?
Do these changes have PoC implementations?