Conversation
Every frame of a multi-frame image (e.g. TIFF) is decoded to RGB, and hi_res holds all frames in memory at once, but blank frames compress to almost nothing: a ~195 KB TIFF of 30 blank 10,000 x 10,000 frames decodes to ~9 GB of RGB. PIL's per-frame limit does not bound the total. Read each frame's size from its header in partition_pdf_or_image() before any strategy runs, without decoding pixels, and raise UnprocessableEntityError once the frames exceed IMAGE_MAX_TOTAL_PIXELS in total (default 500,000,000, matching the per-frame limit pdf.py sets). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
PIL raises DecompressionBombError at open or seek for a single frame over twice MAX_IMAGE_PIXELS, before the frame loop can total the pixels, so the caller got PIL's error instead of UnprocessableEntityError. Translate it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review follow-ups: - ImageSequence.Iterator calls each format's seek(), which allocates the frame (pi-heif) or decodes it (APNG, GIF) before its size can be charged, so the guard could allocate a large secondary frame before rejecting it, and ocr_only, which uses only the first frame, now processed every frame. Iterate frames only for TIFF, whose seek() just reads the frame's IFD; for other formats charge the first frame, whose size Image.open() reads from the header. - Apply the guard after the strategy is resolved: only hi_res, which decodes every frame, is charged for all of a TIFF's frames. - Cover: no pixel allocation or frame load while measuring (TIFF and APNG), mixed frame sizes, stopping at the frame that passes the limit, a rejected borrowed file left open at its position, the hi_res and ocr_only dispatch, and PDFs bypassing the image limit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…coding hi_res decodes every frame of any multi-frame image, but the guard charged only the first frame of formats other than TIFF, so many frames under the limit could add up past it. Every frame is now charged for hi_res, each measured without allocating or decoding it: - TIFF and MPO frames, which can differ in size, from each frame's header (their seek() is metadata-only); - HEIF images from the sizes pi-heif reads from the container on open (its seek() allocates the frame, so it is not used); - canvas-composited frames (APNG, animated WebP, GIF) as canvas size times the frame count Pillow reads from metadata. Covered for each format with an instrumented allocator, using a small two-image HEIF fixture since pi-heif cannot encode. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Pillow accepts a bare int for an RGB putpixel() (setting only the red band), but a tuple states the intent and does not rely on that leniency. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rame The guard charged canvas-composited formats by repeating the canvas size n_frames times, one addition per frame. Pillow takes an APNG's n_frames straight from its acTL chunk, which may declare up to 2^31 frames, so a few-hundred-byte 1x1 APNG cost 500 million iterations before rejection. Measure frames as runs of same-sized frames: TIFF, MPO and HEIF yield one run per frame physically in the file, and canvas formats one run of n_frames, compared against the limit in constant time. The error still names the frame that takes the total past the limit. Also cover: auto and the dependency fallbacks resolving which frames are charged, and a borrowed file left open at its position when reading a frame header fails. Document that the limit bounds decoded pixels, not allocations inside native decoders or process-wide memory. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
# Conflicts: # CHANGELOG.md # unstructured/__version__.py # unstructured/partition/utils/config.py
# Conflicts: # CHANGELOG.md # unstructured/__version__.py # unstructured/partition/utils/config.py
A GIF frame that extends past the logical screen grows the canvas Pillow decodes it and every later frame onto, but Pillow's frame count does not apply that growth, so the pixel limit charged every frame at the first frame's size and a small first frame followed by a large one was undercounted. Read each frame descriptor's position and size from the file instead, skipping color tables and image data by their lengths and stray bytes as Pillow does, without decoding anything; the running canvas size matches the size Pillow decodes each frame at. Also cover: - a frame-header failure after one frame was measured, through partition_image: no partitioner, OCR or conversion runs, the original exception escapes and the borrowed stream stays open at its position; - ocr_only, requested or as hi_res's fallback, never reaching a later frame or a declared frame count (TIFF, APNG, GIF), with a hi_res control showing the same poison trips when frames are read; - a deterministic work count for a forged APNG frame count: one run. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…-followups # Conflicts: # CHANGELOG.md # test_unstructured/partition/pdf_image/test_image.py # unstructured/__version__.py # unstructured/partition/pdf.py
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 4 files
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
Image.open() seeks a stream to offset 0 whatever its position, so the image it measures and partitioning later decodes start there. The frame descriptor walker started at the caller's position instead, so a borrowed stream positioned past 0 was parsed misaligned and its frames mismeasured. Read from 0; the caller's position is still restored afterwards. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
0 issues found across 2 files (changes from recent commits).
Shadow auto-approve: would not auto-approve. This PR does not meet the repository auto-approval settings.
Re-trigger cubic
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-ups to the image pixel limit added in #4520.
n_frames) does not apply that growth, so the limit charged every frame at the first frame's size, and a small first frame followed by a large one was undercounted.hi_res, a GIF's frames are now measured by_iter_gif_canvas_sizes(), which reads each frame descriptor's position and size from the file.Verification
Tests
partition_image. No partitioner, OCR or conversion runs, the original exception escapes, and the borrowed stream stays open at its position.unstructured_inferenceis missing, ocr_only never reaches a later frame or a declared frame count, for TIFF, APNG and GIF. A hi_res control shows the same poison trips when frames are read.acTL: one run, not one per declared frame.🤖 Generated with Claude Code