[19.0][ADD] data_import_log_fs - #3745
Open
AungKoKoLin1997 wants to merge 1 commit into
Open
AungKoKoLin1997 wants to merge 1 commit into
AungKoKoLin1997 wants to merge 1 commit into
Conversation
AungKoKoLin1997
force-pushed
the
19.0-add-data_import_log_fs
branch
2 times, most recently
from
September 28, 2026 09:40
b70b237 to
d7c1a25
Compare
Pick import files up from a filesystem storage and log them, so that a feed of files becomes a feed of import logs without either side knowing about the other: data_import_log stays independent of where a file is stored, and fs_storage of what the file is for. A pick-up definition holds the storage, the directories a file travels through and the pattern its files are named by. The scan skips the files still being written, which a sender names with a suffix until it has finished, and recognizes a file it has already imported in full by the hash of its content rather than by its name alone. A file is held aside while it is imported, so a scan running before that import has finished neither takes it again nor overwrites it, and is released only when the log is closed. A file that was read is filed as done whatever became of its units, with the rejected ones written back on their own for the sender to correct; a file that could not be read is isolated whole. Assisted-by: Claude Opus 5
AungKoKoLin1997
force-pushed
the
19.0-add-data_import_log_fs
branch
from
September 29, 2026 03:27
d7c1a25 to
4879dc8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This module picks import files up from a filesystem storage and logs them.
A pick-up definition is a storage backend, the directories a file travels
through, the pattern the files are named by and the layout of their columns. A
scheduled scan lists the incoming directory, skips the files still being
written, creates one import log per file with the file attached, and holds the
file aside while it is imported, so that a later scan neither takes it again
nor overwrites it.
The file is released when its log closes. One that was read is filed as done
whatever became of its units, and the units that were rejected are written back
to the error directory on their own, for the sender to correct and resend. One
that could not be read is moved to the error directory whole. A file whose
earlier import succeeded in full is left where it is.
It relies on
data_import_logfor the log and onfs_storagefor thetransport, which is any filesystem fsspec supports — sftp, S3, or a local
directory.
Depends on #3744
Assisted-by: Claude Opus 5
@qrtl QT7028