[BUG] Cloud packaging intermittently rejects unchanged Windows files

[Neonsy]

Summary

strix cloud scans start --source ... can report that a file changed while its archive was built even when no process changed the file content

The false rejection happens before upload and is intermittent on Windows

Evidence from Windows

The problem was reproduced on Windows 11 at Strix commit f6d9790ecbc8f25739a94a22a42f430a959af737

• Packaging rejected 3 of 20 newly created unchanged source directories through prepare_source
• An instrumented 100-file run found 7 cases where only st_ctime_ns changed between selection and opening
• File size, device, inode, and st_mtime_ns remained equal in those cases

Why ctime is unreliable here

SelectedFile records st_ctime_ns

_write_archive requires that value to remain unchanged before and after the file is opened

That metadata value can change on Windows without a content change

Desired behavior

Strix should reject a file that is replaced or modified during packaging without rejecting an unchanged file because its Windows metadata settled after creation

The replacement check still needs to protect against content changes and file swaps on every platform

The Windows regression test should repeat the operation enough times to catch the intermittent failure rather than relying on one timing-sensitive attempt

[itzzdev09]

I hit this while triaging the Windows test failures in #1259 and instrumented the guard, so here is the concrete mechanism rather than a reading of the code.

Reproduction

Windows 11, Python 3.12, main @ 95e085e. Running tests/test_cloud_source_upload.py repeatedly, the failing set changes every run — this is exactly the "intermittent" in the title:

run 1: test_source_digest_approval_rejects_a_changed_snapshot, test_ambiguous_scan_launch_retains_completed_source_upload[server]
run 2: ...[network], ...[malformed_json_success], test_failed_automatic_upload_cleanup_reports_retained_id[server], test_oversize_archive_names_largest_files_and_exclude_guidance
run 3: test_source_upload_is_completed_and_attached_to_scan

Which field actually drifts

I temporarily replaced the CloudError in source_upload.py:608 with one that reports the mismatching fields. Across four runs, by far the dominant case is st_ctime_ns alone:

DIFF=['ctime:1789184729803815100->1789184729829390800']   (+25.6 ms)
DIFF=['ctime:1789184734800061300->1789184734801061600']   (+1.0 ms)
DIFF=['ctime:1789184736686980900->1789184736687985300']   (+1.0 ms)

Nothing writes to the file between the two stats. The drift is ~1 ms to ~30 ms.

A second, rarer shape shows st_ino differing entirely along with the timestamps, which is a genuinely different file — parametrized cases reusing a temp path, not a tamper:

DIFF=['ino:1407374885584144->1688849862294801', 'mtime:...', 'ctime:...']

Why

The guard compares os.stat(path) captured at selection time against os.fstat(fd) taken later when the archive is built. On Windows those go through different APIs — the path stat can be served from cached directory-entry metadata, while fstat uses GetFileInformationByHandle — and NTFS settles that metadata asynchronously. So the two disagree for a short window after the file is touched, with no modification involved.

Worth noting the two calls are not inherently inconsistent — back to back on a quiescent file they agree exactly:

st_size   stat=16                     fstat=16                     SAME
st_dev    stat=6970059491574283810    fstat=6970059491574283810    SAME
st_ino    stat=27584547717759075      fstat=27584547717759075      SAME
st_mtime_ns stat=1789184687061573300  fstat=1789184687061573300    SAME
st_ctime_ns stat=1789184687061573300  fstat=1789184687061573300    SAME

It is the elapsed time and the pending metadata flush that create the mismatch, which is why it is intermittent rather than deterministic.

Suggestion

st_ctime_ns is the weak link. On Linux it is inode-change time and a real tamper signal; on Windows Python maps it to a value that does not hold still for a freshly written file, so including it makes the check unsound there without adding security. st_dev + st_ino + st_size + st_mtime_ns still catch replacement and truncation, and the existing post-read final comparison plus the trailing source_file.read(1) still catch a file that grew during the read.

So: drop st_ctime_ns from both comparisons on Windows (os.name == "nt"), or gate the whole timestamp pair behind a platform check and keep identity plus size everywhere.

Happy to put up a PR for that if you'd like — I left it out of #1297 deliberately, since that one is scoped to test assumptions and this is a product bug.

[Kotsur69]

Hit this in real-world use (strix cloud scans start --source ... against a ~200-file Next.js app), not just the test suite - adding a data point since it's more severe than the 1-30ms transient described above.

On a completely static, several-days-old, untouched file (verified via repeated stat polling - no editor, no watcher, no AV activity during the window), Path.lstat() and os.fstat(fd) disagreed on st_ctime_ns by ~7.4 minutes:

lstat():  ctime_ns=1789027534152025400   (matches the file's actual NTFS creation time)
fstat():  ctime_ns=1789469112660874700   (exactly equals st_mtime_ns, both readings)

Size, dev, ino, and mtime were identical between both stats. This reproduced 100% deterministically across 15+ retries, on a fresh isolated copy in a separate temp directory with no other process touching it, and did not resolve after waiting.

So beyond the intermittent 1-30ms async-settle case already described, there's a second, more severe shape: any file whose Windows creation time differs from its last-write time by more than a trivial amount - i.e. almost any ordinary file in a real git checkout - fails this check permanently, not just in a narrow post-write race window. Both open fixes (#1298, #1300) already handle this correctly since they drop st_ctime_ns on Windows entirely rather than trying to tolerate a small drift window.

Small FYI since both look functionally equivalent: #1298 and #1300 appear to be duplicate fixes for this - might be worth picking one to avoid a merge conflict.

Happy to test either PR against this repro if useful.

[itzzdev09]

@Kotsur69 thanks, that data point was the missing piece, and it corrects my earlier explanation.

Your fstat ctime being exactly equal to st_mtime_ns is the giveaway. On Windows, CPython fills st_ctime_ns from different calls depending on how the file is stat'ed:

• a path stat (lstat/stat) gives the creation time;
• os.fstat() on a handle gives the last metadata change.

The guard pins files with lstat at selection and checks them with fstat while archiving, so it compares two different timestamps. They only agree for a file that hasn't changed since it was created. That's your permanent failure on an ordinary checkout, and the "intermittent" test failures are the same thing, where a file written and archived within milliseconds may or may not land on the same tick. It isn't NTFS settling asynchronously, as I suggested above.

I measured it on 3,000 fresh files: lstat ctime vs fstat ctime disagreed for 1,091 of them. fstat vs fstat disagreed for none, whether on the same handle or on two handles opened 0.5 s apart. fstat's ctime also moves on a chmod or a rename, just like POSIX ctime.

So I've updated #1300 to pin files with the handle's fstat at selection too, rather than dropping st_ctime_ns on Windows. The comparison is then like with like on every platform, and a metadata-only change such as a chmod or a rename back and forth is still caught on Windows instead of being ignored. The details and tests are in the PR description.

If you're still up for testing it against your repro, that would be great: strix cloud scans start --source ... on the Next.js app should now go through.