PE: hold a metadata region to the image the section table describes

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Problem

CFGFast returns an empty model for a Windows image whose entry point decodes on the
first try. Two 32-bit samples enter at 0x4bf000 and 0x10001f71, both inside a section
cle reports executable. project.factory.block(proj.entry) lifts 14 instructions at the
first and 4 at the second, each ending in a call, and the CFG is still empty:

>>> proj = angr.Project(OBJECT, auto_load_libs=False)      # sha256 7b00ec19...
>>> b = proj.factory.block(proj.entry); b.instructions, b.vex.jumpkind
(14, 'Ijk_Call')
>>> cfg = proj.analyses.CFGFast(normalize=True, data_references=False)
>>> len(cfg.functions), len(list(cfg.model.nodes())), cfg.graph.number_of_edges()
(0, 0, 0)

No function means no decompilation, no calling conventions and no cross references.

Root cause

PE._meta_load_config builds pointer arrays out of the load-config directory's own fields --
the four guard tables, and on a 32-bit image the SE handler table as well. Each is built when
its address and count are both non-zero, which rejects an empty table and nothing else:

PointerArray(vaddr=lc.SEHandlerTable, entry_size=4, count=lc.SEHandlerCount, ...)
PointerArray(vaddr=lc.GuardCFFunctionTable, entry_size=4, count=lc.GuardCFFunctionCount, ...)

Where the load-config is not a load-config -- a packer wrote over it, the file ends inside it --
pefile hands back whatever bytes are there and those become regions. On 7b00ec19... all five
land outside the object, which spans 0x400000-0x4c2bff: one at 0x61696c41 (the ASCII of
Alia), one at 0x6564 declaring 6,796,609,364 bytes.

The numbers do not stay in the loader. angr's CFGFast._process_metadata_regions marks
every metadata region as data in its segment list, so one region spanning the image leaves
no address code: _generate_cfgnode takes its block size from
_seg_list.next_pos_with_sort_not_in(addr, {"code"}), that returns the address itself, and
every lift is asked for 0 bytes. Measured on both samples, the segment list reports sort
unknown at the entry and every traced _lift call asks for size=0. One junk directory
costs the whole image.

Fix

PE._clip_meta_regions holds each region and sub-region to the image: a region that starts
outside the object is dropped, one that starts inside and runs past the end is shortened, and
PointerArray and StructArray keep count consistent with the new size. It runs after
_register_sections because the extent is max_addr, read off the sections and cached on
first access.

Shortening rather than dropping is the whole of the difference on a well-formed image.
tests/i386/windows/9f2ef84bde1e4ef445708cc5a605a09226363d502b1f5b5bf4a1cfc6dd5fc41e
declares 29,696 bytes of resource directory where the image has 4,096 left. Shortened, its
CFG is the one master already recovers, function for function: with angr at 3f5717544, 197
functions, 1,278 blocks and 1,862 edges on both sides, the same function addresses. Dropped, it
becomes 199 functions, and the two extra are at 0x409204 and 0x409243, inside the resource
span the dropped region would have left unmarked.

Testing

tests/test_pe_meta_regions.py::TestPEMetaRegionsHeldToTheImage asserts that no region or
sub-region of that tracked image leaves it, and that its resource directory is shortened to
0x1000 rather than dropped. Both fail on master: the first on the containment assertion,
the second on assert 29696 == 4096.

$ nix/run.sh --feature fix-c-328 -- pytest --import-mode=append -q tests/test_pe_meta_regions.py tests/test_pe.py
41 passed
$ nix/run.sh --feature fix-c-328 -- python3 -P MEASURE.py after5
$ nix/run.sh --feature fix-c-328 -- python3 -P MEASURE.py before3

MEASURE.py loads each object in its own process and runs the CFGFast call above; its argument
names the output file. The two arms are two builds of cle, pe.py reverted to the base and then
restored, and in each build the installed pe.py is byte-identical to that revision's blob while
every other family package hashes the same.

On the two samples, CFGFast goes from 0 functions to 144 on the one above (290 blocks,
367 edges) and to 59 on the other (531 blocks, 827 edges); 5 out-of-range regions become 0
on each. The samples are live malware and cannot be shared, so they are identified by
SHA-256: 7b00ec194518b62bc726966c0a45c3d992736cf8e75f4d12cf4bd3842bd90aaf is the one that
enters at 0x4bf000, and
3d7fe603def5bc06941033d5e687a990f423947d8d88fe5396f9d359b8930413 enters at 0x10001f71.
Over all 123 files angr/binaries tracks at 67c892ca8be2 whose header is a PE -- found by
magic, not by directory -- the clip shortens one region on one object, drops none, leaves none
outside its image and creates no zero-size region. The regression set is 88 objects -- 80 PE
samples from the corpus that already recovered functions, plus 8 of those tracked files -- and
not one function, block or edge changes on any of them. Counts are in the validation record.

Two other open changes in this repository bound a metadata region and neither subsumes this one.
#834 narrows a region that is oversized inside the image and touches only _meta_imports; it
says as much itself, that the load-config sub-tables and the resource directory Size "each
wants its own change with its own reproducer". #869 bounds the resource directory to the mapped
section that contains it, which is tighter than the image bound here and leaves the out-of-image
pointer arrays untouched. All three append their regression to
tests/test_pe_meta_regions.py at the same anchor, so whichever lands after another needs a
rebase in that one file; merge-tree puts each of the other two into master cleanly.

Validation: #862 (comment)

session: sharpen

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Validation record for head 23e28a704713b8b4f784eb645b655d254a3accf0 against baseline 5d5929e29c22587f0338b2d55f7193a7098326eb.

• Regression: nix/run.sh --feature fix-c-328 -- pytest --import-mode=append -q tests/test_pe_meta_regions.py — 21 passed on this head; on the baseline build the two new tests fail (assert 29696 == 4096, and the containment assertion on the same image) and the other 19 pass
• Focused: nix/run.sh --feature fix-c-328 -- pytest --import-mode=append -q tests/test_pe_meta_regions.py tests/test_pe.py — 41 passed
• Suites inside the gate run below: cle 337 passed and 9 skipped, angr 3,565 passed, 47 skipped, 2 xfailed and 1,203 subtests passed
• Lint/type: nix/run.sh --feature fix-c-328 -- python3 -P .../run-ci-diff-checks.py --repository cle — pylint 10.00 -> 10.00 on both changed files; pyright 55 -> 55 in cle/backends/pe/pe.py and 0 -> 0 in the test
• Workspace gate: ./feature.sh test fix-c-328 at this head, which the gate's own environment record shows built cle from this worktree at 23e28a704 clean — 13 of 14 suites ran and all 13 passed, and angr-management did not run because this collection has not adopted it. The gate's own line about that: a green gate that skipped a suite is green over less than it appears to be

Corpus measurement, one environment per side, each built through the collection from the trees
below, every object in its own process through the harness script the description names:

Objects	Baseline	Head
2 cited samples, functions	0	203
80 corpus PE objects, functions	127054	127054
8 sampled tracked PE files, functions	2695	2695

• Regression set: 80 corpus PE objects plus 8 files from angr/binaries. The 80 are the first 80 of the 82 that a stride-65 slice of sorted SHA-256 yields over the 5,274 rows of one epoch's ledger with load.backend == "PE", cfg.functions >= 1 and wall_s <= 60; the two it leaves out are fc8a2cd12a163004ca07e89af75bee21989ff1e11af119623ff3b6acdcabf6a0 and ffb00825bca49f3a590c8174cc1988b61f87b0e3d9b6a5fa3ce7913d349a1a38. The 8 are a hand-picked subset of angr/binaries' Windows files, chosen to cover the common shapes and to include the one this change's census flagged
• Whole tracked population, measured at this head against angr/binaries 67c892ca8be2: of the 2,150 files it tracks, 123 have a PE header, found by magic rather than by directory, and all 123 load with cle's PE backend. Across those 123 the clip shortens one region on one object (tests/i386/windows/9f2ef84b..., resource directory 29,696 -> 4,096), drops none, leaves none outside its image and creates no zero-size region. Dropping that region instead of shortening it would move the same object to 199 functions, the two extra inside the clipped resource span
• Changed by this patch: 0 of 80 corpus objects, 0 of 8 tracked files. Worse: 0
• 3 of those 88 carried a region that left its image before the change; all 3 recover identical function, block and edge counts after it
• Everything above was measured after angr/cle master moved 10 commits under this branch, one of them #867, which reports a no-DEP image's sections as executable and so moves five of this cluster's other objects off zero on its own. The cited samples' numbers are unchanged by that move: 0 functions on the baseline arm and 59 and 144 here
• Environments: baseline /nix/store/g6qdzr2348j0zswx5msqx3glp1bi7c1j-python3-3.12.13-env, head /nix/store/zzr5vri7mf1ws09jn6q2389h81skg0ky-python3-3.12.13-env. Each one's installed cle/backends/pe/pe.py is byte-identical to the blob of the revision it stands for -- the baseline's to the base's, the head's to this head's, and neither to the other's -- and hashing every .py of each family package in both, angr (1,253 files), pyvex (30), archinfo (17) and pypcode (5) come out identical and only cle (101) differs. Component heads: angr 3f5717544, pyvex fc5b30d2e, archinfo 858a3f337, pypcode 559aacdc9, angr/binaries 67c892ca8be2

Caveats, one line each:

• Samples are live malware: SHA-256 identifies the bytes and does not publish them

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Full metadata-region list and CFG for the image whose SHA-256 is
7b00ec194518b62bc726966c0a45c3d992736cf8e75f4d12cf4bd3842bd90aaf, before and after this
change. It is a live malware sample and cannot be shared, so it is identified by that digest
and its bytes are not reproduced here. The image spans 0x400000-0x4c2bff and enters at
0x4bf000, inside .text, which cle reports executable; the entry block lifts to
14 instructions ending in Ijk_Call on both sides, which is the point of the report
rather than a difference between them.

Both blocks are one run of the same script against the same file, printing the same queries in
the same order. The two sides are two builds of cle and nothing else, which is measured rather
than asserted: in each build the installed cle/backends/pe/pe.py is byte-identical to the blob
of the revision it stands for and not to the other's, and every .py of angr (1,253 files),
pyvex (30), archinfo (17) and pypcode (5) hashes identically across the two, with only cle's 101
files differing.

Before — five pointer arrays out of the load-config directory land outside the image, CFGFast recovers nothing:

cle master 5d5929e

meta_regions, flattened (22 rows, 5 outside the image):
region                     address                size  inside the image
IAT                        0x4c0000                208  yes
IMPORT_DIRECTORY           0x4c00f0                100  yes
  IMPORT_DIRECTORY         0x4c00f0                100  yes
  ILT                      0x4c0154                 88  yes
  ILT                      0x4c01ac                120  yes
  ILT                      0x4c0224                 24  yes
  ILT                      0x4c023c                  8  yes
  IMPORT_HINT_NAME_TABLE   0x4c0244                960  yes
  STRING_BLOB              0x4c03b0                 13  yes
  STRING_BLOB              0x4c0570                 13  yes
  STRING_BLOB              0x4c05e6                 12  yes
  STRING_BLOB              0x4c0604                 11  yes
RESOURCE_DIRECTORY         0x4c2000               2709  yes
DEBUG_DIRECTORY            0x4c0448                 28  yes
  DEBUG_DIRECTORY          0x4c0448                 28  yes
LOAD_CONFIG_DIRECTORY      0x4c0324                124  yes
  LOAD_CONFIG_DIRECTORY    0x4c0324                124  yes
  LOAD_CONFIG_DIRECTORY    0x2ee0000        4526806348  NO
  LOAD_CONFIG_DIRECTORY    0x6564           6796609364  NO
  LOAD_CONFIG_DIRECTORY    0x6e6f4374       6806420940  NO
  LOAD_CONFIG_DIRECTORY    0x61696c41            67020  NO
  LOAD_CONFIG_DIRECTORY    0xf10000         4728132892  NO

CFGFast(normalize=True, data_references=False, resolve_indirect_jumps=True)
functions 0   blocks 0   edges 0
_seg_list.occupied_by_sort(entry), after _pre_analysis        'unknown'
_seg_list.occupied_by_sort(entry), after the analysis         'unknown'
_seg_list.next_pos_with_sort_not_in(entry, {'code'}, max_distance=4096) == entry
    after _pre_analysis True, after the analysis True
_lift calls: 1 in all, 1 of them asked for size 0; the first four sizes [0]

After — the five are gone, every remaining region is inside the image, and the same call recovers 144 functions:

with this change 23e28a7

meta_regions, flattened (17 rows, 0 outside the image):
region                     address                size  inside the image
IAT                        0x4c0000                208  yes
IMPORT_DIRECTORY           0x4c00f0                100  yes
  IMPORT_DIRECTORY         0x4c00f0                100  yes
  ILT                      0x4c0154                 88  yes
  ILT                      0x4c01ac                120  yes
  ILT                      0x4c0224                 24  yes
  ILT                      0x4c023c                  8  yes
  IMPORT_HINT_NAME_TABLE   0x4c0244                960  yes
  STRING_BLOB              0x4c03b0                 13  yes
  STRING_BLOB              0x4c0570                 13  yes
  STRING_BLOB              0x4c05e6                 12  yes
  STRING_BLOB              0x4c0604                 11  yes
RESOURCE_DIRECTORY         0x4c2000               2709  yes
DEBUG_DIRECTORY            0x4c0448                 28  yes
  DEBUG_DIRECTORY          0x4c0448                 28  yes
LOAD_CONFIG_DIRECTORY      0x4c0324                124  yes
  LOAD_CONFIG_DIRECTORY    0x4c0324                124  yes

CFGFast(normalize=True, data_references=False, resolve_indirect_jumps=True)
functions 144   blocks 290   edges 367
_seg_list.occupied_by_sort(entry), after _pre_analysis        None
_seg_list.occupied_by_sort(entry), after the analysis         'code'
_seg_list.next_pos_with_sort_not_in(entry, {'code'}, max_distance=4096) == entry
    after _pre_analysis False, after the analysis False
_lift calls: 555 in all, 0 of them asked for size 0; the first four sizes [400, 32, 400, 6]

The second sample, 3d7fe603def5bc06941033d5e687a990f423947d8d88fe5396f9d359b8930413, moves
the same way: 5 of 27 regions outside the image and 0 functions before,
0 outside and 59 functions after.

[angr-bot]

Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_862