PE: Map a section over its raw size when that is larger than its virtual size

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Problem

PESection reports a section's mapped size as Misc_VirtualSize alone. Windows maps a section
over its raw data as well, so a section whose SizeOfRawData exceeds its Misc_VirtualSize is
short in cle unless the file or the image runs out first. It is not a corner case: over the 106 PE
files angr/binaries tracks at fc07821c, this change grows 706 sections in 101 of them.

tests/x86_64/windows/fauxware.exe is the plain version. .text states
VirtualSize 0x7dcf and SizeOfRawData 0x7e00, and cle stops it at 0x7dcf:

>>> ld = cle.Loader("tests/x86_64/windows/fauxware.exe", auto_load_libs=False)
>>> ld.main_object.sections_map[".text"]
<.text | offset 0x400, vaddr 0x140001000, size 0x7dcf>

The 0x31 bytes it drops are in the file and already in cle's memory. Here they are zero padding,
which is the ordinary case.

Where the shortfall covers the entry point, the binary loads and analyses to nothing.
CFGBase._executable_memory_regions builds its regions for PE and COFF from section.min_addr
and min(section.memsize, section.filesize), so an entry outside every section's memsize is in no
executable region, CFGFast refuses every address, and the result is 0 functions and 0 nodes.
A section that declares VirtualSize 0 — which the PE specification permits, and which Windows
handles by using the raw size — gets memsize 0 and loses its whole content this way.

Root cause

cle/backends/pe/regions.py, PESection.__init__:

super().__init__(
    name or pe_section.Name.decode("latin-1"),
    pe_section.PointerToRawData,
    pe_section.VirtualAddress + remap_offset,
    pe_section.Misc_VirtualSize,
)

Section.__init__ uses that value for memsize, and Region.contains_addr is
vaddr <= addr < vaddr + memsize, which is what find_section_containing walks.

cle#285 reported the same geometry in 2021 and 6cf00b4b fixed half of it: it gave PESection a
filesize of its own, taken from SizeOfRawData, and left the memsize argument passed to
Section.__init__ as Misc_VirtualSize. That half has not been revisited since.

Fix

_mapped_size takes the larger of the virtual size and the raw size, and bounds the raw size
twice: to the bytes the file actually holds, and to the end of the image the optional header
declares. Both bounds only ever hold the result down toward Misc_VirtualSize, so no section
comes out smaller than it does today.

The image bound matters on real files:
tests/i386/windows/aa893de523f58ee14972b94fef7ecdbb930cbdc700d8be097eb8a6de2549ce73 appends
14 MiB to itself and counts it in .reloc's SizeOfRawData, which reaches 0xe25000 against a
SizeOfImage of 0x4f02e. Without the bound that object's span would grow from 0x4f02e to
0xe72000; with it, .reloc stays at 0x202e.

max_addr follows memsize, so the tail pe.py keeps of the mapped image grows on 97 of the
106: more bytes retained, never fewer, and the shortfall between an object's claimed span and its
backed bytes is unchanged object for object. Two of our open items are in the same code. cle#790
backs the last byte and the last section's virtual-size tail; with both changes applied, 105 of
the 106 tracked files come out with no shortfall at all, and the one that does not is cle#794's,
which is about a section whose raw data begins past the end of the file. The two compose and
neither needs the other.

Deliberately not done: filesize still comes from SizeOfRawData; the section is not rounded up
to SectionAlignment; and a section is not clamped to the next section's virtual address. That
last is left out on a measurement: across both corpora no section reaches the clamp, so it would
be an untested branch.

Testing

tests/test_pe.py::TestPESectionMappedSize, two methods on files angr/binaries already tracks.
The first asserts fauxware.exe's .text maps 0x7e00 while .data, which states the reverse
(VirtualSize 0xa19 against SizeOfRawData 0x400), keeps its virtual size. The second asserts
both halves of the rule on the 14 MiB object above. Both fail on master and pass here. The
pre-existing test_loading_incomplete_pe_file covers the file bound: an earlier version of this
change without that bound failed it.

CFGFast(normalize=False) over all 106 tracked PE files, both arms: nothing is lost — no
object loses a function or a node — and four gain one function each, 244,228 -> 244,232. Those
four are not recoveries. Each starts nine bytes before the old end of .text, in bytes master
already had, and runs into the zeros past it, so a block master could not finish now decodes to
its end: one 0xff and 197 zeros. The count that matters is the zero losses.

The binaries whose CFG this recovers are in a private corpus and cannot be named here. Three go
from 0 functions to 125, 281 and 490. A fourth already recovered 1 function and now recovers 4,
and it answers whether this only ever maps padding: its .text states VirtualSize 0xcb4 against
SizeOfRawData 0xe00, and at 0x401db0, 0xfc bytes past the section as cle reports it, sits
mov eax, [esp+4]; sub esp, 0x12c with four callee-saved pushes and push 0x1f0fff
(PROCESS_ALL_ACCESS). That is a compiler-emitted prologue, and today it is outside every
section. No file in angr/binaries shows the entry-point symptom — 105 of the 106 already have
their entry inside a section — so the public evidence is the geometry and the control.

Validation: #835 (comment).

session: sharpen

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Validation record for head 199f5d7ce4aaddba44156ee9ecc1fdd097c689e3 on cle master 0e77ade3c39a3cee05f65051e57955675e1ac21b, run against angr 23b470d9f845e005286c498f70d1e108502fc1c5, pyvex 2324a19e803d4141e7649a939e668b46ec27c3fb and binaries fc07821c89535534979b02760e7e1bfc35faf690.

Tests

suite	result
cle, whole suite	263 passed, 9 skipped
tests/test_pe.py at this head	17 passed
tests/test_pe.py read from this branch, imported against cle master	2 failed, 15 passed, assert 32207 == 32256 and assert 107869 == 108032

The third row is the check that the new tests are regression tests: they fail without the change.

Lint and type

check	result
run-ci-diff-checks.py against the base revision	exit 0. cle/backends/pe/regions.py errors 0 -> 0, cle/backends/pe/pe.py 52 -> 52, tests/test_pe.py 4 -> 4
pre-commit run --files on the three changed files	all hooks passed, exit 0

Section geometry over every PE angr/binaries tracks

Both arms are separate nix store builds of the two trees; the dump records every section's
vaddr, memsize, filesize and executable bit, the object's min_addr, max_addr and entry,
and a sha256 over its mapped memory.

objects                                                      106
load errors                                        before 0, after 0
sections whose memsize changed                               706
sections that shrank                                           0
objects whose executable region set changed                   97   (grew 97, shrank 0)
executable bytes added across the corpus                  43,389
objects whose find_section_containing(entry) changed           0

The dump asserts every section's vaddr, offset, filesize, executable bit and count
identical across the two arms; memsize is the only section field that moves. It does not stop
there at the object level. max_addr is the maximum over sections, and pe.py truncates the
mapped image to max_addr - min_addr, so a larger memsize means less truncation: 97 of the
106 objects change their mapped-memory sha256, total backed bytes 527,206,054 -> 527,241,805
(+35,751), and no object loses a byte. Those 97 are not the 97 whose executable regions
changed — the two sets overlap in 93, four each way. Visible in one command:
tests/i386/test_rol.exe's backer goes 4,120 -> 4,607 bytes and fauxware.exe's 70,049 ->
70,143.

The newly executable bytes are the padding between a section's virtual size and its file-aligned
raw size, across 131 grown ranges. 43,383 of them read back through cle, and those are 42,940
0x00 and 443 0xcc; 128 of the 131 ranges are all zero. The six that do not read back are one
byte each in six objects, at an address pe.py's inclusive max_addr already excludes on master
as well.

CFG regression control over the same 106 files

CFGFast(normalize=False) on every one of the 106, both arms, 900 s alarm, angr
23b470d9f in both.

comparable objects                                           105
unusable (TIMEOUT in BOTH arms, tests/x86_64/windows/7107ab06...bd5)   1
objects that lost a function or a node                         0
objects whose status differs between the arms                  0
functions   gained 4   lost 0   unchanged 101
nodes       gained 5   lost 0   unchanged 100
totals      functions 244,228 -> 244,232      nodes 1,894,087 -> 1,894,092

The four gains are not recoveries and should not be read as a win. Each starts nine bytes before
the old end of .text — inside the executable region master already had, and readable there — and
its single 198-byte block runs on into the zeros the change newly maps: one 0xff then 197
0x00, decoding as inc dword ptr [rax] followed by 98 of add byte ptr [rax], al. What the
change adds is the room for that block to finish, not the address it starts at:
tests/x86_64/windows/1309c899...c94f at 0x14026883f,
known_patterns_containing_record.exe at 0x1400027ff,
tests/x86_64/windows/known_patterns_wdk_ksud.exe at 0x140002c5f (the i386 file of the same
basename is unchanged in both arms), known_patterns_wdk_list.exe at 0x14000282f. A fifth file gains one node and no function.
That is 4 junk functions in 244,232 and 5 junk nodes in 1,894,092, against no losses anywhere.
Those bytes are executable under Windows; suppressing a block of zeros belongs in angr's
sparse-region heuristics, not in the loader.

Corpus this fixes

The corpus this fixes is private and cannot be named. Both arms, CFGFast(normalize=False),
600 s alarm.

• The class it was found in: 27 PE objects that load and have an executable section, 26 of
  which recover no functions on master. Exactly 4 of the 27 move, and 23 are untouched. Three are the target
  shape — the entry's section declares VirtualSize 0 against SizeOfRawData 0x1600, 0x8100
  and 0x6e00 — and go from 0 functions to 125, 281 and 490. The fourth already recovered 1
  function and now recovers 4; that one is the evidence that this maps more than padding, and the
  prologue it gains is in the output comment.
• Direction control: 60 objects from the same corpus that already recover functions on
  master. 0 lose a function or a node. 6 gain functions, by 1, 6, 4, 1, 1 and 6, and a
  seventh gains one node; totals 154,222 -> 154,241 functions and 604,922 -> 604,967 nodes. All 60
  are comparable — none errors or times out in either arm.
• The control is exercised rather than merely quiet: 41 of those 60 have at least one section
  grow, and only 7 change their CFG at all, so 34 objects had their memory map altered and
  produced identical output. Across all 174 runs in this corpus no section shrank and every
  section's vaddr, offset, filesize and count is identical between the arms. The seven
  movers were re-run in the base arm and reproduced exactly, so the small deltas are deltas and
  not run-to-run noise.
• One of cle#812's two declined objects is unblocked by this and by that change together, and by
  neither alone: it goes 0 -> 0 with either change on its own and 0 -> 155 with both. The two
  diffs apply together with no conflict; neither depends on the other.

Workspace gate

Run after publication, on this exact head. It matters here because angr/analyses/cfg/cfb.py
builds a MemoryRegion per section and per segment straight from memsize, and a PE's segments
is its sections, so CFBlanket's regions move on 101 of the 106 tracked files;
reassembler.py, vtable.py and flirt.py read memsize too, and the CFG control above covers
CFGFast and not those.

PARTIAL PASS: 10 of 14 suites ran and passed; 4 did NOT run.
SUITES SKIPPED (feature has not adopted, did NOT run): archinfo pypcode pyvex angr-management

suite	result
cle	263 passed, 9 skipped
angr	2958 passed, 47 skipped, 2 xfailed, 286 subtests passed, 0 failed in 934 s

workspace, test-inputs, test-packages, pre-commit, feature-build, mono, pysoot,
cle, angr and angr-rust ran and passed. The four that did not run are the repositories this
collection did not adopt, and they are named above rather than folded into a pass. The run built
cle at this head with angr 23b470d9f, pyvex 2324a19e, archinfo 06a207fd, and the whole log
holds no failing test.

Gaps

• One tracked file, tests/x86_64/windows/7107ab06446ce4a51226196453066e7d361972364ad1543fe8a3a03a957e1bd5,
  hits the 900 s alarm in both arms, so it is excluded from the comparison rather than counted
  as unchanged.
• The public corpus contains no instance of the failure this fixes: 105 of the 106 already have
  their entry inside a section and one has no entry point. The public evidence is the geometry and
  the regression control.

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Before and after on the reproducer, and the section geometry on a public file.

Before — the section stops at its virtual size, the 14 MiB .reloc is already held down by
its own virtual size, and the three private entry points are in no section at all:

cle master 0e77ade

>>> ld = cle.Loader("tests/x86_64/windows/fauxware.exe", auto_load_libs=False)
>>> ld.main_object.sections_map[".text"]
<.text | offset 0x400, vaddr 0x140001000, size 0x7dcf>       # SizeOfRawData is 0x7e00
>>> ld.main_object.sections_map[".data"]
<.data | offset 0xb200, vaddr 0x14000c000, size 0xa19>       # SizeOfRawData is 0x400

The image that appends 14 MiB to itself and counts it in .reloc:

tests/i386/windows/aa893de523f58ee14972b94fef7ecdbb930cbdc700d8be097eb8a6de2549ce73
  SizeOfImage 0x4f02e, file 0xe59200 bytes
  .text   va 0x1000  VirtualSize 0x1a55d  SizeOfRawData 0x1a600   -> memsize 0x1a55d
  .reloc  va 0x4d000 VirtualSize 0x202e   SizeOfRawData 0xe25000  -> memsize 0x202e

On the three private binaries whose entry point falls in the shortfall:

sample  entry RVA  entry's section  VirtualSize  SizeOfRawData  find_section_containing(entry)
  1     0x2365     AUTO             0            0x1600         None
  2     0x10e4     .test            0            0x8100         None
  3     0x7a4d     .text            0            0x6e00         None

CFGFast(normalize=False), functions inside the main object:  0, 0, 0

After — .text reaches its raw size, .reloc still stops at the end of the declared image,
and each private entry point lands in a section:

with this change

>>> ld.main_object.sections_map[".text"]
<.text | offset 0x400, vaddr 0x140001000, size 0x7e00>
>>> ld.main_object.sections_map[".data"]
<.data | offset 0xb200, vaddr 0x14000c000, size 0xa19>       # unchanged: VirtualSize wins

tests/i386/windows/aa893de5...ce73
  .text   -> memsize 0x1a600     (grows to the raw size)
  .reloc  -> memsize 0x202e      (unchanged: the raw size runs past SizeOfImage)

sample  find_section_containing(entry)   functions in the main object
  1     AUTO                             125
  2     .test                            281
  3     .text                            490

A fourth private sample already recovered 1 function and now recovers 4. Its .text states
VirtualSize 0xcb4 against SizeOfRawData 0xe00, and the bytes between the two hold real code,
not padding:

401db0  mov     eax, dword ptr [esp+0x4]
401db4  sub     esp, 0x12c
401dba  push    ebx
401dbb  push    ebp
401dbc  push    esi
401dbd  push    edi
401dbe  push    eax
401dbf  push    0x0
401dc1  push    0x1f0fff
401dc6  call    dword ptr [0x401084]

0x1f0fff is PROCESS_ALL_ACCESS, and 0x401084 is inside the IAT the data directory declares.
The object is packed and has no import entries to resolve, so the call target is not a named
import; the prologue is what carries the point.

0x401db0 is 0xfc bytes past the end of .text as cle master reports it, and 0x50 bytes short of
its end as Windows maps it.

The 0x31 bytes fauxware.exe's .text gains were already in cle's memory before the change and
equal the file's bytes at PointerToRawData + 0x7dcf. What the section claims moves, and so does
what the object retains: max_addr follows memsize, and pe.py trims the mapped image to it,
so this file's backing goes 70,049 -> 70,143 bytes. No object loses a byte.

[angr-bot]

Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_835