[FIX]: Evaluate attack verdicts over final traces - #150
Draft
Spencer Schoenberg (spencrr) wants to merge 12 commits into
Draft
Spencer Schoenberg (spencrr) wants to merge 12 commits into
Spencer Schoenberg (spencrr) wants to merge 12 commits into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Spencer Schoenberg (spencrr)
force-pushed
the
dev/spencrr/trace-xpia-stopping
branch
from
August 8, 2026 02:31
d80c55b to
84d0197
Compare
Spencer Schoenberg (spencrr)
force-pushed
the
dev/spencrr/trace-xpia-stopping
branch
2 times, most recently
from
August 27, 2026 17:30
cdcad35 to
bedbf6b
Compare
Spencer Schoenberg (spencrr)
force-pushed
the
dev/spencrr/trace-xpia-stopping
branch
2 times, most recently
from
September 8, 2026 22:46
3051684 to
4074ae9
Compare
Keep numeric overflow and evaluation text conversion failures inside the worker-output error boundary so controllers mark runs incomplete and preserve earlier results.
Require explicit response scopes and distinguish online evidence from terminal verdict input without an alias. Apply canonical policies consistently to shared evaluation schemas and version the tightened population contract. Migrate callers and document direct pre-1.0 API replacement while preserving fail-closed records.
Spencer Schoenberg (spencrr)
force-pushed
the
dev/spencrr/trace-xpia-stopping
branch
from
September 25, 2026 01:04
4074ae9 to
97fb0a2
Compare
Use final_trace_evaluation consistently in Result, canonical records, JSON reports, and xdist transport. Update examples and round-trip tests without retaining an alias for the earlier spelling.
Require the same observability and manifest before reusing an online judgment. Copy optional evidence through the shared tolerant renderer so malformed supporting text cannot discard an established verdict. Keep terminal evidence and operand lists independent from online records.
Retire the list reducer after probe execution adopts terminal evaluation. Keep explicit response scopes, online evidence separation, zero-turn errors and xdist v3 semantics consistent across tests, exports and extension guidance.
Remove the old reducers and helper now that both built-in strategies evaluate terminal traces. Keep explicit-scope stopping classification, provenance and extension documentation aligned with the shared runner.
Spencer Schoenberg (spencrr)
force-pushed
the
dev/spencrr/trace-xpia-stopping
branch
from
September 25, 2026 02:26
97fb0a2 to
bb6279f
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Moves XPIA verdict evaluation to the terminal trace and separates it from online stopping.
stop_whenaccepts an explicit evaluator,None, or the default"auto". Auto mode reuses the verdict evaluator only for framework-owned conditions whose detected result is stable as turns are appended, such as cumulative tool and side-effect checks.An identical stop/verdict evaluator is not called again when its latest result already covers the terminal trace. Terminal evidence is stored on
Result.terminal_evaluation, trace completion usesTraceEndReason, and online stop evidence usesEvaluationPurpose.STOP_CHECK. Final evaluation remains inside the active session and injection stack. Observability downgrades are recorded without mutating response metadata, and cleanup failures discard otherwise successful verdict evidence and returnERROR.Depends on #149. Because the branches live on a fork, this PR temporarily includes lower-layer diffs and targets
main; those diffs disappear as dependencies merge.Breaking changes
Behavioral change: unknown or stochastic verdict evaluators no longer run on every prefix by default. They evaluate once over the terminal trace and may run to
max_turnsunless an explicitstop_whenis supplied. Auto-stopped attacks expose online evaluator feedback to adaptive drivers.Checklist
pre-commit run --all-filespassesValidation after reslicing: the full suite passes with 1,178 tests and 5 skips; all pre-commit checks pass.