Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion docs/api/core-types.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# API Reference — Core Types

Data types shared across the entire framework. All importable from `rampart` directly.
Data types shared across the entire framework. Stable execution vocabulary is
available from `rampart.core`; established result types remain importable from
`rampart` directly.

## Data Types

Expand All @@ -14,6 +16,8 @@ Data types shared across the entire framework. All importable from `rampart` dir
- ToolCall
- SideEffect
- Turn
- EvaluationPurpose
- TraceEndReason
- EvalOutcome
- EvalResult
- EvalContext
Expand All @@ -30,6 +34,8 @@ Data types shared across the entire framework. All importable from `rampart` dir
- SafetyStatus
- HarmCategory
- InjectionRecord
- resolve_attack_verdict
- resolve_probe_verdict
- resolve_as_attack
- resolve_as_probe

Expand Down
14 changes: 6 additions & 8 deletions docs/attacks/xpia.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,12 +159,12 @@ Place the cheaper evaluator on the left side of `|` — it short-circuits if the
The `&` above asks whether both happened, so one condition that definitively did not happen settles the result even if the adapter could not observe the other. Use `|` when either condition on its own would count as the attack succeeding. When the adapter does not report the channel the left condition needs, the result records that on [`EvalResult`][rampart.core.types.EvalResult]. Reversing those two operands records nothing, because a `NOT_DETECTED` left operand short-circuits `&` before the other one runs. See the note on undetermined operands in [Authoring Tests](../usage/authoring-tests.md#composing-evaluators).

!!! warning "Multi-turn scope"
State the temporal scope explicitly for multi-turn attacks. The complete
positive and negated mapping is maintained in the
`ResponseContains` requires an explicit temporal scope, even for a
single-turn attack. The complete positive and negated mapping is maintained in the
[Temporal Scope table](../usage/authoring-tests.md#temporal-scope).
Omitting `scope` inspects only the current response and emits a
`FutureWarning` for multi-turn contexts. Scope applies only to turns in the
evaluator context; it does not control execution length or early stopping.
Use `CURRENT_TURN` only when earlier responses should be ignored. Scope
applies only to turns in the evaluator context; it does not control
execution length or early stopping.

### LLMDriver for Adaptive Triggers

Expand Down Expand Up @@ -233,7 +233,7 @@ See [`Attacks.xpia()`][rampart.attacks.Attacks.xpia] for the full API reference.
| `inject` | `InjectionHandle \| list[InjectionHandle] \| None` | `None` | Prepared injections from `surface.inject()`. `None` for inline XPIA. |
| `trigger` | `str \| list[str] \| Request \| list[Request] \| PromptDriver` | required | Benign prompt(s) that cause retrieval of injected content. |
| `evaluator` | [`Evaluator`][rampart.core.evaluator.Evaluator] | required | What attack condition to detect. |
| `max_turns` | `int` | `5` | Maximum prompt-response exchanges before `ERROR`. |
| `max_turns` | `int` | `5` | Maximum prompt-response exchanges; reaching the limit resolves the trace normally. |
| `event_handlers` | `list[ExecutionEventHandler] \| None` | `None` | Additional lifecycle event handlers. |

---
Expand All @@ -249,5 +249,3 @@ This only fires when all three conditions hold:
3. Zero tool calls were observed

It is a backstop for evaluators that cannot say up front what evidence they need, such as `LLMJudge`, where the answer depends on the objective. `ToolCalled` and `SideEffectOccurred` return `UNDETERMINED` themselves, so on their own they do not reach this check as `SAFE`. A composition still can, so the backstop stays.


6 changes: 2 additions & 4 deletions docs/concepts/probes.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,11 @@ All probes are created through the [`Probes`][rampart.probes.Probes] class:

```python
from rampart import Probes
from rampart.evaluators import ResponseContains
from rampart.evaluators import ResponseContains, ResponseScope

execution = Probes.behavior(
prompt="What is 2 + 2?",
evaluator=ResponseContains("4"),
evaluator=ResponseContains("4", scope=ResponseScope.ALL_TURNS),
)

result = await execution.execute_async(adapter=my_adapter)
Expand All @@ -60,5 +60,3 @@ Provide exactly one of `prompt`, `prompts`, or `driver`.
| [Behavioral](../probes/behavioral.md) | `Probes.behavior(...)` | Verify the agent produces expected responses or behaviors |

More probe types will be added. Each new probe is a new factory method on `Probes`.


116 changes: 75 additions & 41 deletions docs/concepts/trace-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,10 +49,22 @@ IDs must be recorded, not generated during deserialization. These boundary
rules do not replace the normal dataclass constructors used during execution.

When recorded, `result_index` must be a nonnegative integer. Population references
require a positive `size`, a zero-based `index` less than `size`, and a `threshold`
in `[0, 1]`. These invariants are enforced by the canonical record boundary and
adapter, without changing live `PopulationRef` construction or other adapters.
Scalar bounds are included in the generated JSON Schema.
require a nonempty string `id`, a positive `size`, a zero-based `index` less than
`size`, and a finite `threshold` in `[0, 1]`. Live `PopulationRef` constructors
enforce these invariants, and the canonical adapter revalidates existing instances
at the write boundary as well as decoded records. Scalar bounds, including the
nonempty ID, are included in the generated JSON Schema.

`Result.final_trace_evaluation` records evaluation of the trace when execution
stops, including when the turn budget is reached.
`Turn.eval_result` remains separate online evidence; `Turn.eval_purpose` records
why that online evaluation ran. A non-null purpose requires an evaluation on the
same turn. `Result.trace_end_reason` records why turn production stopped. These
provenance fields are optional: missing or null means the producer did not record
them, not that the last online evaluation is the terminal one. The codec never
infers terminal evidence or a stop reason from the result status or turns.
Both placements of `EvalResult` receive the same strict type, finite-confidence,
Unicode-scalar, and closed-enum validation.

Body encoding validates the live result and uses Pydantic's JSON-mode
serialization, with adapter-local Unicode validation and Python ISO datetime
Expand All @@ -66,18 +78,24 @@ re-encoded when Pydantic's JSON-mode writer has a lower nesting limit.
These failures raise `SchemaError`; successful decoding alone does not guarantee
that an unusually deep record can be emitted again.

These policies belong to the cached canonical adapter, not to the public
dataclass annotations or configuration. Fields remain `dict[str, Any]` and
Canonical serialization policies belong to the cached adapter, not to the public
dataclass annotations or configuration. Shared dataclass definitions reuse the
adapter's configured copy at every occurrence. Fields remain `dict[str, Any]` and
`datetime | None`. Independently constructed Pydantic adapters retain their
normal behavior, including live binary payload support.
normal behavior, including live binary payload support and ordinary instance
validation. Constructor invariants still apply when those adapters construct a
new instance.
The canonical adapter supplies its own `datetime` / `Path` resolution namespace;
the shared types module keeps those imports under `TYPE_CHECKING`.

`ResultRecord.json_schema()` returns the adapter-derived body schema plus the
versioned envelope. Small schema customizations describe the trace-only payload
restrictions and the request invariant (a prompt or at least one attachment).
It also describes the dependency between a turn's purpose and evaluation.
`JsonSchemaValue` is the return type, not a separate model or validator.
The open Draft 2020-12 contract is committed at `schemas/trace.v1.schema.json`.
The active open Draft 2020-12 contract is committed at
`schemas/trace.v2.schema.json`; `schemas/trace.v1.schema.json` is retained unchanged
as the historical description of v1.

Regenerate it with `uv run python scripts/generate_trace_schema.py`.
The generator selects the filename from `TRACE_SCHEMA_VERSION`. CI runs the same
Expand All @@ -88,7 +106,8 @@ command with `--check` to detect drift.
Schema drift checking alone does not establish compatibility. A separate CI gate
requires a checked-in decision in `schemas/trace-compatibility.json`, bound to the
contract content by SHA-256 fingerprints. The inputs are `result.py`, `types.py`,
`serialization.py`, `_schema.py`, and all published `trace.v*.schema.json` files.
`serialization.py`, `_schema.py`, `_population.py`, and all published
`trace.v*.schema.json` files.
Watching the models and codec policies also catches changes that do not appear
in JSON Schema. This is deliberately conservative: even a nonsemantic edit to
these inputs needs a compatibility rationale.
Expand All @@ -100,7 +119,8 @@ For a contract change, update the declaration:
compatibility, such as an additive-optional field with a defined absence behavior.
- **`new-major`** increments the major by one, retains earlier published schema
files, and references a nonempty repository migration document in `migration_note`.
The migration obligations below still apply, including an upcaster and API/CLI.
The note explains the break and the actual reader/migration support shipped;
it does not require an upcaster or dual reader.

Historical schema files remain unchanged in subsequent same-major PRs, not just
during a major bump. A `compatible` decision may update the active major's schema;
Expand All @@ -122,7 +142,7 @@ Without `--base-ref`, including on main-branch pushes, the command checks the
declaration's version and current content fingerprint only.

**The declaration is a review gate, not proof of compatibility.** Reviewers must
assess the rationale, semantic behavior, and required migration implementation.
assess the rationale, semantic behavior, and any claimed migration support.
A regenerated schema or a `compatible` assertion does not make a breaking change
safe. Keep the input list current if contract policy moves to additional modules.

Expand Down Expand Up @@ -190,10 +210,10 @@ does not make currently rejected formats readable by older readers.
## Versioning

- Every serialized record carries one root `version` field. The current schema
is **`rampart.trace.v1`**.
is **`rampart.trace.v2`**.
- The record version is **independent** of transport or projection versions,
including the existing xdist envelope version (`rampart.xdist.v2`). Each
version describes its own layer and may evolve separately.
including xdist's versioned envelope. A transport's cadence or version number
does not select the canonical trace major.
- There is a **single root version** — nested types (`Turn`, `Payload`,
`EvalResult`, …) do not carry their own versions.

Expand All @@ -210,13 +230,14 @@ does not make currently rejected formats readable by older readers.
defaults and does not retain which fields were absent. For example, omitted
`turns` becomes `[]` and is emitted when re-encoded.
- **Structural change = major bump.** Removing, renaming, or retyping a field,
or changing its meaning or nesting, bumps `vN → vN+1` with a changelog and a
migration note.
changing its meaning or nesting, or narrowing its accepted value domain bumps
`vN → vN+1` with a changelog and a migration note. Rejecting previously accepted
empty population IDs is a domain-narrowing change, not an optional addition.

```mermaid
flowchart TD
change([proposed schema change]) --> q1{"adds a field only?"}
q1 -- no --> struct["structural:<br/>remove / rename / retype /<br/>change meaning or nesting"]
q1 -- no --> struct["structural:<br/>remove / rename / retype /<br/>change meaning, nesting, or accepted values"]
q1 -- yes --> q2{"optional with a<br/>well-defined default?"}
q2 -- no --> struct
q2 -- yes --> add["additive-optional"]
Expand All @@ -227,17 +248,20 @@ flowchart TD

## Reader posture

- Readers tolerate unknown fields and **fail closed on an unknown major** — a
- Readers tolerate unknown fields and **fail closed on an unsupported major** — a
record is never best-effort parsed across a major boundary.
- This reader supports only v2. Retaining the v1 schema does not register a v1
decoder; v1 records raise `UnsupportedSchemaVersionError`, as do future majors.
- Forward compatibility is **additive-only within a major**. A newer major read
by an older framework fails closed by design.
- Schema descriptions and validators derived from this format must remain open
to unknown properties within a major version.

## Enum posture

- The closed enums — `SafetyStatus`, `EvalOutcome`, `ObservabilityLevel`, and
`PayloadFormat` — **fail closed** on an unknown value. A serialized safety
- The closed enums — `SafetyStatus`, `EvalOutcome`, `EvaluationPurpose`,
`TraceEndReason`, `ObservabilityLevel`, and `PayloadFormat` — **fail closed**
on an unknown value. A serialized safety
result must never silently misread one; there is no warn-and-degrade path.
- `HarmCategory` is the sole exception: it travels as a **passthrough string**
and is never coerced, so a new harm label from a future producer round-trips
Expand All @@ -256,26 +280,37 @@ flowchart TD
- Timestamps retain Python's ISO 8601 representation, including naive datetimes
and UTC offsets. The schema describes strings rather than RFC 3339
`date-time`, which would exclude some supported Python datetimes.
- `rampart.trace.v1` does not define a durable representation for binary or
- `rampart.trace.v2` does not define a durable representation for binary or
opaque payload artifacts. Encoding or decoding one fails closed rather than
coercing it to text.
- Encoding and decoding preserve supported metadata, including keys used for
transport bookkeeping or loss/truncation markers. Unsupported values are
rejected regardless of the key name. Metadata hygiene belongs to consumer
preparation, not to the canonical codec.

## Migration mechanics

Only `rampart.trace.v1` exists today. No upcaster or persisted-data migration
tooling is implemented. If a later structural change introduces a new major,
the migration policy requires:

- writers emit the latest supported major;
- each major bump ships an adjacent upcaster (`vN-1 → vN`) and an explicit
migration API/CLI;
- migrating persisted data is an explicit operation; reading never rewrites an
artifact in place; and
- encountering an unsupported major fails closed.
## v1 to v2 migration note

V2 narrows population IDs to nonempty strings. It also records optional terminal
evaluation, trace-end reason, and online evaluation purpose, and consistently
applies canonical validation to both terminal and online evaluations. The new
optional fields alone would not require a major bump; the narrowed ID domain
does. The field is named `final_trace_evaluation` in the Python API, canonical
records, JSON reports, and xdist transport. The earlier `terminal_evaluation`
spelling is removed without an alias.

Writers emit v2, and this reader accepts only v2. No v1 reader, adjacent upcaster,
or persisted-data migration API/CLI is shipped. Historical v1 schema files are
retained for consumers that need to inspect old records, not as a support-window
promise.

Persisted v1 data must not be silently relabeled or parsed through the v2 reader.
Consumers choosing to migrate it must perform an explicit, application-owned
conversion into a separate v2 record and validate the result with
`deserialize_record()`. An empty population ID requires a legitimate identifier
from the producer's provenance or regeneration of the record; do not invent one.
Leave unrecorded terminal evaluation, stop reason, and turn purpose absent or
null rather than inferring them from the last online evaluation. Preserve the
original artifact; reading never rewrites persisted data in place.

## Future extensions

Expand All @@ -289,12 +324,11 @@ default do not require a major bump. Structural changes do. Apply the compatibil
review and declaration requirements to each extension rather than promising
compatibility for an unimplemented representation.

## Support window

This is a release-support commitment; the current reader supports only
`rampart.trace.v1`.
## Pre-1.0 support policy

Starting with the first release that writes durable trace records by default,
RAMPART supports reading `vN` and `vN-1` for **two subsequent framework
releases** (one deprecation cycle). The window is keyed on releases, not time.
Any major bump includes a changelog entry and migration note.
RAMPART does not promise deprecation periods, compatibility aliases, a two-release
support window, dual readers, or mandatory upcasters. Breaking changes may replace
old APIs directly. Every canonical major change still requires an explicit
version/compatibility decision, unchanged historical schema descriptions, and a
changelog entry and migration note describing actual support. Unsupported
versions always fail closed.
11 changes: 8 additions & 3 deletions docs/contributing/release-process.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,16 @@ RAMPART follows [Semantic Versioning](https://semver.org/) (`MAJOR.MINOR.PATCH`)
!!! note "Pre-1.0 stability"
While RAMPART is below `1.0`, minor version bumps may include breaking changes. The API is stabilizing but not yet frozen. The first stable release will be `1.0.0`.

## 3. Remove Deprecated Functionality
## 3. Review Breaking Changes

If you are incrementing the minor version, search the codebase for the new minor version (no leading `v`) to find occurrences where functionality was deprecated and announced for removal in this version. Typically, functionality is deprecated and stays for two minor versions before being removed.
During pre-1.0 development, remove obsolete APIs directly rather than maintaining
deprecated aliases, warnings, or a fixed support window. Migrate in-tree callers,
tests, and documentation together, and explain required caller changes in the
release notes.

If you find functionality to remove, merge the removal PR to `main` before proceeding.
Breaking persisted-data changes still require an explicit schema-version and
compatibility decision; see [Trace Schema](../concepts/trace-schema.md).
Merge the completed changes to `main` before proceeding.

## 4. Prepare Release Metadata

Expand Down
Loading
Loading