Add an inspect_ai eval engine behind --engine - #19
Draft
rominf wants to merge 20 commits into
Draft
Conversation
The legacy engine shells out to the claude CLI with permissions bypassed on the runner, and hand-rolls workspace staging, LLM judging and routing termination around it. This adds a second engine that hands that work to inspect_ai, selected with `--engine inspect` and off by default until a benchmark shows what it changes. Its agent is harness-independent rather than the claude CLI, so a skill is graded on whether its instructions work rather than on one harness's reading of them. That also removes what blocked Windows: with no CLI in the sandbox there is no agent bridge, and the bridge is Linux-only. Both engines produce the same outcome objects, so `summarize`, `render_markdown` and the report writers are untouched. Three pieces are not thin wrappers. The tools are ours because inspect's assume a POSIX guest, which the Windows legs do not have. The judge sees artifacts rather than the agent's prose, attaches images so "did it draw a cat" is answerable, and reports whether a requirement is satisfied so a "must not" is never scored by negating a verdict. Routing is a single model call with no agent loop: the decision is a tool call, visible in the first turn, so there is no turn to pay for and kill. machine.yml gains a `sandbox` key naming a compose file, so a skill that must reach the network to pull a model can say so.
Neither engine reported its own cost. The legacy engine discards the `total_cost_usd` the CLI returns with every result, and inspect keeps usage in its own log where skillscope's report never looks. That was fine with one engine and nothing to compare it against. Both now record into `usage`, which the report meta carries, and `tools/benchmark_engines.py` runs the same dataset through both and reports per-case agreement plus what each run spent. Agreement is compared case by case rather than in aggregate: two runs can post identical accuracy while individual cases flip in both directions and cancel out. `--noise` runs the legacy engine twice so flips can be read against how much it already disagrees with itself. The spend columns are not equally trustworthy and the report says so rather than tabulating them as though they were. Legacy token counts are a floor -- assistant events omit the system prompt and cached input, and a routing case is killed before its totals arrive -- so cost is its reliable figure. inspect has a cost only when the provider supplies pricing, which a gateway does not, so wall time is often the only column comparable on both sides.
The inspect engine grades a skill with a harness-independent agent, which tests whether the skill's instructions work rather than how one product reads them. That choice raises an obvious question, so `--engine claude-code` answers it: run the real harness in the sandbox and see whether it agrees. It slots into the existing flag rather than adding a command, so the benchmark tool can diff its report against either other engine with no new machinery. Reporting only -- harness runs are nondeterministic and the harness is not what we grade, so a divergence is a question about the skill rather than a build failure. Linux only, because inspect_swe shells `bash -c` to find the CLI and the proxy it installs in the guest is a Linux binary. Also fixes how the inspect engines count model calls. They recorded one per sample while the legacy engine records one per assistant response: identical for routing, where a case is a single turn, but a large undercount for behavioral, where the agent loops. The benchmark puts the two columns side by side and names that one comparable, so it had to mean the same thing on both sides. Counted from assistant messages now.
The two legacy commands read the same `claude` stream but recorded from different events: routing counted assistant replies, behavioral counted only the result event, which is one per case. A behavioral run of three cases reported three model calls while costing $0.72 -- far more than three small calls -- and the benchmark puts that column beside the other engine's and calls it comparable. Both now go through one recorder in `usage`. The same three cases report fourteen calls against the new engine's twelve, which is the like-for- like reading: two agents looping to similar depth.
`--engine` changes what drives the agent while leaving the dataset, the CLI and the reports identical, so it belongs in usage rather than being discoverable only from --help. Says what each engine needs, why the inspect one is cheaper, why the claude-code leg never gates, and where the sandbox does and does not exist.
…config Which provider to use is a property of the runner -- docker, podman on a host that has that instead, local where there is no container -- and whoever runs the job knows it while a skill does not. What the sandbox has to provide is the opposite: the skill declares a compose file when it needs network egress or a device. `SKILLSCOPE_SANDBOX` already selected the provider but returned it bare, discarding any compose file the skill had declared. A skill that needs the network would then run without it and fail for a reason nothing in the report explains. The two are now resolved independently. podman works with no registration: `inspect-podman` publishes an `inspect_ai` entry point and inspect resolves the bare name, so installing the extra is the whole setup.
A behavioral report showed the same numbers whether the agent had been contained or had worked directly in the harness's filesystem with permissions bypassed, which invites the reader to assume the former. Both engines now record what held the run, and the markdown says it outright -- the legacy engine reports `host`, and the Windows legs, which have no sandbox available at all, no longer look like the Linux ones. Routing under the inspect engine reports `none` rather than a provider: it offers the skill tool and never calls it, so nothing is executed and there is nothing to isolate. That is not the same claim as unprotected. Also stops the gateway-versus-federation guard firing on runs that reach no provider. `--model mockllm/model` is the wiring check that costs nothing, and refusing it because the shell holds both Anthropic variables broke it on exactly the machines most likely to have an OAuth token.
A graded run on the inspect engine starts a sandbox and installs a skill before it first reaches a provider, so a bad key surfaced as a task that failed after all that work -- a 401 buried in a sample error, eighteen seconds and a container in. One tiny call up front turns that into a message on the first line. The legacy engine already had this; the inspect one skipped it on the grounds that the CLI-based probe tested the wrong thing, which was true and left nothing in its place. mockllm reaches no provider, so it is skipped rather than charged for a round trip that proves nothing.
`list_paths` returned an empty list when the listing command failed, so a sandbox that could not be listed was indistinguishable from one the agent had left empty. That is not a cosmetic difference: `files_exist` fails, and the judge builds its evidence from the same listing, so every judged expectation fails too with "no files were produced". Nine checks blame the skill for what the harness did. It now raises. The scorer reports that it could not list the sandbox, and the judge declines to rule rather than being handed an empty workspace as fact -- a judge told "no files" will confidently conclude the agent did nothing.
A container sandbox starts at `/`. So `write_file("router.json")` landed
at `/router.json`, `find . -type f` walked the whole image and died on
`/proc`, and the judge -- had the listing survived -- would have read
twenty arbitrary system files as the case's artifacts. An agent left to
guess reasonably tried `/app`, then `~`, and scattered its output.
Everything a case does now happens in `/workspace`: fixtures are seeded
there, the file tools resolve against it, shell commands run in it, and
it is what gets listed. The agent is told so, because where its output
lands should not be something the dataset has to predict.
`local` keeps the harness's own working directory -- it already has a
sensible one, and creating `/workspace` on someone's machine would not be.
inspect_swe solves the same problem the same way; its agent cwd falls
back to the home directory when the sandbox default is `/`.
Two reasons a case could not pass in a container. The judge read artifacts by the path `list_paths` reported, which is relative to the case's working directory, while `read_file` resolves against the sandbox's own -- `/`. Every artifact came back unreadable and the judge concluded, reasonably, that nothing had been produced. It now resolves them the same way the tools do. And it never saw the agent's final message, so an expectation about what the agent *told* the user -- "output the curl commands they need" -- was unanswerable however well the agent had done. The legacy judge was given both halves and this one was not. They stay labelled apart, because the distinction is what stops an agent writing "I won't call the cloud API" from settling an expectation that it avoided doing so: actions are judged from tool calls and artifacts, and only what the agent told the user is judged from its message.
The judge's transcript was cut at the end, so the agent's most recent tool calls were the first to go -- and those are usually the ones an expectation is about. An agent writes a file, validates it, and reports the result; losing the validation makes the run look like it claimed something it never did, which is precisely how the last failing check read: "only the agent's unsupported claim". Tool calls are now kept whole, since each is short and the list of them is the record of what happened. Only results are capped, individually, because they are what grows without bound. If the whole still overflows, the middle goes rather than the end.
This repository is public and the fixture named a real internal CI service account alongside the gateway's header, which together describe how model access is fronted. The test is about parsing `Key: value` lines; the values were never the point.
Every bug in this branch was found by running against a real container, and none of them would have been caught by the unit suite. `mockllm` reaches no provider, so the machinery around the model can be driven for free, with no key, on a pull request from a fork -- on Ubuntu, where Docker is the default sandbox, and on Windows, where there is none and the cross-platform tools are the only thing that works. The assertions are about the harness rather than the skill, because the mock satisfies nothing. It checks that no case errored, that the sandbox that ran is the one the platform should have chosen, and that no check failed because the sandbox could not be listed -- which otherwise reports identically to an agent that did nothing. The load-bearing one is the seeded fixture. The file exists because the case staged it, not because the agent acted, so `files_exist` passing proves the whole path: staged into the working directory, listed there, and matched against what the case asked for. That is exactly the chain that was broken in a container an hour ago.
Two failures on the first run of the new job, both worth having. The sandbox-selection tests asserted `docker` on any host, which is only true on a POSIX one -- on the Windows runners the answer is `local`, because inspect's sandbox layer assumes a POSIX guest. They pin the platform now and assert both answers, which is what they should have said in the first place. The engine job pointed at its fixture with `working-directory`, but a `--skills-dir` glob that was passed is resolved against the repo root, and `find_root` takes that from the nearest `.git`. Running from a subdirectory of a checkout still globbed the checkout, so the fixture was never found. `SKILLSCOPE_REPO` is the documented way to say which repo is under test, and is what the action already uses. The local reproduction missed this because the throwaway fixture was not inside a git repository, so the root happened to be the fixture.
The CI pin coerces any non-opus model to opus so paid runs stay comparable between runs. `mockllm` reaches no provider, grades nothing and costs nothing, so pinning it turned the free wiring check into a run that needed a key -- in the one environment where not needing a key is the entire point. The new engine job failed on exactly that.
`mockllm` never calls the submit tool, so the agent looped to the full 120-message budget and every turn was a real sandbox round trip. The CI grade step took 43 seconds to learn what it knows in three turns, and the same thing made a mock run over a real catalogue take 157 seconds for three cases. A model that reaches no provider now gets six turns. The job's grade step drops to about two seconds and the assertion it rests on -- the seeded fixture being found -- is unchanged. Also caches the pip download. inspect-ai pulls in the order of eighty packages, which was the other twenty seconds.
A trial on a self-hosted runner installed the podman binary, selected `SKILLSCOPE_SANDBOX=podman`, and failed with a ValueError from inspect's registry naming neither the variable that chose the provider nor the package that supplies it. The binary being present proves nothing: inspect resolves a third-party provider through an entry point, so the Python package has to be installed too, and that is the `[podman]` extra rather than `[inspect]`. Checked once before any container starts, with the install line in the message. The resolver is injectable so this is testable without the inspect extra, which the unit suite runs without on purpose.
A `react` agent delivers its answer through the submit tool, which lands in `output.completion`. The judge read the last assistant message instead, and that is often the preamble -- so an expectation about what the agent told the user could be graded against "the commands are below" rather than the commands. On a real skill this showed up as the judge reporting that the final message "only describes having printed registration/test commands but contains no actual curl commands", which is exactly what it was shown. Falls back to the last assistant message for agents with no submit tool.
Three things, each found by the previous one failing on a real runner: the provider package, podman's own compose rather than the shim that delegates to Docker's, and a search registry because podman will not guess one for an unqualified image name. Worth the setup where the runner's user cannot reach the Docker socket, which is the case this came from.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A second eval engine, selected with
--engine inspectand off by default. The dataset format, the CLI, and the reports are unchanged; only the thing driving the agent differs.The legacy engine shells out to the
claudeCLI with permissions bypassed on the runner, and hand-rolls the machinery around it: workspace staging, an LLM judge with bespoke verdict parsing, and ~850 lines of streaming and SIGKILL logic to end routing runs early. This hands that work toinspect_ai.Its agent is harness-independent rather than the
claudeCLI, so a skill is graded on whether its instructions work rather than on how one product reads them. That is the stronger claim, and the one a product repo can adopt. It also removes what blocked Windows: with no CLI in the sandbox there is no agent bridge, and that bridge is Linux-only.Both engines produce the same outcome objects, so
summarize,render_markdownand the report writers are untouched.Risk: low while it stays opt-in. The default is
legacyand that path is unchanged.Evidence
Routing over a whole catalogue (8 skills installed together, 68 cases):
All 12 disagreements favour this engine and none go the other way — 8 legacy misses, 3 legacy over-triggers, 1 wrong skill, 1 error. Noise would be symmetric.
These are single runs. Repeating the routing leg on a five-skill room gave 46/46, then 45/46, then 46/46 — so this engine has roughly 2% run-to-run variance of its own. Well inside legacy's, but not zero, and worth knowing before anyone treats a single number as a baseline.
The more interesting number is the noise floor: the legacy engine disagrees with itself on 7 of 68 cases (~10%) across two identical runs. That is a property of the status quo rather than of this change, and it is larger than most real regressions would be.
One caveat stated plainly: "legacy misses 8 activations" is probably not "the product harness routes worse". Legacy kills each case after 4 non-bookkeeping tool calls and infers activation from the stream, so an agent that plans before activating is recorded as a miss. Some of that gap is the legacy measurement being lossy. Still an argument for this engine, but a different claim.
tools/benchmark_engines.pyproduces all of the above, and refuses to present the token columns as comparable, because they are not.Non-obvious decisions
bash(),text_editor()and friends), which the Windows legs do not have.machine.ymlgainssandbox:, naming a compose file for a skill that needs network egress or a device.SKILLSCOPE_SANDBOXselects the provider (docker,podman,local) without discarding what the skill declared.Podman needs three things beyond the binary, each found by the previous one failing on a real runner: the
[podman]extra (the provider is registered by a package, not the binary),podman-composeplusINSPECT_PODMAN_COMPOSE(barepodman composedelegates to Docker's plugin, which then needs the daemon podman was chosen to avoid), and a search registry (podman will not guess Docker Hub for an unqualified image name). All three are indocs/usage.md. Worth it where a runner's user cannot reach the Docker socket, which is the case this came from.Test plan
268 unit tests, Linux and Windows
On self-hosted runners, against a live catalogue, through an enterprise gateway: routing 46/46, and behavioral 3/3 cases with 21/21 expectations met, isolated in a podman sandbox. That is the whole stack — tools, scorers and the LLM judge — on the infrastructure that would host it.
New CI job drives the engine on
mockllm— free, no key, works from a fork — and is green on both: Ubuntu reportssandbox: docker, Windows reportssandbox: local. Its load-bearing assertion is a seeded fixture file: it exists because the case staged it, not because the agent acted, sofiles_existpassing proves staging, working directory, listing and matching all hold. That is the exact chain that was broken in a container earlier in this branch.Two consequences worth naming. The Docker sandbox is now exercised, not just podman. And the Windows leg runs the cross-platform tools on Windows, including the PowerShell listing path, which nothing else covers.
Not verified — why this is a draft
mockllmsatisfies nothing, so the LLM judge is only ever exercised by hand.--engine claude-codehas never executed. Wired, guarded, unrun.Bugs this found in itself
Every one came from running against a real container or a real runner, and none would have been caught by the unit suite: a failed sandbox listing reading as an idle agent; the container working directory defaulting to
/; artifact paths resolved against the wrong root; the judge never seeing the agent's final message; transcript truncation dropping the very action a check turns on; and a credentials guard firing on runs that reach no provider.The new CI job then found two more on its first run: tests that asserted a POSIX answer on any host, and the CI model pin coercing
mockllmto a real model — which turned the one check that needs no key into one that does.