[feature] let a classifier judge fail open on client failures and deadlines (judge_failure = "fail_open")

[himorishige]

Problem

A capability-mode llm_classifier route stops the whole request when its judge cannot be reached. Since #702 the HTTP driver treats every routing-time client failure as terminal, so a judge that is down returns 502 upstream_error and a judge that stalls past timeout_ms returns 504 upstream_timeout. Only an unparseable verdict still fails open.

That contradicts what the rest of the code base says the judge is:

• crates/libsy/src/algorithms/util/llm_judge.rs (verdict()): "A judge is an optimization, not a dependency: failing the caller's request because the judge is down would be worse than routing without it, so every failure — transport, mid-stream, or unparseable reply — is logged and folded into None". The fold runs, but the driver has already been stopped by serve() in libsy-llm-client/src/run.rs (Err(error) => call.fail(error)), so the request errors anyway.
• docs/reference/opentelemetry.md lists timeout, transport, upstream_5xx and upstream_non_5xx as switchyard.classifier_fail_open reasons (client_error_reason in llm_judge.rs). None of them can be observed through the server today: the counter increments, then the request fails.
• advisor_gate documents fail_open = true (its default) as "an advisor failure degrades to APPROVE", while crates/libsy-llm-client/README.md says a client failure stops the run "also when an advisor has fail_open = true".
• feat(libsy): bound judge consultations with a configurable deadline #346 proposed a judge deadline that falls back "through exactly the same fail-open path as any other judge failure" and was closed in favour of feat: bound how long a routed request may take #277, which is still open.

So the current state reads as "deadline first (#702), fail-open posture still to be decided (#277)". This issue is to ask what that posture should be, with data from a deployment where the judge is a local model in front of a coding agent.

What we observed (0.3.0, /v1/decision, fault drills on a sidecar instance)

Route: llm_classifier / capability, classify_trigger = "user_turn", judge on its own [llm_clients] entry with timeout_ms = 3000, weak/strong on a hosted provider.

Drill	Judge state	Response	libsy.run span
baseline	up	200, efficient tier, 0.28 s	evidence.source = llm-classifier, score 0.87
judge answers a valid verdict with p_solve = 0.0 (proxy in front of a stopped model)	reachable	200, capable tier, 0.05 s	llm-classifier, score 0, threshold 0.95
judge process paused	reachable, stalled	504 response did not finish within 3000 ms, retries included after 3.03 s	no evidence, outcome = error
judge process stopped	unreachable	502 error sending request after 0.8 s	no evidence, outcome = error

The second row shows the behaviour operators expect from the first quote: the judge is unavailable, the request goes to the capable tier, the evidence records why. Rows three and four show the same situation, one hop earlier, turning into a client-visible error. For a coding agent that means a dead judge (or one restarting after an image pull, several minutes for a 30B model) fails every turn of every session until it is back, while switchyard.classifier_fail_open keeps counting fail-opens that never routed.

Proposed solution

Make the posture explicit and per judge, keeping #702 as the default so nothing changes for existing configurations:

[routes.auto]
type = "llm_classifier"
mode = "capability"
# "error" (default, today's behaviour): a judge client failure or deadline fails the request.
# "fail_open": a judge client failure or deadline routes without a verdict, like an invalid verdict does.
judge_failure = "fail_open"

Mechanically this only needs the driver to hand a failed routing-time call back to the algorithm instead of stopping the run when the algorithm asked for that — e.g. a flag on CallModel set by JudgeClassifier (and by advisor_gate when fail_open = true), honoured in serve() as call.respond(Err(error)) instead of call.fail(error). Terminal answer calls and algorithms that propagate the error (?) keep #702 semantics, so a stalled answer still returns 504 and never tries another candidate.

Two refinements that fit the same knob, if wanted:

• With classify_trigger = "user_turn" most turns already carry a session decision. On judge failure the route could keep the session's last tier (evidence.source = retained) and fall to the capable tier only when there is none.
• timeout_ms on the judge's client then becomes safe to tighten (a 1 s deadline costs at most 1 s of latency, not the request), which is the case feat(libsy): bound judge consultations with a configurable deadline #346 was written for.

Alternatives considered

• Keep fix(client): enforce response deadlines and stop routing on errors #702 and put a translating proxy in front of every judge that answers p_solve = 0.0 on upstream failure (row two above). Works, but it moves a routing policy out of the router into a sidecar, and a failure of the proxy itself is still a 502.
• Restore the pre-fix(client): enforce response deadlines and stop routing on errors #702 behaviour unconditionally. Rejected: routes that use a judge as a safety or permission gate need fail-closed, and fix(client): enforce response deadlines and stop routing on errors #702's deadline semantics for answer calls should stay.
• Circuit breakers ([feature] Skip repeatedly unavailable routing candidates with circuit breakers #594) reduce the cost of a dead judge but do not change what the caller sees on the first failed request.

Scope notes

• Surface: routing algorithm (libsy: CallModel / JudgeClassifier / advisor_gate) plus the upstream client driver (libsy-llm-client serve()), and one deployment-TOML key on llm_classifier routes.
• Backward compatibility: default "error" keeps today's behaviour and the fix(client): enforce response deadlines and stop routing on errors #702 tests; fail_open is opt-in. The advisor_gate fail_open documentation and libsy-llm-client/README.md / toml_schema.md would need to agree again.

Additional context

• fix(client): enforce response deadlines and stop routing on errors #702 (deadline + stop routing on errors), feat(libsy): bound judge consultations with a configurable deadline #346 (judge deadline with fail-open, closed for feat: bound how long a routed request may take #277), feat: bound how long a routed request may take #277 (open), feat(libsy): count classifier fail-open fallbacks on /metrics #205 (classifier_fail_open metric), [feature] Skip repeatedly unavailable routing candidates with circuit breakers #594 (circuit breakers).
• The drills above were run with POST /v1/decision; the chat completion path uses the same run() driver.