Skip to content

FIX Retry out-of-range judge scores in scale scorers instead of failing - #2812

Merged
hannahwestra25 merged 9 commits into
microsoft:mainfrom
u7k4rs6:fix/scale-scorer-range-retry
Sep 29, 2026
Merged

hannahwestra25 merged 9 commits into
microsoft:mainfrom
u7k4rs6:fix/scale-scorer-range-retry

Conversation

@u7k4rs6

@u7k4rs6 Utkarsh Bahuguna (u7k4rs6) commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #2809.

Adds NumericRangeResponseHandler, a wrapper in the same style as TrueFalseResponseHandler. It raises InvalidJsonException when the value is non-numeric, non-finite, or outside [minimum_value, maximum_value], so the normal JSON retry handles it. SelfAskScaleScorer, SelfAskGeneralFloatScaleScorer and InsecureCodeScorer (fixed 0 to 1 range) now wrap their handler with it. That includes caller-supplied handlers, which matches what Likert and true/false do.

Scorer identifiers don't change since the handler isn't part of them, so eval hashes aren't affected. The judgment replay identifier now includes the range.

Tests: handler tests in test_response_handler.py, plus a retry test for each of the three scorers (a bad value then a good one returns the good score after two calls). The scorer tests fail on main. tests/unit/score and tests/unit/executor match main plus 14 new passing tests. The test_scorer_info and test_otel_tool_call_scorer failures I see locally happen on main too.

@hannahwestra25 hannahwestra25 self-assigned this Sep 24, 2026
Comment thread pyrit/score/float_scale/self_ask_general_float_scale_scorer.py

@hannahwestra25 hannahwestra25 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for contributing !

@hannahwestra25
hannahwestra25 added this pull request to the merge queue Sep 29, 2026
Merged via the queue into microsoft:main with commit de521f4 Sep 29, 2026
49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG SelfAskScaleScorer crashes on an out-of-range judge score instead of retrying

2 participants