Skip to content

feat: add code-review-calibration skill (non-default) - #512

Draft
rbren wants to merge 2 commits into
mainfrom
add-code-review-calibration-skill
Draft

rbren wants to merge 2 commits into
mainfrom
add-code-review-calibration-skill

Conversation

@rbren

@rbren rbren commented Aug 25, 2026

Copy link
Copy Markdown
Member
  • A human has tested these changes.

Why

The existing code-review skill reviews from general principles, so its feedback is much the same in every repository. What a reviewer actually needs is knowledge of how this codebase breaks and what "correct" looks like here - and that knowledge is recoverable from the repo's own commit log, bug fixes, reverts, and past PR reviews.

This skill generates that calibrated reviewer instead of hand-writing it.

Summary

  • Adds skills/code-review-calibration/SKILL.md, a skill that investigates a repository in two halves - how it breaks (hotspots, reverts, SZZ fix-inducing commits, temporal coupling) and how it works (conventions, testing strategy, CI boundary, ownership and blast radius) - then writes or updates .agents/skills/code-review.md with failure patterns that each cite two or more SHAs.
  • Registers it in marketplaces/openhands-extensions.json under code-quality. defaultEnabled is deliberately omitted, so the skill is opt-in from the catalog UI rather than seeded into new workspaces.
  • Regenerates skills/index.js and the README catalog table.

Issue Number

How to Test

npm run build:skills                      # regenerates cleanly, no diff
python scripts/sync_extensions.py --check # only the pre-existing issue-duplicate-checker warning
uv sync --group test && uv run pytest tests/test_skills_catalog.py tests/test_catalogs.py -q

Confirm the skill is registered but not default-enabled:

node -e "import('./skills/index.js').then(m=>{
  const s=m.SKILLS_CATALOG.find(x=>x.name==='code-review-calibration');
  console.log('found:', !!s, 'defaultEnabled:', s.defaultEnabled);
  console.log('in defaults:', m.DEFAULT_ENABLED_SKILL_NAMES.includes('code-review-calibration'));
})"

Expected: found: true defaultEnabled: undefined and in defaults: false, with DEFAULT_ENABLED_SKILL_NAMES unchanged at 11 entries.

To exercise the skill itself, enable it in a workspace with a repo that has real history and ask for a calibrated code review skill; it should produce .agents/skills/code-review.md where every failure pattern cites SHAs from that repo.

Video/Screenshots

n/a - skill content and catalog metadata only.

Notes

  • Em dashes in the source were converted to plain hyphens per the punctuation convention in AGENTS.md.
  • Per AGENTS.md, "defaultEnabled": false is never written - absence is how a skill stays off - so the entry simply omits the key.
  • The not in any marketplace: ./plugins/issue-duplicate-checker warning from the sync script predates this branch and is unrelated.

This PR was created by an AI agent (OpenHands) on behalf of @rbren.

Adds skills/code-review-calibration, a skill that mines a repository's
own commit history, bug fixes, reverts, conventions, testing strategy,
CI coverage, and past PR reviews to generate a calibrated
.agents/skills/code-review.md with cited failure patterns.

Registered in marketplaces/openhands-extensions.json without
defaultEnabled, so it stays opt-in from the catalog UI.

Co-authored-by: openhands <openhands@all-hands.dev>
@rbren rbren changed the title Add code-review-calibration skill (non-default) feat: add code-review-calibration skill (non-default) Aug 25, 2026
CI requires every skill directory to ship a README.md and a
.plugin/plugin.json with vendor symlinks so Codex and Claude Code can
load it.

Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions github-actions Bot added the type: feat A new feature label Aug 25, 2026

@enyst enyst left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm an AI agent (Claude Code, based on Opus 5.5) helping Engel Nyst (@enyst) with project work.

Leaving a comment because this is a draft and about 80 commits behind main. The approach complements code-review well: mine the repo's own history for review rules and cite SHAs for each one. A few things need settling before it's ready:

  1. The output path and skill name bypass the established custom-guide mechanism. SKILL.md:508-514 writes .agents/skills/code-review.md with name: code-review. The documented way to add repo-specific review guidance is .agents/skills/custom-codereview-guide.md, with a unique name and the /codereview trigger. The point of that setup is that it supplements the public code-review skill instead of overriding it (see plugins/pr-review/README.md "Customizing Review Guidelines" and the footer in skills/code-review/SKILL.md). skills/github-pr-reviewer also auto-loads only that path (REPO_REVIEW_GUIDE_PATH). As written, the generated file shadows the public skill by name, and the PR reviewer automation never picks it up. I suggest generating or updating custom-codereview-guide.md with triggers: [/codereview] and changing lines 7, 23 and 73 to match.
  2. Overlap with skills/learn-from-code-review. That skill already turns PR review comments into repo skills and AGENTS.md guidance (written under .openhands/skills/), which is what Phase 9 does here too. Either build on it, or explain in the description and README how the two differ. Otherwise the catalog offers two skills that write competing review guidance to different places.
  3. Housekeeping. README.md conflicts with main in the catalog table. Rebase, then rerun python scripts/sync_extensions.py and npm run build:skills. The PR body predates the current template, so un-drafting needs the HUMAN:/AGENT: sections and a linked issue labeled ready-for-dev.

Minor, not blocking: the Phase 0 language breakdown at L89 uses \+ in a basic regex. That is a GNU extension, and on macOS/BSD sed the command prints nothing (checked locally). sed -nE 's/.*\.([a-zA-Z0-9]+)$/\1/p' works with both.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: feat A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants