Skip to content

fix(stale-ci): age failures by required check result #683

Description

@neubig

Problem

The stale-CI automation selects new candidates with GitHub's pull-request updated_at timestamp. Comments and reviews update that timestamp, so an automated reviewer comment can hide a PR whose required CI has already been failing for weeks.

An audit on 2026-09-24 found 43 open, non-draft PRs whose newest required-check failure was more than seven days old, but only seven were tracked. Thirty-six were excluded because recent comments or reviews refreshed the PR timestamp.

Acceptance criteria

  • Candidate age comes from the latest result of the checks required by the PR's base branch, not the PR updated_at value.
  • A PR is warned once its required CI has been definitively failing for seven days.
  • Pending or missing required checks, drafts, and repositories without configured required checks remain excluded.
  • Optional failures do not start the lifecycle.
  • A new head resets the lifecycle to the timestamps of that head's required checks.
  • Candidate discovery remains bounded enough for an hourly automation across the four OpenHands OSS repositories.
  • Tests cover a recent bot comment on an old required-check failure and mixed-age required failures.

OpenHands AI triage

The following comments and acceptance criteria were added by the OpenHands AI agent.

Triage

This is a follow-up defect in the already-merged github-stale-ci-pr-closer automation (issue #670, PR #680), which owns the deterministic scan in skills/github-stale-ci-pr-closer/scripts/worker.py and ships it through automations/catalog/github-stale-ci-pr-closer/manifest.json. stale_failure_numbers selects new candidates with a GitHub search constrained by updated:<cutoff, so any comment or review refreshes updated_at and hides a PR whose required checks have been failing far longer. The fix is to age each candidate from the newest result of the checks required by its base branch, using the existing shared skills/github/scripts/github_client.py transport and the existing KV/file state.

Scope: change candidate selection and failure aging only; the warning and close lifecycle, comment markers, close conditions, state mechanism, and credential scopes stay as they are, and the pass still creates no conversation and consumes no LLM slot.

Non-goals: no change to warning/close semantics, marker text, or the seven-day windows; no new automation, catalog entry, or integration; no change to which PRs are eligible beyond correctly aging them; no merge, label, or review side effects. (The linked open PR #684 already fixes this under the accepted criteria below.)

Acceptance Criteria

  • Candidate age is derived from the newest result among the checks required by the PR's base branch, not the PR updated_at: a non-draft PR whose required check most recently failed more than seven days ago is selected and warned even when a comment or review refreshed updated_at within the last seven days (observable in run output and in a focused test).
  • When the required checks for a head failed at different times, the age comes from the most recent required-check failure, so a PR with one long-failing and one recently-failing required check is not treated as older than its newest failure.
  • The following produce no warning and no close: a base branch with no required checks configured, required checks that are pending or missing a result, and a draft PR.
  • A failing optional (non-required) check does not start the warning or close lifecycle.
  • A new head resets the lifecycle to the timestamps of that head's required checks, so no warning or close is attributed to an earlier head's failure time.
  • Existing persisted per-repository state from the previous release is re-aged from the newest required-check failure instead of the prior first-observation time, and any already-posted warning for the current window is not duplicated.
  • Warning and close behavior is otherwise unchanged: the same warning/close markers, the same seven-day windows, and the same authorization scopes; the run still creates no agent conversation and issues no LLM call.
  • Candidate discovery for the four configured OpenHands OSS repositories stays bounded for an hourly run, and a live run over those repositories completes within the automation's configured timeout.
  • automations/catalog/github-stale-ci-pr-closer/manifest.json bumps version and bundle.version and its exampleImplementation (and the skill's documentation) describe aging from the newest required-check failure; the generated automations/bundle-index.js is regenerated so it matches the shipped worker.py.
  • Focused tests cover a recent bot comment on an old required-check failure, mixed-age required failures, and the pending/draft/optional/unconfigured exclusion cases.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority:lowready-for-devScoped for contribution; managed by repository readiness checks.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions