Skip to content

Don't record an overlay status when the job was cancelled - #4122

Open
henrymercer wants to merge 2 commits into
mainfrom
henrymercer/friendly-potato
Open

Don't record an overlay status when the job was cancelled#4122
henrymercer wants to merge 2 commits into
mainfrom
henrymercer/friendly-potato

Conversation

@henrymercer

Copy link
Copy Markdown
Contributor

The overlay status mechanism records a marker in the Actions cache when building an overlay-base database fails, so that later runs skip overlay analysis instead of failing the same way, but it records one whenever the analyze step did not report success, which also covers runs that were cancelled, since the init post step still runs in that case. A cancellation tells us nothing about whether the analysis would have succeeded, and a marker disables overlay analysis for the whole repository until the next CodeQL release, so this adds an internal job-status input defaulting to ${{ job.status }} and skips recording when the job was cancelled.

Risk assessment

For internal use only. Please select the risk level of this change:

  • Low risk: Changes are fully under feature flags, or have been fully tested and validated in pre-production environments and are highly observable, or are documentation or test only.

Which use cases does this change impact?

Workflow types:

  • Advanced setup - Impacts users who have custom CodeQL workflows.
  • Managed - Impacts users with dynamic workflows (Default Setup, Code Quality, ...).

Products:

  • Code Scanning - The changes impact analyses when analysis-kinds: code-scanning.

Environments:

  • Dotcom - Impacts CodeQL workflows on github.com and/or GitHub Enterprise Cloud with Data Residency.
  • GHES - Impacts CodeQL workflows on GitHub Enterprise Server.

How did/will you validate this change?

  • Test repository - This change will be tested on a test repository before merging, specifically to confirm that a cancelled run reports cancelled to the post step.
  • Unit tests - I am depending on unit test coverage (i.e. tests in .test.ts files).

If something goes wrong after this change is released, what are the mitigation and rollback strategies?

  • Feature flags - The behaviour change only takes effect where overlay_analysis_status_save is enabled, so disabling that flag restores the previous behaviour. The new input is inert on its own.

How will you know if something goes wrong after this change is released?

  • Telemetry - I rely on existing telemetry. The codeql-action/overlay-disabled diagnostic reports skipped-due-to-cached-status, so the rate of that reason shows whether we are recording fewer markers as intended, and overlay adoption shows whether we have stopped recording ones we needed.

Are there any special considerations for merging or releasing this change?

  • No special considerations - This change can be merged at any time.

Merge / deployment checklist

  • Confirm this change is backwards compatible with existing workflows.
  • Consider adding a changelog entry for this change. Not added, since the behaviour is gated behind an existing feature flag, matching earlier changes to this mechanism.
  • Confirm the readme and docs have been updated if necessary.

henrymercer and others added 2 commits September 3, 2026 18:21
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@henrymercer
henrymercer requested a balanced review from Copilot September 3, 2026 17:24
@github-actions github-actions Bot added the size/M Should be of average difficulty to review label Sep 3, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

  • Copilot's review of this pull request may be incomplete because some of the changed files are excluded by your Copilot content exclusion settings. See Excluding content from Copilot for details.

Copilot review overview

🟢 Approval recommended

The focused implementation is consistent with existing input handling and covers the new behavior with unit tests.

Review tier: Balanced
Findings: None

What changed in this PR

Prevents cancelled jobs from incorrectly disabling overlay analysis through cached failure markers.

Changes:

  • Passes the runtime job status into post-processing.
  • Skips overlay failure recording for cancellations unless CodeQL already reported an error.
  • Adds unit coverage for cancellation scenarios.
File Description
src/​init-action-post.ts Captures post-step job status before restoring inputs.
src/​init-action-post-helper.ts Adds cancellation-aware overlay status handling.
src/​init-action-post-helper.test.ts Tests cancellation and prior-error behavior.
init/​action.yml Defines the internal job-status input.
lib/​entry-points.js Generated output; excluded from review by policy.
Files excluded by content exclusion policy (1)
  • lib/entry-points.js

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@henrymercer
henrymercer marked this pull request as ready for review September 3, 2026 17:36
@henrymercer
henrymercer requested a review from a team as a code owner September 3, 2026 17:36

@oscarsj oscarsj left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch

@henrymercer
henrymercer added this pull request to the merge queue Sep 4, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 4, 2026
@henrymercer
henrymercer added this pull request to the merge queue Sep 4, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 4, 2026

@mbg mbg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Broadly looks good and makes sense, thanks! Just a few considerations that came to my mind while reviewing this.

Comment thread init/action.yml
[Internal] The ID of the check run, as provided by the Actions runtime environment. Do not set this value manually.
default: ${{ job.check_run_id }}
required: false
job-status:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never nice to have to feed this in through an extra action input, but I don't see a better approach for getting hold of the job status. An alternative option might be to set CODEQL_ACTION_STEP_(init|analyze|...) environment variables / state that we set to e.g. starting when the respective action starts and then to success or failure depending on the outcome. That should then allow us to identify which step started, succeeded, or failed (gracefully or not). For the overlay status, we could then check that all available environment variables with a CODEQL_ACTION_STEP_ prefix are success and none are starting or failure. The downside is that it wouldn't catch if the failure isn't related to what happens in CodeQL Action steps, or we fail to even set the starting value.

// a failure would disable overlay analysis needlessly. Note that we still record a failure if one
// of our own Actions reported an error before the run was cancelled.
if (
jobStatus?.trim().toLowerCase() === "cancelled" &&

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than checking for the case where we don't want to record the overlay status, would it make sense to make this a check for the cases where we do want to record the overlay status?

That would be more defensive and guard against the hypothetical case of a new status value that we don't recognise.

I would also expect that we'd have a jobStatus value here unless something broke badly?

* it can gather telemetry, does not get to report anything.
*/
function didCodeQlReportError(): boolean {
const jobStatus = process.env[EnvVar.JOB_STATUS];

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a ReadOnlyEnv parameter and use getOptional.

Comment on lines +426 to +429
return (
jobStatus === JobStatus.FailureStatus ||
jobStatus === JobStatus.ConfigErrorStatus
);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we end up with one of these if the workflow job was cancelled? E.g. because it caused a thread abort style exception to be thrown at an inconvenient moment?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Should be of average difficulty to review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants