feat: support chaining containerized index builds

[yashvardhannanavati]

Allow add, rm, and fbc-operations requests to use the output of a completed IIB request as their input index.

Resolve the immediate parent for catalog and index.db content while retaining the original ancestor's Git and Konflux scaffolding. Classify sources as standard, divergent, or chained, and permit merges only for standard builds.

Publish request-specific index.db artifacts using the final recorded output digest. Normalize new artifact payloads to index.db while supporting legacy divergent artifacts named extracted_index.db.

Reject overwrite options for chained requests and ensure chained and divergent merge requests are always closed without merging.

Add ancestry, artifact, extraction, preparation, handler, and compatibility coverage, plus containerized workflow documentation.

Assisted-By: OpenAI Sol

[qodo-for-releng]

PR Summary by Qodo

Support chaining containerized index builds

✨ Enhancement 🧪 Tests 📝 Documentation 🕐 40+ Minutes

AI Description

• Enable supported containerized requests to chain completed IIB output images.
• Reconstruct parent content while retaining original Git and Konflux scaffolding.
• Publish digest-keyed artifacts and prevent overwrites or merges for throw-away builds.

Diagram

graph TD
  A["Build Request"] --> B["Ancestry Resolver"] --> C["Request API"]
  B --> D["Ancestor Git"] --> G["Containerized Build"] --> H["Resolved Output"]
  C --> E["Parent Image"] --> G
  C --> F["ORAS index.db"] --> G

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Persist explicit parent request IDs

• ➕ Avoids deriving request identity from configurable image templates
• ➕ Makes lineage directly queryable without parsing pullspecs
• ➖ Requires API and database schema changes
• ➖ Needs migration and compatibility handling for existing completed requests

2. Extract all state from the parent image

• ➕ Removes the request-specific ORAS artifact dependency
• ➕ Uses one immutable source for catalog and database content
• ➖ Depends on every output image containing a usable hidden database
• ➖ Weakens compatibility with existing artifact-based database workflows
• ➖ Duplicates extraction behavior already handled by ORAS artifacts

Recommendation: The implemented hybrid approach is appropriate for backward-compatible delivery: it derives ancestry from existing output conventions, preserves original Git/Konflux onboarding, and reconstructs exact parent content from immutable image and ORAS outputs. Explicit lineage fields would be a worthwhile future evolution if schema changes become acceptable.

Files changed (14) +920 / -243

Enhancement (6) +310 / -97

build.pyReturn the recorded resolved output image +4/-1

Return the recorded resolved output image

• Changes output metadata publication to return 'index_image_resolved' for add/remove-style builds, allowing artifact tags to use the final recorded destination digest.

iib/workers/tasks/build.py

build_containerized_add.pyIntegrate chaining safeguards into add requests +7/-8

Integrate chaining safeguards into add requests

• Passes overwrite credentials into source resolution, keys database artifacts from the final resolved image, and permits MR merging only for standard sources.

iib/workers/tasks/build_containerized_add.py

build_containerized_fbc_operations.pyIntegrate chaining into FBC operation requests +7/-8

Integrate chaining into FBC operation requests

• Uses chain-aware source preparation and resolved output digests while preventing chained and divergent merge requests from merging.

iib/workers/tasks/build_containerized_fbc_operations.py

build_containerized_rm.pyIntegrate chaining into remove requests +7/-8

Integrate chaining into remove requests

• Propagates overwrite credentials for validation, publishes database artifacts against the recorded output digest, and applies source-based merge policy.

iib/workers/tasks/build_containerized_rm.py

containerized_utils.pyResolve chained ancestry and reconstruct parent build state +276/-72

Resolve chained ancestry and reconstruct parent build state

• Adds IIB output recognition, validated multi-hop ancestry resolution, chained source preparation, and standard/divergent/chained source classification. It also fetches request-specific database artifacts, supports legacy payload names, and normalizes new ORAS payloads to 'index.db'.

iib/workers/tasks/containerized_utils.py

oras_utils.pyCentralize request-specific artifact pullspec generation +9/-0

Centralize request-specific artifact pullspec generation

• Adds a helper that combines the content-addressed output digest tag with the producing request ID.

iib/workers/tasks/oras_utils.py

Tests (7) +592 / -146

test_build.pyVerify resolved output image propagation +24/-3

Verify resolved output image propagation

• Covers returned resolved digests, internal copy resolution, and unchanged behavior for request types that do not record resolved images.

tests/test_workers/test_tasks/test_build.py

test_build_containerized_add.pyCover chained add publication and merge policy +38/-14

Cover chained add publication and merge policy

• Updates source fixtures and verifies overwrite-token propagation, resolved-digest artifact publication, publication ordering, and non-mergeable chained sources.

tests/test_workers/test_tasks/test_build_containerized_add.py

test_build_containerized_create_empty_index.pyAlign artifact digest mocks with ORAS ownership +3/-3

Align artifact digest mocks with ORAS ownership

• Updates patch targets after digest derivation was centralized in the ORAS utility module.

tests/test_workers/test_tasks/test_build_containerized_create_empty_index.py

test_build_containerized_fbc_operations.pyCover chained FBC operation safeguards +37/-8

Cover chained FBC operation safeguards

• Verifies overwrite-token propagation, destination-digest artifact keys, publication ordering, and identical no-merge behavior for divergent and chained sources.

tests/test_workers/test_tasks/test_build_containerized_fbc_operations.py

test_build_containerized_rm.pyCover chained remove publication and merge policy +44/-15

Cover chained remove publication and merge policy

• Updates digest mocks and verifies source preparation credentials, final-destination artifact keys, publication ordering, and closure of non-mergeable merge requests.

tests/test_workers/test_tasks/test_build_containerized_rm.py

test_containerized_utils.pyAdd ancestry, extraction, artifact, and source preparation coverage +431/-103

Add ancestry, extraction, artifact, and source preparation coverage

• Adds comprehensive tests for output recognition, multi-hop validation, cycles, overwrite rejection, artifact compatibility, payload normalization, catalog extraction, source classification, and ancestor scaffolding selection.

tests/test_workers/test_tasks/test_containerized_utils.py

test_oras_utils.pyTest request-specific artifact references +15/-0

Test request-specific artifact references

• Verifies per-request artifact pullspecs combine configured content-addressed tags with request IDs.

tests/test_workers/test_tasks/test_oras_utils.py

Documentation (1) +18 / -0

README.mdDocument chained containerized build behavior +18/-0

Document chained containerized build behavior

• Explains supported request types, parent content sources, overwrite restrictions, throw-away merge-request behavior, and unsupported workflows.

docker/containerized/README.md

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 4:31 AM UTC · Completed 4:51 AM UTC

Commit: dff14a9 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $8.80

[qodo-for-releng]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Chained pipeline builds are rejected ✓ Resolved

Description

get_iib_output_request_id only full-matches the configured request-tag template, while
operator-pipelines records index_image_resolved and uses that digest pull specification as the
next removal request's from_index. Release pipelines also attach overwrite credentials, so these
inputs bypass chained resolution and fail during source preparation instead of reaching the new
chained path.

Code

iib/workers/tasks/containerized_utils.py[R81-85]

+        pattern = re.escape(rendered).replace(
+            re.escape(_REQUEST_ID_SENTINEL),
+            r'(?P<request_id>[0-9]+)',
+        )
+        match = re.fullmatch(pattern, image)

Relevance

●● Moderate

Concrete cross-repository compatibility risk, but no closely matching historical acceptance or
rejection precedent was found.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The PR detector full-matches only the rendered request-tag template. operator-pipelines writes
index_image_resolved into its shared output file, reloads that value as iib_build_image, prefers
it as the next input, and supplies overwrite fields when credentials are configured; therefore its
existing chained FBC-to-removal workflow cannot activate the new chained path.

iib/workers/tasks/containerized_utils.py[69-105]
iib/workers/tasks/containerized_utils.py[850-857]
External repo: redhat-openshift-ecosystem/operator-pipelines, operatorcert/entrypoints/add_fbc_fragments_to_index.py [224-242]
External repo: redhat-openshift-ecosystem/operator-pipelines, operatorcert/entrypoints/rm_operator_from_index.py [94-102]
External repo: redhat-openshift-ecosystem/operator-pipelines, operatorcert/entrypoints/rm_operator_from_index.py [132-142]
External repo: redhat-openshift-ecosystem/operator-pipelines, operatorcert/entrypoints/rm_operator_from_index.py [176-183]
External repo: redhat-openshift-ecosystem/operator-pipelines, ansible/roles/operator-pipeline/templates/openshift/tasks/build-fbc-index-images.yml [125-155]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
operator-pipelines passes a prior build's `index_image_resolved` as the next request's `from_index`, but chained-build detection only recognizes the configured request-tag form. Resolve digest-form IIB outputs to their completed parent request so these pipeline requests enter the chained path.

## Fix Focus Areas
- iib/workers/tasks/containerized_utils.py[69-150]

## Recommended Fix
Extend chained-source resolution to recognize both the configured request-tag pull specification and an exact recorded `index_image_resolved` value. Locate and validate the corresponding completed parent request, retain the existing ancestry checks, and add coverage using the digest-form output emitted by operator-pipelines.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Chained merge requests stay open ✓ Resolved

Description

cleanup_merge_request_if_exists swallows merge-request closure failures, while the new
merge_allowed branch routes every chained build through that helper. When GitLab returns an error
during successful chained add, remove, or catalog-operation builds, execution continues to the
complete state with the throw-away merge request still open.

Code

iib/workers/tasks/build_containerized_add.py[376]

+            if overwrite_from_index and sources.merge_allowed:

Relevance

●●● Strong

Accepted precedent supports fixing lifecycle failures that can leave requests or related resources
in inconsistent states.

PR-#1061

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new source classification makes chained builds non-mergeable and sends them to
cleanup_merge_request_if_exists; that helper catches IIBError and only logs a warning. Each
handler immediately marks the request complete after the helper returns, so a failed close is
reported as a successful chained build.

iib/workers/tasks/containerized_utils.py[787-809]
iib/workers/tasks/containerized_utils.py[1160-1177]
iib/workers/tasks/build_containerized_add.py[374-386]
iib/workers/tasks/build_containerized_rm.py[312-325]
iib/workers/tasks/build_containerized_fbc_operations.py[267-284]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Successful chained builds can be marked complete even when their required throw-away merge request cannot be closed because `cleanup_merge_request_if_exists` catches and suppresses the closure error.

## Fix Focus Areas
- iib/workers/tasks/containerized_utils.py[1160-1177]
- iib/workers/tasks/build_containerized_add.py[374-383]
- iib/workers/tasks/build_containerized_rm.py[312-321]
- iib/workers/tasks/build_containerized_fbc_operations.py[267-276]

## Recommended Fix
Make merge-request closure failure propagate after an appropriate bounded retry so handlers do not mark a chained or divergent request complete while its merge request remains open. Preserve the existing successful cleanup behavior and ensure all three handlers route the propagated error through their failure handling.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Cross-repo context — repo relationships

  Explored: repo: redhat-openshift-ecosystem/operator-pipelines (sha: 89e036a0)

Review mode: 🧠 Deep: This is a high-density behavioral change spanning ancestry resolution, artifact publication, Git/Konflux merge policy, extraction compatibility, and multiple request paths, creating many independent opportunities for subtle defects.

Tip of the day

💡 Did you know, you can keep summaries lean with Findings visible per group, which tucks the rest behind a View link

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[fullsend-ai-review]

Risk Assessment: moderate (2/5)

Details

Large-blast-radius feature addition (1311 lines, large blast radius) mitigated by a 50% test-file ratio, no protected or security-sensitive paths, no CI or dependency changes, low churn, zero regression history, and an established author.

Previous run

Risk Assessment: moderate (2/5)

Details

Signals are unchanged from the prior assessment -- large blast radius and high line count (1311) are the primary risk drivers, but 50% test file ratio, no security/dependency/CI changes, stable git history across the containerized files, and an experienced author keep the composite score at moderate (2).

Previous run (2)

Risk Assessment: moderate (2/5)

Details

Large blast radius and high line count (1163) across core containerized build utilities push the size signal to maximum, but strong test coverage (50% test-file ratio), no security or dependency changes, no CI modifications, and an experienced non-first-time author heavily mitigate that risk, yielding a moderate composite score of 2.

[fullsend-ai-review]

Review

Findings

Medium

• [missing-authorization] — This PR introduces a substantial new feature (chained builds) with no linked issue. The PR has been triaged (risk/moderate label), but linking an issue remains good practice for a feature of this scope.
  Remediation: Link this PR to an approved issue that explicitly authorizes the chained build feature.

• [unauthorized-change] iib/workers/tasks/containerized_utils.py:1200 — The raise_on_error=True parameter is now passed by all three handlers (add, rm, fbc-operations) on their non-overwrite/non-merge branch. This means standard builds with overwrite_from_index=False now fail the request if MR close fails, where previously the failure was only logged. This is an intentional, tested behavioral change, but it affects pre-existing paths and is bundled without being called out in the PR description.
  Remediation: Either scope raise_on_error=True exclusively to CHAINED/DIVERGENT source kinds, or explicitly document this behavior change in the PR description.

Low

• [documentation-comment-format] iib/workers/tasks/containerized_utils.py:69 — get_iib_output_request_id has a single-line summary docstring with no :param, :return:, or :rtype: sections. Other new public functions in this PR use the full Sphinx-style format.
  Remediation: Expand the docstring to include :param str image:, :return:, :rtype: Optional[int] sections.

• [documentation-comment-format] iib/workers/tasks/containerized_utils.py:91 — resolve_chained_build_source has only a one-line summary docstring. It accepts three parameters, returns Optional[ChainedBuildSource], and raises IIBError in multiple branches — all of which the established codebase pattern requires to be listed explicitly.
  Remediation: Add :param, :return, :rtype, and :raises entries to the docstring.

• [scope-creep] iib/workers/tasks/containerized_utils.py:705 — The push_index_db_artifact refactor normalizes all ORAS payloads to the filename index.db regardless of the source file name, and replaces inline tag-derivation logic with helper calls. This affects all three existing build paths (standard, divergent), not only chained builds.
  Remediation: Document the filename normalization as an explicit part of the authorized scope.

• [fail-open] iib/workers/tasks/containerized_utils.py:112 — resolve_chained_build_source has no maximum chain depth limit. Each iteration makes an API call via get_request(). Cycle detection ensures termination, but an arbitrarily long chain could cause elevated latency.
  Remediation: Add a maximum chain depth constant (e.g., MAX_CHAIN_DEPTH = 100) and raise IIBError if exceeded.

• [naming-convention] iib/workers/tasks/containerized_utils.py:797 — BuildSourceKind inherits from (str, Enum), a pattern not used anywhere else in the codebase. Existing enums use BaseEnum(Enum).
  Remediation: Change to class BuildSourceKind(Enum): to match the existing pattern, or document why (str, Enum) is needed here.

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

Previous run

Review

Findings

Medium

• [missing-authorization] — No linked issue for this non-trivial feature spanning 14 files and ~600 net new lines. The PR body is detailed but contains no issue reference, ticket number, or other authorization signal. Non-trivial changes require an explicit authorization trail.
  Remediation: Link the PR to the authorizing issue (e.g. Closes #NNN or Ref #NNN). If no issue exists, create one describing the requirement and cross-reference it before merging.

Low

• [fail-open] iib/workers/tasks/containerized_utils.py:112 — The chain resolution loop in resolve_chained_build_source() has no configurable depth limit. Cycle detection (via seen set) prevents infinite loops, but a deep chain causes O(N) sequential get_request() API calls with no ceiling. In practice, each link requires a full completed IIB request (authenticated and resource-intensive), so the amplification factor is approximately 1x.
  Remediation: Add a configurable maximum chain depth (e.g., from worker config) and reject chains that exceed it.

• [edge-case] iib/workers/tasks/containerized_utils.py:126 — resolve_chained_build_source performs a strict string equality check between the user-supplied from_index and the stored index_image. When iib_index_image_output_registry is configured, a user passing the internal registry URL gets a confusing "does not match" error. The behavior fails safe (rejects rather than proceeding incorrectly).

• [naming-conventions] iib/workers/tasks/containerized_utils.py:108 — seen: set[int] = set() uses PEP 585 lowercase generic syntax while the rest of the file consistently uses typing module generics (List, Dict, Optional, Tuple).
  Remediation: Add Set to the from typing import line and change to seen: Set[int] = set().

• [behavioral-change-undocumented] docker/containerized/README.md:275 — The divergent-tag section states the throw-away MR "is always closed after the pipeline completes (or on failure), regardless of outcome." After this PR, cleanup_merge_request_if_exists is called with raise_on_error=True on the success path; an MR close failure now fails the request instead of being swallowed with a warning.
  Remediation: Revise to clarify that MR closure failure on the success path now fails the request.

• [scope-coherence] iib/workers/tasks/build.py:39 — _update_index_image_pull_spec() return type broadened from None to Optional[str] in the same PR that introduces the chaining feature. The change is minimal (one line) and tested, but bundles a contract modification with new feature work.

• [abstraction-trajectory] iib/workers/tasks/containerized_utils.py:817 — merge_allowed property encodes merge policy on the BuildSources dataclass. Architecturally sound for the current three-kind taxonomy (STANDARD/DIVERGENT/CHAINED).

• [scope-coherence] iib/workers/tasks/containerized_utils.py:1012 — prepare_git_repository_for_build() gains a guard rejecting IIB output images, extending behavior to create-empty-index beyond the PR description's stated scope of add, rm, and fbc-operations. The guard is tested in test_containerized_utils.py.

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

Previous run (2)

Review

Findings

Medium

• [error-handling-idiom] iib/workers/tasks/build_containerized_add.py:355 — Uses assert index_image_resolved is not None in production runtime code across four locations (build_containerized_add.py:355, build_containerized_fbc_operations.py:248, build_containerized_rm.py:293, containerized_utils.py:143). The codebase consistently raises IIBError for runtime guard checks; assert statements are silently removed when Python runs in optimized mode (-O).
  Remediation: Replace all four assert ... is not None occurrences with if ... is None: raise IIBError(...) checks.

• [scope-coherence] iib/workers/tasks/containerized_utils.py:987 — No explicit guard prevents IIB output URLs from being passed to unsupported request types (merge-index-image, create-empty-index, regenerate-bundle, legacy workers). Those handlers call prepare_git_repository_for_build which doesn't invoke resolve_chained_build_source, so users get a confusing "Git repository mapping not found" error instead of a clear "chaining not supported" message.
  Remediation: Add an explicit check that calls get_iib_output_request_id and raises IIBError if the from_index is an IIB output URL.

• [documentation-comment-format] iib/workers/tasks/containerized_utils.py:572 — fetch_and_verify_request_index_db_artifact has only a one-line docstring while every comparable function in the same file uses full Sphinx-style blocks with :param:, :return:/:rtype:, and :raises: sections.
  Remediation: Expand the docstring to include :param:, :return:, :rtype:, and :raises IIBError: entries.

Low

• [missing-authorization] iib/workers/tasks/containerized_utils.py — This PR introduces a substantial new feature (14 files, ~600 net lines) with no linked issue. Non-trivial changes benefit from an explicit authorization trail to an approved issue or design document.
  Remediation: Link a GitHub issue capturing requirements and acceptance criteria.

• [edge-case] iib/workers/tasks/containerized_utils.py:112 — The chain traversal loop in resolve_chained_build_source has no depth limit. While cycle detection prevents infinite loops, a legitimately deep chain would issue one get_request() API call per ancestor. Practical risk is low since creating a deep chain requires completing N prior builds.
  Remediation: Add a maximum chain depth constant (e.g., _MAX_CHAIN_DEPTH = 100) and raise IIBError when exceeded.

• [naming-conventions] iib/workers/tasks/containerized_utils.py:108 — seen: set[int] uses PEP 585 built-in generic form while the rest of the file uses typing imports (Dict, List, Optional, Tuple, Union). This is the only occurrence of a lower-case generic annotation in the file.
  Remediation: Use Set[int] from typing for consistency.

• [documentation-comment-format] iib/workers/tasks/oras_utils.py:718 — get_request_indexdb_artifact_pullspec has a bare one-liner docstring; the preceding peer function has full Sphinx-style entries.
  Remediation: Add :param:, :return:, :rtype: entries to the docstring.

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 5:47 AM UTC · Completed 6:10 AM UTC

Commit: 6728984 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $9.73

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 5:52 AM UTC · Completed 6:13 AM UTC

Commit: c1b7a44 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $10.05

[fullsend-ai-review]

See the review comment for full details.