AppWrapper fails with MissingComponent immediately after successfully creating its Job

[mike-ainsel]

Describe the Bug

An AppWrapper is failed terminally with MissingComponent immediately after it has
successfully created its child Job.

The controller transitions Resuming → Running → Failed in roughly 660 ms, reporting:

MissingComponent: Only found 0 deployed components, but was expecting 1

The Kubernetes audit log shows the child Job creation returned success 119 ms before
the controller reported zero deployed components. The workload is then failed with no
grace period and no retry, and the Job the controller created is deleted.

It occurs intermittently and appears correlated with controller restarts. We suspect
getComponentStatus is reading through the manager's cached client, and that a Job which
has been created but has not yet propagated to the informer cache is indistinguishable
from one that was externally deleted — but we would like that confirmed rather than
assume we have read the code correctly.

Codeflare Stack Component Versions

Please specify the component versions in which you have encountered this bug.

Codeflare SDK: not used
MCAD: not used
Instascale: not used
Codeflare Operator: not used
Other:

• AppWrapper v1.2.0, installed from the released install.yaml
• Kueue v0.16.1 (drives admission)
• Deployment as shipped: replicas: 1, container args ["--zap-log-level=2"] (no --leader-elect)

Steps to Reproduce the Bug

Probabilistic, but reliable enough to hit within seconds. In our run 1 of 20 workloads
failed, 4 seconds after step 3.

1. Install AppWrapper v1.2.0 and Kueue v0.16.1, with a ClusterQueue and a LocalQueue for
   batch work.
2. Submit ~20 AppWrappers at once, each wrapping a single batch/v1 Job with small
   resource requests (resource pressure is not needed — we used 200m CPU / 512Mi), each
   labelled kueue.x-k8s.io/queue-name: <local-queue>.
3. While Kueue is still admitting them, kubectl rollout restart deployment/appwrapper-controller-manager -n appwrapper-system
   so the controller starts with a cold informer cache.
4. Within a few seconds, at least one AppWrapper reaches Failed with
   MissingComponent: Only found 0 deployed components, but was expecting 1.

Submitting the same 20 workloads without the restart did not reproduce it — Kueue
admitted them gradually and all ran normally. The restart appears to be what widens the
window.

What Have You Already Tried to Debug the Issue?

• Correlated the controller log against the Kubernetes audit log for the same Job name,
  and confirmed the jobs.create call returned status=0 (success) 119 ms before the
  controller reported zero components.
• Found a second signal pointing the same way: on one occurrence the controller issued
  two jobs.create calls for the same Job 245 ms apart, the first returning success
  and the second ALREADY EXISTS — suggesting it did not observe its own successful
  write.
• Confirmed the jobs.delete that follows is the controller's own cleanup after
  transitioning to Failed, not an external actor removing the component.
• Ruled out admission rejection: no denials from the cluster's policy webhooks, and no
  create failures in the audit log.
• Ruled out resource exhaustion of the controller: terminations were exitCode 1 /
  reason: Error (a startup configuration failure while the API server was briefly
  unreachable), never OOMKilled. Raising the controller memory limit did not change
  this behaviour.
• Read getComponentStatus and the AppWrapperRunning branch at tag v1.2.0, which is
  where our suspicion comes from.
• Compared v1.2.0...v1.2.2 (18 commits): dependency bumps, CI matrix updates and an
  Event/Eventf → EventRecorder migration. Nothing appears to touch component
  counting or the failure semantics, so we do not believe upgrading resolves it — please
  correct us if that is wrong.

Expected Behavior

A component that exists in the cluster but is not yet visible to the controller's
informer cache should not be treated as externally deleted, and the workload should not
be failed terminally on that basis.

We understand the check is there to detect a component someone removed out from under
the AppWrapper, and that such a situation will not self-correct — hence the deliberate
absence of a grace period. The case we are hitting does self-correct, milliseconds
later.

Screenshots, Console Output, Logs, etc.

Controller log for one failure (timestamps truncated to seconds.milliseconds):

25.021  Finalizer Added
25.067  Resuming
25.453  Resuming
25.636  Running
25.659  Running
25.660  MissingComponent: Only found 0 deployed components, but was expecting 1
25.683  Failed
25.972  Finalizer Deleted

Kubernetes audit log, same Job name, same second:

25.541  io.k8s.batch.v1.jobs.create   appwrapper-controller-manager   status=0   SUCCESS
25.727  io.k8s.batch.v1.jobs.delete   appwrapper-controller-manager   status=0   SUCCESS

So the Job existed from 25.541 and the controller reported it missing at 25.660.

A separate occurrence, showing the duplicate create:

12:52:12.980  io.k8s.batch.v1.jobs.create   status=0    SUCCESS
12:52:13.225  io.k8s.batch.v1.jobs.create   status=10   ALREADY EXISTS

AppWrapper conditions as the user sees them — note all four share the same second:

QuotaReserved      True    Suspend is false
ResourcesDeployed  True    Suspend is false
PodsReady          False   Suspend is false
Unhealthy          True    MissingComponent: Only found 0 deployed components, but was expecting 1

For contrast, a genuine capacity failure in the same cluster looks quite different —
InsufficientPodsReady, ~30 minutes after admission, with a pod that exists and cannot
be scheduled. The failure reported here is immediate and reports no component at all.

Affected Releases

• v1.2.0 — confirmed, reproduced.
• v1.2.2 — not tested directly, but the code path appears unchanged relative to
  v1.2.0, so we expect it to be affected. Would welcome correction.

Additional Context

• OS: 2) Linux — nodes run Container-Optimized OS
• OS Version: COS (managed node image)
• Browser (UI issues): N/A
• Browser Version (UI issues): N/A
• Cloud: 3) Other — Google Cloud
• Kubernetes: 2) Other K8s — GKE (managed)
• OpenShift or K8s version: 1.34.9
• Other relevant info:
  • Every AppWrapper in our workload wraps exactly one batch/v1 Job (podSets count 1).
  • Admission is driven by Kueue; the AppWrapper is unsuspended by
    kueue-controller-manager immediately before the sequence above.
  • The controller runs as shipped: a single replica with no leader election, so a restart
    is a full interruption of reconciliation. Our reproduction uses a restart deliberately,
    but in normal operation we see the same failure without intervening — the controller
    had restarted 9 times over three days, all during a period when the API server was
    intermittently unreachable.

Questions

We may be misreading the cause, so rather than propose a change we would rather ask:

1. Is the AppWrapperRunning component check expected to be resilient to informer cache
   lag, or is the cache assumed to be current by the time the workload reaches Running?
2. Is the duplicate create returning ALREADY EXISTS expected behaviour, or does it
   indicate the same stale read?
3. Is there a supported configuration that makes MissingComponent retry rather than fail
   terminally? We could not find one.
4. Is the shipped single-replica deployment without leader election expected to be
   susceptible to this after a restart, or are we likely hitting something specific to our
   environment?

Happy to gather more detail, run a patched build, or test a proposed change if that would
help.