Skip to content

Replace traced readiness query polls with untraced waits - #1848

Merged
DABH merged 7 commits into
mainfrom
flake/untraced-readiness-probes
Sep 17, 2026
Merged

DABH merged 7 commits into
mainfrom
flake/untraced-readiness-probes

Conversation

@DABH

@DABH DABH commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

TLDR: Tracing tests polled a traced readiness query against a running workflow; the server attaches such a query to every attempt of a pending workflow task, so it could run twice or time out client-side, and every poll ran through the instrumented client. The langgraph tests now query the completed workflow (no task can be in flight), and the OpenAI tracing tests use an untraced history probe for the one case that genuinely needs to stop a worker mid-run.

What was changed

  • langgraph test_summary_fn.py: the ran/invoked query polls, the _done park and its finish signal are gone. The workflow returns the graph result directly and the tests query __temporal_workflow_metadata after handle.result(): the workflow-side node sets current details last-writer-wins and nothing clears them at graph end, so the completed workflow answers the same as a mid-run probe would.
  • openai_agents test_openai_tracing.py: the four ready query polls become an untraced history poll (plain client) for ACTIVITY_TASK_COMPLETED then WORKFLOW_TASK_COMPLETED. These tests must stop the first worker mid-run for the cross-worker span-parenting coverage, so run-to-completion is not an option there; history works on both the dev server and the time-skipping server.
  • assert_eventually: a cancelled RPC now retries on the normal interval and gives up at the deadline instead of spinning until pytest's timeout.
  • The now-unused queries and flags are removed. An earlier revision added a wait_for_workflow_idle helper; it was dropped per review since these were its only callers and it was a silent no-op on the time-skipping server (which never reports pending_workflow_task).

Testing

  • Earlier revision: --flake-finder --flake-runs=20 per converted test under CPU load, 120/120 on the dev server and 80/80 for the OpenAI tests on time-skipping (the originals also passed, so this is hardening, not a reproduced failure).
  • Final revision: test_summary_fn.py 3× (14/14 each), test_openai_tracing.py 8/8, full lint gate clean. The converted langgraph tests skip on the time-skipping server as before (metadata query unreliable there).

Several contrib tests polled a workflow query (ready, ran, invoked) until
the workflow reached its signal wait. A query issued while a workflow
task is pending is buffered by the server and attached to every attempt
of that task, so the poll can run twice or time out client-side before
an attempt completes, and in the tracing tests each poll also runs
through the instrumented client and worker interceptors.

Add tests.helpers.wait_for_workflow_idle, which waits via describe()
until the workflow is RUNNING with no pending workflow task, activity,
child or Nexus operation, and use it in the langgraph summary tests,
whose workflow parks on a signal wait right after its first task.

The OpenAI Agents tracing tests also run on the time-skipping test
server in CI, which never reports pending_workflow_task and returns no
events for wait_new_event history polls on a running workflow. Those
tests instead poll history through the plain client for a workflow task
completion after the activity completion (assert_event_subsequence),
which is exact on both servers.

Remove the now-unused ready/ran/invoked queries and their backing flags.
@DABH DABH added the skip-changelog PR changes do not require changelog updates label Sep 10, 2026
@DABH
DABH marked this pull request as ready for review September 11, 2026 05:39
@DABH
DABH requested review from a team as code owners September 11, 2026 05:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The test synchronization changes are consistent, targeted, and account for time-skipping server limitations.

Pull request overview

Replaces traced readiness queries with untraced synchronization to reduce tracing-test flakiness.

Changes:

  • Adds a shared workflow-idle polling helper.
  • Uses idle polling in LangGraph tests.
  • Uses workflow-history polling in OpenAI tracing tests.
File summaries
File Description
tests/helpers/__init__.py Adds wait_for_workflow_idle.
tests/contrib/openai_agents/test_openai_tracing.py Replaces readiness queries with history polling.
tests/contrib/langgraph/test_summary_fn.py Replaces query polling with idle waits.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 0
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Build wait_for_workflow_idle on assert_eventually so it shares the
timeout, interval and cancelled-RPC retry of the other helpers, while
still failing fast when the workflow is no longer running. Name the
pending work in the timeout message and print the status name instead
of its integer value.

Document that timer-only waits count as idle, that the time-skipping
test server never reports a pending workflow task, and that the OpenAI
history probe assumes the first completed activity precedes the park.
A cancelled RPC used to retry immediately with no sleep and no deadline
check, so a persistently cancelled probe spun until pytest's timeout and
hid the helper's own diagnostic. Retry on the normal interval and give
up at the deadline instead. Also name the elapsed timeout in the
wait_for_workflow_idle message.
@DABH
DABH added this pull request to stack #1872 September 16, 2026 05:06
Comment thread tests/helpers/__init__.py Outdated
Per review: the two langgraph tests were the helper's only callers and
do not need a mid-run probe. The workflow-side node sets current details
last-writer-wins and nothing clears them at graph end, so querying
__temporal_workflow_metadata on the completed workflow returns the same
value while guaranteeing no workflow task is in flight. Removes the
_done park and finish signal along with the helper, which was also a
silent no-op on the time-skipping server. The OpenAI tracing tests keep
the untraced history probe: they must stop the first worker mid-run for
the cross-worker span-parenting coverage.
@DABH
DABH merged commit 714c07d into main Sep 17, 2026
46 of 49 checks passed
@DABH
DABH deleted the flake/untraced-readiness-probes branch September 17, 2026 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog PR changes do not require changelog updates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants