test: Get the orphaned Playwright specs running again, as pytest

[jat255]

Closes #243.

playwright/ held four specs that nothing ran. make test called npm run playwright test, a script that does not exist, and two of the four shelled out to shinylive export from the Python shinylive package — which depends on this repository, the circular dependency that kept the playwright job in build.yml commented out for years. All four are now pytest tests in tests/, next to the example suite #238 established, and they run in CI where a failure blocks the merge.

Stacked conceptually on #244 (the jest harness, #242). The two overlap only in package.json and the README's suite listing, so expect a small conflict in whichever lands second.

Groundwork (#240 needs this too)

fail_on_example_errors is now fail_on_page_errors and takes an opt-out: a test that provokes an error on purpose marks itself @pytest.mark.allow_page_errors and asserts on the failure instead of being failed by the gate.

Every test also carries an examples or a site marker, and the example Make targets select on examples, so a non-example test is no longer swept into the engine × shard matrix in test-apps.yml. make site-test runs the other side in one job. The marker is enforced at collection rather than documented — since make examples-smoke-test selects -m "examples and py" and make site-test selects -m site, a file with neither marker is deselected by both and would silently run in no CI job at all. Collection now errors on one.

Two things the review turned up in this area, both fixed here:

• The site tests are the first suite ever to run against a build with GOOGLE_TAG_MANAGER_ID set. build.yml is the only workflow that sets it, so make all there templates a Google Tag Manager loader into the site pages; test-apps.yml never has, which is why the examples suite has never met it. A tag manager that fails to load is a console error, and the gate would have failed on it — a hard dependency on a third-party host inside the workflow that gates the deploy. An autouse fixture now aborts analytics requests and the gate forgives the abort. Measured overhead on a Pyodide-booting test: none.
• --shard was splitting the whole of tests/, not the selected engine. A conftest's pytest_collection_modifyitems runs before _pytest.mark's, so the sharding was dealing out all 91 tests and only then dropping the other engine's. Today's shard sizes come out identical either way, but this PR is what put a second suite into that collection, so the balance was luck. Sharding moved into a trylast plugin. Engine selection and shard sizes are byte-identical to fe8c0b6: 54 py, 22 r, shards 18/18/18 and 8/7/7, and the shard union equals the full selection.

The export fixture

tests/export_app.py assembles a static export out of what make all already produces under build/: app.json (the files, verbatim), export_template/index.html with its variables filled in, edit/index.html as-is, and the shinylive bundle, symlinked rather than copied. That is everything runExportedApp() reads, so nothing in this repository needs pip install shinylive any more, and the stale rationale comment in build.yml goes away with it. The exported_app fixture writes one and hands back the URL it is served at.

The migrated tests

• test_site_editor.py and test_site_url_loading.py, from examples-viewer.spec.ts and load-from-url.spec.ts, against the _shinylive/ build the session static_server fixture already serves.
• test_site_export.py, from shiny-static.spec.ts and editor-cell.spec.ts, against the export fixture. The app mode comes from ?_shinylive-mode=, which replaces the sed -e 's/viewer/editor-cell/' in playwright.config.ts that had not matched anything in export_template/index.html in a long time — the drift the issue describes.

Three things the migration turned up in the originals:

• expect_terminal_has_text() (playwright/helpers.ts:73) returned a boolean that every call site discarded, so both terminal assertions in examples-viewer.spec.ts passed vacuously. The migrated helper actually asserts.
• "App view can show header bar" navigated to /editor/, so it had never tested the app view. It now goes to /app/.
• Mod-Enter in the editor is Cmd on a Mac, which the old spec papered over by pressing both modifiers. MOD_KEY picks by platform; both branches were checked against the real editor, including the Control one CI takes.

Removals

playwright.config.ts, the four specs and their helpers and fixture apps, the cypress:open script (cypress is not a dependency), the @playwright/test devDependency, the broken test target, and the commented-out playwright job in build.yml. examples-test-deps becomes test-deps, since both suites use it.

CI

The site tests run in build.yml, immediately after make all and before the deploy — that is what they test, it reuses a build that already takes most of the job's time, and it is where the disabled playwright job used to sit.

Worth stating plainly: this means a flaky site test blocks the S3 deploy and the release-asset upload, not just the merge. That is what #243 asks for, and it is the same failure shape that got the old job commented out, minus the circular dependency that made it unfixable. It also adds several minutes to a job that currently runs in about 2–4 minutes: test-deps builds Shiny from the submodule and downloads Chromium, and the tests themselves take ~55s. If that trade is wrong, the alternative is a seventh leg on test-apps.yml, which costs a second full make all.

The examples matrix in test-apps.yml is unchanged apart from the marker selection.

Verification

All 15 site tests pass against a local build, over three runs (~55s each). The error gate was checked in all four directions: it catches console errors, catches page errors, allow_page_errors genuinely skips both, and a clean page passes. The GTM failure was reproduced against a real GTM-templated page and confirmed fixed. make -n site-test and make -n examples-smoke-test both expand correctly and the test-deps rename doesn't break test-apps.yml. npm ls --package-lock-only --all exits 0 and a throwaway npm ci installs with zero playwright entries.

The example suite still collects 91 and passes on a sample of both engines. One example test, test_app_with_plot, fails against a local _shinylive/ that predates the input_slider min change made in fe8c0b6; it fails identically on main, and CI builds fresh.

Known gaps

Gate violations surface as pytest teardown ERRORs rather than FAILs — pre-existing from #238; the exit code is still non-zero, so CI blocks. The site suite covers only the Python site, same as the originals it replaces.

[jat255]

Superseded by #247 — same commits, from a branch on posit-dev rather than a fork.