[test] Expose first-run e2e test failures in CI - #99528
Draft
unstubbable wants to merge 3 commits into
Draft
unstubbable wants to merge 3 commits into
unstubbable wants to merge 3 commits into
Conversation
Contributor
Tests PassedCommit: caecc2b |
lubieowoce
approved these changes
Oct 1, 2026
PR #89929 added automatic retries of individual e2e tests to reduce CI time. These retries reuse the fixture without rerunning `beforeAll`. A failed attempt can write an ISR cache or change other state. The retry can then pass against a different starting state, potentially hiding a real issue. Each CI test job runs a shard containing multiple test files. Previously, rerunning a failed job also reran the files that had already passed. PR #92832 caches passing results so a job rerun can skip those files and rerun only the failed files. This reduces both retry cost and time, allowing us to remove the individual-test retries. This change preserves the timeout improvements from #89929. It also removes the redundant retry opt-out and updates stale comments and test-gate documentation.
The concurrent root-param cache tests require requests to share a server process. Deployed requests can reach different function instances, so these tests cannot reliably check cross-request cache behavior there. This change excludes both cases from deploy mode and keeps them in dev and `next start`. The per-test retries introduced in #89929 could hide the cold-cache equality assumption after the first attempt populated the cache.
The first prefetch for unprerendered root params can receive generic fallback segments on deploy while ISR generates the specific shell. `next start` generates the shell blockingly on demand. This change refreshes the browser and repeats only the preparatory prefetch until it receives the specific shell. The cache-reuse and navigation assertions still run once. The static-shell test loads a fresh home page so BFCache cannot restore visible accordions. It then primes both language variants in the client cache. The per-test retries introduced in #89929 could hide this serving difference after an earlier attempt triggered ISR. The TODOs leave the direction for consolidating the serving behavior open.
unstubbable
force-pushed
the
hl/revert-automatic-test-retries
branch
from
October 1, 2026 20:58
3653f96 to
caecc2b
Compare
lubieowoce
reviewed
Oct 1, 2026
Comment on lines
+1166
to
+1169
| // The test loads a fresh home page to prevent BFCache from restoring | ||
| // visible accordions and triggering uncontrolled prefetches. It primes | ||
| // English again before French to keep both root-param entries in the | ||
| // client cache. |
lubieowoce
approved these changes
Oct 1, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR #89929 added automatic retries of individual e2e tests to reduce CI time. These retries reuse the fixture without rerunning
beforeAll. A failed attempt can write an ISR cache or change other state. The retry can then pass against a different starting state, potentially hiding a real issue.Each CI test job runs a shard containing multiple test files. Previously, rerunning a failed job also reran the files that had already passed. PR #92832 caches passing results so a job rerun can skip those files and rerun only the failed files. This reduces both retry cost and time, allowing us to remove the individual-test retries.
This change preserves the timeout improvements from #89929. It also removes the redundant retry opt-out and updates stale comments and test-gate documentation.
The retry removal also exposed test assumptions that required these changes:
next start.next startgenerates that shell blockingly. The cache-reuse and navigation assertions remain outside the retry. The static-shell test also loads a fresh home page to prevent BFCache from restoring visible accordions, then primes both language variants. The TODOs leave the direction for consolidating the serving behavior open.The deploy run and rerun exposed the failures addressed by these test corrections.