Integrate Test262 into WPT #229
Conversation
|
|
||
| ## Alternatives Considered | ||
|
|
||
| ### Using `--test-types=test262` |
There was a problem hiding this comment.
We should move to this instead.
There was a problem hiding this comment.
If we do use a separate test type internally (not testharness.js) then I agree, this would be the natural way.
There was a problem hiding this comment.
I don't think the internal implementation details are the important thing. From the point of view of a consumer test262 tests are just another kind of test we can run, just like crashtests are basically a special kind of reftest, but we expose them as a separate test type because that makes more sense to users.
|
Should protect the WPT source from landing manual changes to that repo since they won't be synced back upstream. |
|
Can we open web-platform-tests/wpt@master...o-:wpt:test262-upstream as a draft PR? |
|
|
||
| ## Summary | ||
|
|
||
| This RFC proposes integrating the Test262 (ECMAScript) test suite into WPT. Due to its large size (>50,000 files), tests will be fetched on-demand rather than vendored into the repository. Execution will be opt-in via a new `--test262` flag. This provides a unified way to run this conformance suite while protecting the WPT repository from bloat and performance degradation. |
There was a problem hiding this comment.
I strongly prefer just vendoring the test262 files directly.
Although that number is large it's about a 50% increase on the current number of files in the repo, so it's not unmanagable.
Doing the vendoring directly means that there's no novel requirements for wpt consumers who want to run these tests (e.g. in vendor repos), apart from selecting the new test type.
I'd also propose a path like third_party/test262 so that it's at least clear that this is not code that lives in wpt. I do think we're likely to run into cases where people are surprised when they make changes to this directory and they aren't upstreamed; at least the third_party convention provides a clue that this might not work.
|
|
||
| ## Motivation | ||
|
|
||
| * **Browser-based Testing:** Run Test262 in real browsers (stable and experimental), not just JS shells. This provides a more complete conformance signal for how ECMAScript behaves in the full web platform environment. |
There was a problem hiding this comment.
I'd suggest "Allowing ECMAScript features to be proposed and selected in Interop" as a reason for wanting this.
|
|
||
| ### 1. Source Management | ||
|
|
||
| The WPT repository will contain a reference to the official `tc39/test262` repository (e.g., via a `.gitmodules` entry). This reference specifies the expected local path for the Test262 source and a pinned commit SHA. |
There was a problem hiding this comment.
I think we should try and be specific and not use "e.g." here.
I don't think we should use git submodules, or .gitmodules, but just use our own format e.g. a file like third_party/vendor.toml with a format like:
[test262]
source = <git url>
commit = <sha1>| WPT's manifest generation and serving logic will be extended to support the on-demand nature of Test262 tests. | ||
|
|
||
| * **Manifest Generation:** The `wpt manifest` command will recognize the Test262 source directory and traverse its `test` directory to discover tests. | ||
| * **Metadata Parsing:** A new, Test262-specific parser will read the YAML frontmatter from each `.js` file to extract metadata (e.g., ES features, negative test expectations, execution flags). |
There was a problem hiding this comment.
To be clear: where will this end up? In MANIFEST.json? Or is the proposal to have a new special manifest format just for test262? Ideally I'd like to avoid that, but the RFC isn't providing enough information to understand the tradeoffs.
|
|
||
| ### 3. Acquiring Test262 Source Code | ||
|
|
||
| The Test262 source code is acquired on-demand. `wpt` can handle this automatically for users in a Git environment, while providing a manual path for CI systems. |
There was a problem hiding this comment.
As already mentioned I don't think this is the correct approach.
I think we should have a CI job that fetches the latest commit of test262, copies that over to the destination path and then opens a PR with the update. The CI should run some kind of smoketests to ensure that the update hasn't broken test262 support. Possibly a bot should be trusted to auto-merge these PRs (although that means we have to trust the test262 repository).
Rendered