From 0b4d345373583418437faed48b66a0db0dde2afa Mon Sep 17 00:00:00 2001 From: James Scott Date: Tue, 4 Nov 2025 01:18:46 +0000 Subject: [PATCH 1/6] Integrate Test262 into WPT --- rfcs/test262_integration.md | 69 +++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 rfcs/test262_integration.md diff --git a/rfcs/test262_integration.md b/rfcs/test262_integration.md new file mode 100644 index 0000000..998c117 --- /dev/null +++ b/rfcs/test262_integration.md @@ -0,0 +1,69 @@ +# RFC XXX: Integrating Test262 + +## Summary + +This RFC proposes the integration of the official ECMAScript test suite (Test262) into the web-platform-tests (WPT) project. Given the suite's substantial size (over 50,000 files), this proposal avoids vendoring the files directly into the WPT repository. Instead, Test262 will be integrated via a mechanism that ensures tests are fetched on-demand, as detailed in Section 3, "Acquiring Test262 Source Code". + +Execution of these tests will be strictly opt-in, controlled by a new `wpt` command-line flag. This approach provides access to a critical conformance suite, enables unified test execution and analysis, and protects the WPT repository from excessive bloat and performance degradation. + +## Motivation + +* **Comprehensive Conformance:** Test262 is the authoritative test suite for ECMAScript (JavaScript). Its inclusion provides a complete and standards-aligned signal for JS engine conformance, which is a core part of the web platform. +* **Unified Infrastructure:** Integrating Test262 allows browser vendors to run this essential suite using the same mature infrastructure (`wptrunner`, CI) and analyze results on the same platform (`wpt.fyi`) as all other web platform features. +* **Operational Efficiency:** This removes the need for vendors to maintain separate infrastructure for running Test262 and attempting to correlate its results with WPT. It streamlines the engineering workflow for testing and triaging JS engine behavior. + +## Detailed Design + +### 1. Source Management + +To ensure reproducibility without vendoring the entire suite, the WPT repository will contain a **reference** to the official `tc39/test262` repository. This reference will specify the expected local path for the Test262 source code and a pinned commit SHA. + +* The pinned commit SHA guarantees that any given WPT commit is tied to a precise version of Test262, ensuring repeatable test runs. +* An automated process, such as a weekly GitHub Action, will create pull requests to update this pinned SHA. This provides a regular, managed cadence for ingesting upstream Test262 updates. + +For details on how to acquire the Test262 source code based on this reference, please refer to Section 3, "Acquiring Test262 Source Code." + +### 2. Test Integration + +The existing WPT manifest generation and serving logic will be extended to support the on-demand nature of the Test262 tests. + +* **Manifest Generation:** The `wpt manifest` command will recognize the Test262 source directory. When the source is present, the command will traverse its `test` directory to discover tests. +* **Metadata Parsing:** A new, Test262-specific parser will be used to read the YAML frontmatter from each `.js` file. This parser will extract critical metadata, such as ES features, negative test expectations, and execution flags (e.g., `strict`, `module`, `raw`). +* **Harness and Server:** The specialized Test262 harness files and `wptserve` handlers will use the parsed metadata to construct the correct execution environment for each test, generating distinct URLs for different test modes. + +### 3. Acquiring Test262 Source Code + +It is the responsibility of the user or CI system to ensure the Test262 source code is present at the location specified in the `.gitmodules` file. `wptrunner` will not perform the sync itself. This provides flexibility for different development and CI environments. + +There are two primary methods for acquiring the source: + +* **Method 1: Git Submodule (Recommended)** + For users in a standard Git environment, the tests can be synced by running `git submodule update --init`. This will fetch the precise Test262 revision pinned in the main WPT repository. + +* **Method 2: Manual/Custom Sync** + For CI systems or vendors using non-Git source control, the Test262 repository can be acquired through other means, such as a direct `git clone` or a custom mirroring process. The system must ensure that the repository is checked out to the specific commit SHA defined in the `.gitmodules` file and placed at the correct local path. + +### 4. Execution Control + +To prevent performance degradation for users not focused on JS engine testing, running Test262 will be **opt-in**. + +* **Default Behavior:** By default, `wpt run` will **not** look for the Test262 directory and will **not** include its tests in the run. +* **Opt-in Flag:** A new command-line flag, `--test262`, will be added to `wpt run`. +* **Flag Behavior:** When the `--test262` flag is present, `wpt` will verify that the Test262 source directory exists. If the directory is missing, `wpt` will raise an error with instructions on how to sync the code (as described in the "Acquiring Test262 Source Code" section). If present, it will include the tests in the run. +* **CI Integration:** CI systems can explicitly enable Test262 runs by adding this flag to their configuration, after ensuring the source has been acquired. + +## Implementation Considerations & Risks + +* **Vendor Integration:** The responsibility for acquiring the Test262 source code is deliberately left to the user. This allows vendors with non-Git source control systems to implement their own mechanisms for syncing the required Test262 revision. +* **`WEB_FEATURES.yml` Mapping:** The current mechanism for mapping tests to features relies on `WEB_FEATURES.yml` files being co-located with the tests they describe. Since the Test262 source will not be in the main WPT repository, a new mechanism will need to be devised to associate these tests with web features for analysis in tools like `wpt.fyi`. +* **CI Performance:** A full Test262 run is lengthy. The impact on CI resources and runtime must be monitored. It is anticipated that full runs will be scheduled on a nightly or weekly basis rather than on every pull request. +* **Initial Source Checkout:** The first time a user runs with `--test262`, they must ensure the Test262 source is checked out. The one-time delay of fetching the repository should be clearly documented. +* **Source Code Visibility on wpt.fyi:** A key trade-off is that `wpt.fyi` cannot easily link to the source code of Test262 tests. The necessity of this feature and potential solutions (e.g., linking to a harness page with a note) will need to be discussed. + +## Proof of Concept + +A prototype implementation of the required harness, server, and manifest changes exists in the following commit. + +**Link:** [https://github.com/o-/wpt/commit/baa8415800161122e7d9d1170513f5aad82a4504](https://github.com/o-/wpt/commit/baa8415800161122e7d9d1170513f5aad82a4504) + +*Caveat: As part of the final implementation, the YAML parsing logic (`parseTestRecord.py`) will be moved to a more Test262-specific location (e.g., `tools/test262/`) to clarify that it is not a generic WPT manifest parser.* From b2bd35757c8bb1dc7c5bd3239c98a4a666bdd527 Mon Sep 17 00:00:00 2001 From: James Scott Date: Tue, 4 Nov 2025 15:00:57 +0000 Subject: [PATCH 2/6] updates --- rfcs/test262_integration.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/rfcs/test262_integration.md b/rfcs/test262_integration.md index 998c117..514bd02 100644 --- a/rfcs/test262_integration.md +++ b/rfcs/test262_integration.md @@ -8,7 +8,7 @@ Execution of these tests will be strictly opt-in, controlled by a new `wpt` comm ## Motivation -* **Comprehensive Conformance:** Test262 is the authoritative test suite for ECMAScript (JavaScript). Its inclusion provides a complete and standards-aligned signal for JS engine conformance, which is a core part of the web platform. +* **Testing in Browsers:** Provides a standardized way to run Test262 within actual web browsers (both stable and experimental channels), capturing the behavior of the integrated JS engine rather than standalone shells. This offers a complementary and more holistic conformance signal, specifically for how ECMAScript behaves within the full web platform environment, which differs from results obtained in isolated JS shells (e.g., as typically presented on `test262.fyi`). * **Unified Infrastructure:** Integrating Test262 allows browser vendors to run this essential suite using the same mature infrastructure (`wptrunner`, CI) and analyze results on the same platform (`wpt.fyi`) as all other web platform features. * **Operational Efficiency:** This removes the need for vendors to maintain separate infrastructure for running Test262 and attempting to correlate its results with WPT. It streamlines the engineering workflow for testing and triaging JS engine behavior. @@ -49,16 +49,24 @@ To prevent performance degradation for users not focused on JS engine testing, r * **Default Behavior:** By default, `wpt run` will **not** look for the Test262 directory and will **not** include its tests in the run. * **Opt-in Flag:** A new command-line flag, `--test262`, will be added to `wpt run`. -* **Flag Behavior:** When the `--test262` flag is present, `wpt` will verify that the Test262 source directory exists. If the directory is missing, `wpt` will raise an error with instructions on how to sync the code (as described in the "Acquiring Test262 Source Code" section). If present, it will include the tests in the run. +* **Flag Behavior:** When the `--test262` flag is present, `wpt` will verify that the Test262 source directory exists. If the directory is missing, `wpt` will raise an error with instructions on how to sync the code (as described in the "Acquiring Test262 Source Code" section). If present, it will include the Test262 tests **in addition to** the standard WPT tests discovered. * **CI Integration:** CI systems can explicitly enable Test262 runs by adding this flag to their configuration, after ensuring the source has been acquired. +## Alternatives Considered + +### Using `--test-types=test262` + +One alternative would be to treat Test262 as a new "test type" within `wptrunner`, discoverable via a flag like `--test-types=test262`. + +* **Reason for Rejection:** This approach was rejected because Test262 tests are fundamentally different from other WPT test types (e.g., `testharness`, `reftest`). They do not live in the WPT repository, have their own unique metadata format (YAML frontmatter), and require a completely different harness and server setup. A new, dedicated `--test262` flag provides a clearer separation of concerns. It explicitly signals that the user is opting into a distinct, large, and externally managed test suite, rather than just another flavor of a standard WPT test. This explicitness also leads to a cleaner implementation, as the logic for acquiring and locating the external test suite can be cleanly tied to a single, unambiguous flag. + ## Implementation Considerations & Risks * **Vendor Integration:** The responsibility for acquiring the Test262 source code is deliberately left to the user. This allows vendors with non-Git source control systems to implement their own mechanisms for syncing the required Test262 revision. -* **`WEB_FEATURES.yml` Mapping:** The current mechanism for mapping tests to features relies on `WEB_FEATURES.yml` files being co-located with the tests they describe. Since the Test262 source will not be in the main WPT repository, a new mechanism will need to be devised to associate these tests with web features for analysis in tools like `wpt.fyi`. +* **CI Submodule Behavior:** While WPT has historically moved away from submodules for vendoring dependencies, using one here as an *optional, on-demand pointer* presents a different trade-off. However, a risk remains that some CI systems are configured to check out git submodules by default. This could cause an unintentional, resource-intensive checkout of the large Test262 repository. The impact and configuration options for common CI systems should be investigated. * **CI Performance:** A full Test262 run is lengthy. The impact on CI resources and runtime must be monitored. It is anticipated that full runs will be scheduled on a nightly or weekly basis rather than on every pull request. * **Initial Source Checkout:** The first time a user runs with `--test262`, they must ensure the Test262 source is checked out. The one-time delay of fetching the repository should be clearly documented. -* **Source Code Visibility on wpt.fyi:** A key trade-off is that `wpt.fyi` cannot easily link to the source code of Test262 tests. The necessity of this feature and potential solutions (e.g., linking to a harness page with a note) will need to be discussed. +* **Source Code Visibility on wpt.fyi:** A key trade-off is that `wpt.fyi` cannot easily link to the source code of Test262 tests without special handling. A solution is likely feasible but would require `wpt.fyi` to have access to a manifest that maps test names to their original file paths within the Test262 repository and to be aware of the pinned submodule SHA for a given run. This is a similar challenge to other generated tests and can be considered a future enhancement. ## Proof of Concept From 1d618f8a5f5b59eaabf71ff7050229aafe5d2fa1 Mon Sep 17 00:00:00 2001 From: James Scott Date: Tue, 4 Nov 2025 15:29:43 +0000 Subject: [PATCH 3/6] minor edits --- rfcs/test262_integration.md | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/rfcs/test262_integration.md b/rfcs/test262_integration.md index 514bd02..c69f553 100644 --- a/rfcs/test262_integration.md +++ b/rfcs/test262_integration.md @@ -1,4 +1,4 @@ -# RFC XXX: Integrating Test262 +# RFC 229: Integrating Test262 ## Summary @@ -9,7 +9,7 @@ Execution of these tests will be strictly opt-in, controlled by a new `wpt` comm ## Motivation * **Testing in Browsers:** Provides a standardized way to run Test262 within actual web browsers (both stable and experimental channels), capturing the behavior of the integrated JS engine rather than standalone shells. This offers a complementary and more holistic conformance signal, specifically for how ECMAScript behaves within the full web platform environment, which differs from results obtained in isolated JS shells (e.g., as typically presented on `test262.fyi`). -* **Unified Infrastructure:** Integrating Test262 allows browser vendors to run this essential suite using the same mature infrastructure (`wptrunner`, CI) and analyze results on the same platform (`wpt.fyi`) as all other web platform features. +* **Unified Infrastructure:** Integrating Test262 allows browser vendors to run this suite using the same infrastructure and analyze results on the same platform (`wpt.fyi`) as all other web platform features. * **Operational Efficiency:** This removes the need for vendors to maintain separate infrastructure for running Test262 and attempting to correlate its results with WPT. It streamlines the engineering workflow for testing and triaging JS engine behavior. ## Detailed Design @@ -33,15 +33,13 @@ The existing WPT manifest generation and serving logic will be extended to suppo ### 3. Acquiring Test262 Source Code -It is the responsibility of the user or CI system to ensure the Test262 source code is present at the location specified in the `.gitmodules` file. `wptrunner` will not perform the sync itself. This provides flexibility for different development and CI environments. +The Test262 source code is acquired on-demand. The `wpt` command will handle this automatically for users in a standard Git environment, while still providing a manual path for CI systems. -There are two primary methods for acquiring the source: +When the `--test262` flag is used, `wpt` will first check for the existence of the Test262 source directory. If the directory is missing, `wpt` will determine if the main WPT repository is a Git checkout. +- **If it is a Git repository,** `wpt` will attempt to automatically sync the source by running `git submodule update --init --recursive`. +- **If it is not a Git repository,** `wpt` will fail with an error, instructing the user to acquire the source manually. -* **Method 1: Git Submodule (Recommended)** - For users in a standard Git environment, the tests can be synced by running `git submodule update --init`. This will fetch the precise Test262 revision pinned in the main WPT repository. - -* **Method 2: Manual/Custom Sync** - For CI systems or vendors using non-Git source control, the Test262 repository can be acquired through other means, such as a direct `git clone` or a custom mirroring process. The system must ensure that the repository is checked out to the specific commit SHA defined in the `.gitmodules` file and placed at the correct local path. +This allows CI systems and vendors with non-Git checkouts to populate the directory using their own custom tooling (e.g., `git clone` or a mirrored copy), while providing a seamless, automated experience for the common user. ### 4. Execution Control @@ -49,14 +47,14 @@ To prevent performance degradation for users not focused on JS engine testing, r * **Default Behavior:** By default, `wpt run` will **not** look for the Test262 directory and will **not** include its tests in the run. * **Opt-in Flag:** A new command-line flag, `--test262`, will be added to `wpt run`. -* **Flag Behavior:** When the `--test262` flag is present, `wpt` will verify that the Test262 source directory exists. If the directory is missing, `wpt` will raise an error with instructions on how to sync the code (as described in the "Acquiring Test262 Source Code" section). If present, it will include the Test262 tests **in addition to** the standard WPT tests discovered. +* **Flag Behavior:** When the `--test262` flag is present, `wpt` will ensure the Test262 source code is present, fetching it automatically if possible (as described in "Acquiring Test262 Source Code"). It will then include the Test262 tests **in addition to** the standard WPT tests discovered. * **CI Integration:** CI systems can explicitly enable Test262 runs by adding this flag to their configuration, after ensuring the source has been acquired. ## Alternatives Considered ### Using `--test-types=test262` -One alternative would be to treat Test262 as a new "test type" within `wptrunner`, discoverable via a flag like `--test-types=test262`. +One alternative would be to treat Test262 as a new "test type" within `wpt`, discoverable via a flag like `--test-types=test262`. * **Reason for Rejection:** This approach was rejected because Test262 tests are fundamentally different from other WPT test types (e.g., `testharness`, `reftest`). They do not live in the WPT repository, have their own unique metadata format (YAML frontmatter), and require a completely different harness and server setup. A new, dedicated `--test262` flag provides a clearer separation of concerns. It explicitly signals that the user is opting into a distinct, large, and externally managed test suite, rather than just another flavor of a standard WPT test. This explicitness also leads to a cleaner implementation, as the logic for acquiring and locating the external test suite can be cleanly tied to a single, unambiguous flag. From 4ac9407d70b7ff033339a9f1e0d141b1cc074932 Mon Sep 17 00:00:00 2001 From: James Scott Date: Tue, 4 Nov 2025 15:30:09 +0000 Subject: [PATCH 4/6] remove fluff --- rfcs/test262_integration.md | 65 +++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 35 deletions(-) diff --git a/rfcs/test262_integration.md b/rfcs/test262_integration.md index c69f553..282b5f0 100644 --- a/rfcs/test262_integration.md +++ b/rfcs/test262_integration.md @@ -2,74 +2,69 @@ ## Summary -This RFC proposes the integration of the official ECMAScript test suite (Test262) into the web-platform-tests (WPT) project. Given the suite's substantial size (over 50,000 files), this proposal avoids vendoring the files directly into the WPT repository. Instead, Test262 will be integrated via a mechanism that ensures tests are fetched on-demand, as detailed in Section 3, "Acquiring Test262 Source Code". - -Execution of these tests will be strictly opt-in, controlled by a new `wpt` command-line flag. This approach provides access to a critical conformance suite, enables unified test execution and analysis, and protects the WPT repository from excessive bloat and performance degradation. +This RFC proposes integrating the Test262 (ECMAScript) test suite into WPT. Due to its large size (>50,000 files), tests will be fetched on-demand rather than vendored into the repository. Execution will be opt-in via a new `--test262` flag. This provides a unified way to run this conformance suite while protecting the WPT repository from bloat and performance degradation. ## Motivation -* **Testing in Browsers:** Provides a standardized way to run Test262 within actual web browsers (both stable and experimental channels), capturing the behavior of the integrated JS engine rather than standalone shells. This offers a complementary and more holistic conformance signal, specifically for how ECMAScript behaves within the full web platform environment, which differs from results obtained in isolated JS shells (e.g., as typically presented on `test262.fyi`). -* **Unified Infrastructure:** Integrating Test262 allows browser vendors to run this suite using the same infrastructure and analyze results on the same platform (`wpt.fyi`) as all other web platform features. -* **Operational Efficiency:** This removes the need for vendors to maintain separate infrastructure for running Test262 and attempting to correlate its results with WPT. It streamlines the engineering workflow for testing and triaging JS engine behavior. +* **Browser-based Testing:** Run Test262 in real browsers (stable and experimental), not just JS shells. This provides a more complete conformance signal for how ECMAScript behaves in the full web platform environment. +* **Unified Infrastructure:** Use the existing WPT infrastructure (`wpt`, CI) and `wpt.fyi` for test execution and results analysis. +* **Operational Efficiency:** Remove the need for vendors to maintain separate infrastructure for running Test262 and correlating its results with WPT. ## Detailed Design ### 1. Source Management -To ensure reproducibility without vendoring the entire suite, the WPT repository will contain a **reference** to the official `tc39/test262` repository. This reference will specify the expected local path for the Test262 source code and a pinned commit SHA. +The WPT repository will contain a reference to the official `tc39/test262` repository (e.g., via a `.gitmodules` entry). This reference specifies the expected local path for the Test262 source and a pinned commit SHA. -* The pinned commit SHA guarantees that any given WPT commit is tied to a precise version of Test262, ensuring repeatable test runs. -* An automated process, such as a weekly GitHub Action, will create pull requests to update this pinned SHA. This provides a regular, managed cadence for ingesting upstream Test262 updates. +* The pinned commit SHA ties each WPT commit to a precise version of Test262, ensuring reproducible test runs. +* An automated process (e.g., a weekly GitHub Action) will create pull requests to update this pinned SHA, providing a regular cadence for ingesting upstream updates. -For details on how to acquire the Test262 source code based on this reference, please refer to Section 3, "Acquiring Test262 Source Code." +See Section 3 for details on how the Test262 source is acquired. ### 2. Test Integration -The existing WPT manifest generation and serving logic will be extended to support the on-demand nature of the Test262 tests. +WPT's manifest generation and serving logic will be extended to support the on-demand nature of Test262 tests. -* **Manifest Generation:** The `wpt manifest` command will recognize the Test262 source directory. When the source is present, the command will traverse its `test` directory to discover tests. -* **Metadata Parsing:** A new, Test262-specific parser will be used to read the YAML frontmatter from each `.js` file. This parser will extract critical metadata, such as ES features, negative test expectations, and execution flags (e.g., `strict`, `module`, `raw`). -* **Harness and Server:** The specialized Test262 harness files and `wptserve` handlers will use the parsed metadata to construct the correct execution environment for each test, generating distinct URLs for different test modes. +* **Manifest Generation:** The `wpt manifest` command will recognize the Test262 source directory and traverse its `test` directory to discover tests. +* **Metadata Parsing:** A new, Test262-specific parser will read the YAML frontmatter from each `.js` file to extract metadata (e.g., ES features, negative test expectations, execution flags). +* **Harness and Server:** Specialized Test262 harness files and `wptserve` handlers will use the parsed metadata to construct the correct execution environment and generate distinct URLs for different test modes. ### 3. Acquiring Test262 Source Code -The Test262 source code is acquired on-demand. The `wpt` command will handle this automatically for users in a standard Git environment, while still providing a manual path for CI systems. +The Test262 source code is acquired on-demand. `wpt` can handle this automatically for users in a Git environment, while providing a manual path for CI systems. -When the `--test262` flag is used, `wpt` will first check for the existence of the Test262 source directory. If the directory is missing, `wpt` will determine if the main WPT repository is a Git checkout. -- **If it is a Git repository,** `wpt` will attempt to automatically sync the source by running `git submodule update --init --recursive`. +When the `--test262` flag is used, `wpt` first checks if the Test262 source directory exists. If missing, `wpt` determines if the main WPT repository is a Git checkout. +- **If it is a Git repository,** `wpt` will attempt to run `git submodule update --init --recursive`. - **If it is not a Git repository,** `wpt` will fail with an error, instructing the user to acquire the source manually. -This allows CI systems and vendors with non-Git checkouts to populate the directory using their own custom tooling (e.g., `git clone` or a mirrored copy), while providing a seamless, automated experience for the common user. +This allows CI systems and vendors with non-Git checkouts to use custom tooling, while providing a seamless experience for the common user. ### 4. Execution Control -To prevent performance degradation for users not focused on JS engine testing, running Test262 will be **opt-in**. +Running Test262 is **opt-in**. -* **Default Behavior:** By default, `wpt run` will **not** look for the Test262 directory and will **not** include its tests in the run. -* **Opt-in Flag:** A new command-line flag, `--test262`, will be added to `wpt run`. -* **Flag Behavior:** When the `--test262` flag is present, `wpt` will ensure the Test262 source code is present, fetching it automatically if possible (as described in "Acquiring Test262 Source Code"). It will then include the Test262 tests **in addition to** the standard WPT tests discovered. -* **CI Integration:** CI systems can explicitly enable Test262 runs by adding this flag to their configuration, after ensuring the source has been acquired. +* **Default Behavior:** By default, `wpt run` will not look for the Test262 directory and will not include its tests. +* **Opt-in Flag:** A new `--test262` flag will be added to `wpt run`. +* **Flag Behavior:** When present, `wpt` ensures the Test262 source code is available (fetching it if possible, per Section 3) and includes the Test262 tests **in addition to** the standard WPT tests. +* **CI Integration:** CI systems can enable Test262 runs by adding this flag, after ensuring the source has been acquired. ## Alternatives Considered ### Using `--test-types=test262` -One alternative would be to treat Test262 as a new "test type" within `wpt`, discoverable via a flag like `--test-types=test262`. - -* **Reason for Rejection:** This approach was rejected because Test262 tests are fundamentally different from other WPT test types (e.g., `testharness`, `reftest`). They do not live in the WPT repository, have their own unique metadata format (YAML frontmatter), and require a completely different harness and server setup. A new, dedicated `--test262` flag provides a clearer separation of concerns. It explicitly signals that the user is opting into a distinct, large, and externally managed test suite, rather than just another flavor of a standard WPT test. This explicitness also leads to a cleaner implementation, as the logic for acquiring and locating the external test suite can be cleanly tied to a single, unambiguous flag. +This was rejected because Test262 is fundamentally different from internal WPT test types (e.g., `testharness`, `reftest`). It is an external suite with its own metadata format and harness. A dedicated `--test262` flag provides a clearer separation of concerns and a cleaner implementation. ## Implementation Considerations & Risks -* **Vendor Integration:** The responsibility for acquiring the Test262 source code is deliberately left to the user. This allows vendors with non-Git source control systems to implement their own mechanisms for syncing the required Test262 revision. -* **CI Submodule Behavior:** While WPT has historically moved away from submodules for vendoring dependencies, using one here as an *optional, on-demand pointer* presents a different trade-off. However, a risk remains that some CI systems are configured to check out git submodules by default. This could cause an unintentional, resource-intensive checkout of the large Test262 repository. The impact and configuration options for common CI systems should be investigated. -* **CI Performance:** A full Test262 run is lengthy. The impact on CI resources and runtime must be monitored. It is anticipated that full runs will be scheduled on a nightly or weekly basis rather than on every pull request. -* **Initial Source Checkout:** The first time a user runs with `--test262`, they must ensure the Test262 source is checked out. The one-time delay of fetching the repository should be clearly documented. -* **Source Code Visibility on wpt.fyi:** A key trade-off is that `wpt.fyi` cannot easily link to the source code of Test262 tests without special handling. A solution is likely feasible but would require `wpt.fyi` to have access to a manifest that maps test names to their original file paths within the Test262 repository and to be aware of the pinned submodule SHA for a given run. This is a similar challenge to other generated tests and can be considered a future enhancement. +* **Vendor Integration:** The responsibility for acquiring the Test262 source is left to the user/CI system. This allows vendors with non-Git source control to implement their own sync mechanisms. +* **CI Submodule Behavior:** Some CI systems may check out submodules by default, causing an unintentional, resource-intensive checkout of the large Test262 repository. The impact and configuration options for common CI systems require investigation. +* **CI Performance:** A full Test262 run is lengthy. It is anticipated that full runs will be scheduled on a nightly or weekly basis, not on every pull request. +* **Initial Source Checkout:** The first-time fetch of the Test262 repository will cause a one-time delay. +* **Source Code Visibility on wpt.fyi:** Linking to test source on `wpt.fyi` is not straightforward. A solution would require `wpt.fyi` to have special-case logic for Test262, including access to a manifest mapping test names to file paths and knowledge of the pinned submodule SHA for a given run. ## Proof of Concept -A prototype implementation of the required harness, server, and manifest changes exists in the following commit. - -**Link:** [https://github.com/o-/wpt/commit/baa8415800161122e7d9d1170513f5aad82a4504](https://github.com/o-/wpt/commit/baa8415800161122e7d9d1170513f5aad82a4504) +A prototype implementation exists in the following commit: +[https://github.com/o-/wpt/commit/baa8415800161122e7d9d1170513f5aad82a4504](https://github.com/o-/wpt/commit/baa8415800161122e7d9d1170513f5aad82a4504) -*Caveat: As part of the final implementation, the YAML parsing logic (`parseTestRecord.py`) will be moved to a more Test262-specific location (e.g., `tools/test262/`) to clarify that it is not a generic WPT manifest parser.* +*Caveat: The YAML parsing logic in this prototype will be moved to a more Test262-specific location.* From 0e196b067a4bb6c1d6ea942afd9156d62f881073 Mon Sep 17 00:00:00 2001 From: James Scott Date: Wed, 12 Nov 2025 16:47:50 +0000 Subject: [PATCH 5/6] address feedback --- rfcs/test262_integration.md | 70 ++++++++++++++++++++----------------- 1 file changed, 37 insertions(+), 33 deletions(-) diff --git a/rfcs/test262_integration.md b/rfcs/test262_integration.md index 282b5f0..65e5867 100644 --- a/rfcs/test262_integration.md +++ b/rfcs/test262_integration.md @@ -2,69 +2,73 @@ ## Summary -This RFC proposes integrating the Test262 (ECMAScript) test suite into WPT. Due to its large size (>50,000 files), tests will be fetched on-demand rather than vendored into the repository. Execution will be opt-in via a new `--test262` flag. This provides a unified way to run this conformance suite while protecting the WPT repository from bloat and performance degradation. +This RFC proposes integrating the Test262 (ECMAScript) test suite into WPT by vendoring it directly into the repository under `tc39/test262`. Execution will be opt-in via the existing `--test-types=test262` flag. This provides a unified and simple way for consumers to run this conformance suite. ## Motivation * **Browser-based Testing:** Run Test262 in real browsers (stable and experimental), not just JS shells. This provides a more complete conformance signal for how ECMAScript behaves in the full web platform environment. * **Unified Infrastructure:** Use the existing WPT infrastructure (`wpt`, CI) and `wpt.fyi` for test execution and results analysis. * **Operational Efficiency:** Remove the need for vendors to maintain separate infrastructure for running Test262 and correlating its results with WPT. +* **Interop:** Allow ECMAScript features to be proposed and selected for inclusion in cross-browser Interop efforts. ## Detailed Design ### 1. Source Management -The WPT repository will contain a reference to the official `tc39/test262` repository (e.g., via a `.gitmodules` entry). This reference specifies the expected local path for the Test262 source and a pinned commit SHA. +The Test262 test suite will be vendored directly into the WPT repository under the `tc39/test262/` directory. This path structure follows the precedent set by other external test suites like the WebAssembly tests, which are located in `wasm/core/`. The top-level `tc39/` directory serves as a namespace for tests originating from that standards body. -* The pinned commit SHA ties each WPT commit to a precise version of Test262, ensuring reproducible test runs. -* An automated process (e.g., a weekly GitHub Action) will create pull requests to update this pinned SHA, providing a regular cadence for ingesting upstream updates. - -See Section 3 for details on how the Test262 source is acquired. +* A dedicated CI job will be responsible for keeping this vendored copy up-to-date, similar to the `update-wasm-tests.yml` workflow. +* This CI job will periodically: + 1. Checkout the official `tc39/test262` repository into a temporary location. + 2. Copy the relevant Test262 source directories (`test/` and `harness/`) into the `tc39/test262` directory within the WPT repository. WPT's manifest generation and serving logic will then interpret Test262's metadata and dynamically configure the test environment at runtime. + 3. Create a new branch, commit the updated Test262 files, and open a pull request against the `main` branch. The body of the pull request will note the full commit SHA from the `tc39/test262` repository that is being vendored, ensuring traceability. +* The update PR will include a smoketest to ensure that the Test262 integration has not been broken by the upstream changes. This smoketest will consist of a small, dedicated suite of tests located in a new `infrastructure/test262/` directory. These tests will live permanently in the WPT repository and will be designed to verify the key integration points between `wptrunner` and the vendored Test262 files, such as correctly parsing Test262's YAML frontmatter (e.g., for `negative` tests or `flags`) and ensuring Test262's harness files (e.g., `assert.js`) are properly injected and functional. The expected results for these smoketests will be defined in corresponding `.ini` files within the `infrastructure/metadata/test262/` directory, following the established WPT metadata pattern where each test file can have its own metadata file (e.g., the test `infrastructure/testharness/lone-surrogates.html` has its metadata defined in `infrastructure/metadata/infrastructure/testharness/lone-surrogates.html.ini`). Crucially, this smoketest is the only Test262-related suite that runs as a blocking check in the update pull request. The newly vendored tests themselves are executed against browsers later, as part of the regularly scheduled full test runs (e.g., nightly), which populate the results on wpt.fyi. +* This approach ensures that WPT consumers have no novel requirements to run these tests, beyond selecting the new test type. ### 2. Test Integration -WPT's manifest generation and serving logic will be extended to support the on-demand nature of Test262 tests. +WPT's manifest generation will be extended to support Test262. -* **Manifest Generation:** The `wpt manifest` command will recognize the Test262 source directory and traverse its `test` directory to discover tests. -* **Metadata Parsing:** A new, Test262-specific parser will read the YAML frontmatter from each `.js` file to extract metadata (e.g., ES features, negative test expectations, execution flags). -* **Harness and Server:** Specialized Test262 harness files and `wptserve` handlers will use the parsed metadata to construct the correct execution environment and generate distinct URLs for different test modes. +* **Manifest Generation:** The `wpt manifest` command will recognize the `tc39/test262` directory and traverse its `test` directory to discover tests. All discovered tests will be added to the main `MANIFEST.json`. +* **Metadata Parsing:** A new, Test262-specific parser will read the YAML frontmatter from each `.js` file to extract metadata. This is a key difference from other vendored suites like the `wasm/core` tests, which are pre-processed into self-contained HTML files. Test262's reliance on YAML frontmatter necessitates a metadata-driven approach. This metadata (e.g., ES features, negative test expectations, execution flags) will be used by the `wptserve` handlers at runtime to dynamically configure the test environment. +* **Harness and Server:** Specialized Test262 harness files and `wptserve` handlers will use the parsed metadata to construct the correct execution environment. Different test modes will be served via distinct URL conventions based on file extensions (e.g., a test at `path/to/test.js` might be served as `path/to/test.test262.html` for a normal test or `path/to/test.test262-module.html` for a module test). -### 3. Acquiring Test262 Source Code +### 3. Execution Control -The Test262 source code is acquired on-demand. `wpt` can handle this automatically for users in a Git environment, while providing a manual path for CI systems. +Running Test262 is **opt-in**. -When the `--test262` flag is used, `wpt` first checks if the Test262 source directory exists. If missing, `wpt` determines if the main WPT repository is a Git checkout. -- **If it is a Git repository,** `wpt` will attempt to run `git submodule update --init --recursive`. -- **If it is not a Git repository,** `wpt` will fail with an error, instructing the user to acquire the source manually. +* **Default Behavior:** By default, `wpt run` will not include Test262 tests. +* **Opt-in Flag:** The existing `--test-types` flag will be used to select Test262 tests. +* **Flag Behavior:** When `--test-types=test262` is present, `wpt` will include the Test262 tests from the `tc39/test262` directory. +* **CI Integration:** CI systems can enable Test262 runs by simply adding `--test-types=test262` to their `wpt run` command. -This allows CI systems and vendors with non-Git checkouts to use custom tooling, while providing a seamless experience for the common user. +## Alternatives Considered -### 4. Execution Control +### Storing Test262 Metadata in the Manifest -Running Test262 is **opt-in**. +Storing the parsed YAML metadata from Test262 files directly within the `MANIFEST.json` entry for each test was considered. This approach would offer significant performance benefits by avoiding the need to re-read and re-parse the YAML frontmatter from each `.js` file at runtime when a test is served. However, for the initial implementation, this optimization is being deferred to prioritize a simpler manifest generation process. The metadata will instead be parsed dynamically by `wptserve` handlers at runtime. -* **Default Behavior:** By default, `wpt run` will not look for the Test262 directory and will not include its tests. -* **Opt-in Flag:** A new `--test262` flag will be added to `wpt run`. -* **Flag Behavior:** When present, `wpt` ensures the Test262 source code is available (fetching it if possible, per Section 3) and includes the Test262 tests **in addition to** the standard WPT tests. -* **CI Integration:** CI systems can enable Test262 runs by adding this flag, after ensuring the source has been acquired. +### Pre-generating HTML Wrappers -## Alternatives Considered +An alternative to dynamic HTML wrapper generation at serve-time, as done for some other vendored test suites (e.g., WebAssembly), would be to pre-generate static `.js.html` files for each Test262 test during the vendoring process. This approach was considered but rejected for the following reasons: + +* **Repository Bloat:** Test262 contains over 50,000 `.js` test files. Pre-generating a corresponding HTML wrapper for each would double the number of files, significantly increasing repository size and potentially impacting filesystem performance. +* **Complexity of Multiple Contexts:** Test262 tests can be executed in various contexts (e.g., as a raw script, as a module, in strict mode, non-strict mode). A single `.js` file might need to be served with different HTML wrappers depending on the desired execution mode. Pre-generating all possible combinations would lead to a combinatorial explosion of static files and introduce significant complexity in managing and updating them. -### Using `--test-types=test262` +Dynamic HTML wrapper generation at serve-time, driven by Test262's metadata, offers a more flexible and efficient solution by constructing the appropriate test environment on-the-fly without duplicating files in the repository. -This was rejected because Test262 is fundamentally different from internal WPT test types (e.g., `testharness`, `reftest`). It is an external suite with its own metadata format and harness. A dedicated `--test262` flag provides a clearer separation of concerns and a cleaner implementation. +### Using a dedicated `--test262` flag + +This was considered to provide a clear separation from WPT's internal test types. However, using the existing `--test-types` flag is more consistent with how other test types are selected and is the preferred approach. ## Implementation Considerations & Risks -* **Vendor Integration:** The responsibility for acquiring the Test262 source is left to the user/CI system. This allows vendors with non-Git source control to implement their own sync mechanisms. -* **CI Submodule Behavior:** Some CI systems may check out submodules by default, causing an unintentional, resource-intensive checkout of the large Test262 repository. The impact and configuration options for common CI systems require investigation. +* **Repository Size:** Vendoring Test262 will significantly increase the size of the WPT repository. While the number of files is large (~50,000), this is a manageable increase on the current repository size. * **CI Performance:** A full Test262 run is lengthy. It is anticipated that full runs will be scheduled on a nightly or weekly basis, not on every pull request. -* **Initial Source Checkout:** The first-time fetch of the Test262 repository will cause a one-time delay. -* **Source Code Visibility on wpt.fyi:** Linking to test source on `wpt.fyi` is not straightforward. A solution would require `wpt.fyi` to have special-case logic for Test262, including access to a manifest mapping test names to file paths and knowledge of the pinned submodule SHA for a given run. +* **Upstream Changes:** Care must be taken when updating the vendored Test262 tests to ensure that any changes to the test harness or metadata format are handled correctly in the WPT integration. The CI job that opens update PRs should include a smoketest to catch basic integration issues. +* **Unit Tests for New Code:** Any new Python code developed for Test262 integration (e.g., for manifest parsing or serving logic) will be covered by its own dedicated unit tests, separate from the integration-focused smoketests. ## Proof of Concept -A prototype implementation exists in the following commit: -[https://github.com/o-/wpt/commit/baa8415800161122e7d9d1170513f5aad82a4504](https://github.com/o-/wpt/commit/baa8415800161122e7d9d1170513f5aad82a4504) - -*Caveat: The YAML parsing logic in this prototype will be moved to a more Test262-specific location.* +A prototype implementation of the Test262 integration exists in the following pull request: +[https://github.com/web-platform-tests/wpt/pull/55997](https://github.com/web-platform-tests/wpt/pull/55997) From b19bb88d9efcff8d0c2505caf120e28c26877993 Mon Sep 17 00:00:00 2001 From: James Scott Date: Wed, 12 Nov 2025 17:11:01 +0000 Subject: [PATCH 6/6] edits --- rfcs/test262_integration.md | 55 +++++++++++++++++++------------------ 1 file changed, 29 insertions(+), 26 deletions(-) diff --git a/rfcs/test262_integration.md b/rfcs/test262_integration.md index 65e5867..eeff7a4 100644 --- a/rfcs/test262_integration.md +++ b/rfcs/test262_integration.md @@ -2,7 +2,7 @@ ## Summary -This RFC proposes integrating the Test262 (ECMAScript) test suite into WPT by vendoring it directly into the repository under `tc39/test262`. Execution will be opt-in via the existing `--test-types=test262` flag. This provides a unified and simple way for consumers to run this conformance suite. +This RFC integrates the Test262 (ECMAScript) test suite into WPT by vendoring it directly under `tc39/test262`. Execution is opt-in via `--test-types=test262`. ## Motivation @@ -15,47 +15,49 @@ This RFC proposes integrating the Test262 (ECMAScript) test suite into WPT by ve ### 1. Source Management -The Test262 test suite will be vendored directly into the WPT repository under the `tc39/test262/` directory. This path structure follows the precedent set by other external test suites like the WebAssembly tests, which are located in `wasm/core/`. The top-level `tc39/` directory serves as a namespace for tests originating from that standards body. +The Test262 test suite will be vendored directly into the WPT repository under `tc39/test262/`. This path mirrors external test suites like WebAssembly (`wasm/core/`), with `tc39/` serving as a namespace for TC39-originated tests. -* A dedicated CI job will be responsible for keeping this vendored copy up-to-date, similar to the `update-wasm-tests.yml` workflow. -* This CI job will periodically: - 1. Checkout the official `tc39/test262` repository into a temporary location. - 2. Copy the relevant Test262 source directories (`test/` and `harness/`) into the `tc39/test262` directory within the WPT repository. WPT's manifest generation and serving logic will then interpret Test262's metadata and dynamically configure the test environment at runtime. - 3. Create a new branch, commit the updated Test262 files, and open a pull request against the `main` branch. The body of the pull request will note the full commit SHA from the `tc39/test262` repository that is being vendored, ensuring traceability. -* The update PR will include a smoketest to ensure that the Test262 integration has not been broken by the upstream changes. This smoketest will consist of a small, dedicated suite of tests located in a new `infrastructure/test262/` directory. These tests will live permanently in the WPT repository and will be designed to verify the key integration points between `wptrunner` and the vendored Test262 files, such as correctly parsing Test262's YAML frontmatter (e.g., for `negative` tests or `flags`) and ensuring Test262's harness files (e.g., `assert.js`) are properly injected and functional. The expected results for these smoketests will be defined in corresponding `.ini` files within the `infrastructure/metadata/test262/` directory, following the established WPT metadata pattern where each test file can have its own metadata file (e.g., the test `infrastructure/testharness/lone-surrogates.html` has its metadata defined in `infrastructure/metadata/infrastructure/testharness/lone-surrogates.html.ini`). Crucially, this smoketest is the only Test262-related suite that runs as a blocking check in the update pull request. The newly vendored tests themselves are executed against browsers later, as part of the regularly scheduled full test runs (e.g., nightly), which populate the results on wpt.fyi. -* This approach ensures that WPT consumers have no novel requirements to run these tests, beyond selecting the new test type. +* A dedicated CI job, similar to `update-wasm-tests.yml`, will keep this vendored copy up-to-date. It will periodically: + 1. Checkout the official `tc39/test262` repository. + 2. Copy the `test/` and `harness/` directories into `tc39/test262`. + 3. Open a PR against `main`, noting the upstream commit SHA in the body for traceability. +* The update PR will include a smoketest to ensure Test262 integration is not broken. This small, dedicated test suite in `infrastructure/test262/` will verify key integration points (e.g., YAML frontmatter parsing for `negative` tests/`flags`, `assert.js` harness injection). Expected results will be in corresponding `.ini` files within `infrastructure/metadata/test262/` (e.g., `infrastructure/testharness/lone-surrogates.html` metadata in `infrastructure/metadata/infrastructure/testharness/lone-surrogates.html.ini`). This smoketest is the *only* blocking Test262-related check in the PR; full test runs occur in scheduled nightly runs, populating wpt.fyi. +* This approach ensures WPT consumers have no novel requirements beyond selecting the test type. ### 2. Test Integration WPT's manifest generation will be extended to support Test262. -* **Manifest Generation:** The `wpt manifest` command will recognize the `tc39/test262` directory and traverse its `test` directory to discover tests. All discovered tests will be added to the main `MANIFEST.json`. -* **Metadata Parsing:** A new, Test262-specific parser will read the YAML frontmatter from each `.js` file to extract metadata. This is a key difference from other vendored suites like the `wasm/core` tests, which are pre-processed into self-contained HTML files. Test262's reliance on YAML frontmatter necessitates a metadata-driven approach. This metadata (e.g., ES features, negative test expectations, execution flags) will be used by the `wptserve` handlers at runtime to dynamically configure the test environment. -* **Harness and Server:** Specialized Test262 harness files and `wptserve` handlers will use the parsed metadata to construct the correct execution environment. Different test modes will be served via distinct URL conventions based on file extensions (e.g., a test at `path/to/test.js` might be served as `path/to/test.test262.html` for a normal test or `path/to/test.test262-module.html` for a module test). +* **Manifest Generation:** `wpt manifest` will, by default, recognize `tc39/test262`, traverse its `test` directory, and add discovered tests to `MANIFEST.json`. +* **Metadata Parsing:** A new Test262-specific parser will read YAML frontmatter from `.js` files to extract metadata (e.g., ES features, negative expectations, execution flags). Unlike `wasm/core` tests, which are pre-processed, Test262 metadata will be used by `wptserve` handlers at runtime. +* **Harness and Server:** Specialized Test262 harness files and `wptserve` handlers will use parsed metadata to construct the execution environment. Test modes will use distinct URL conventions based on file extensions (e.g., `path/to/test.js` served as `path/to/test.test262.html` or `path/to/test.test262-module.html`). ### 3. Execution Control Running Test262 is **opt-in**. -* **Default Behavior:** By default, `wpt run` will not include Test262 tests. -* **Opt-in Flag:** The existing `--test-types` flag will be used to select Test262 tests. -* **Flag Behavior:** When `--test-types=test262` is present, `wpt` will include the Test262 tests from the `tc39/test262` directory. -* **CI Integration:** CI systems can enable Test262 runs by simply adding `--test-types=test262` to their `wpt run` command. +* **Default Behavior:** `wpt run` will not include Test262 tests by default, even if they are present in the manifest. +* **Opt-in Flag:** The existing `--test-types` flag selects Test262 tests. +* **Flag Behavior:** When `--test-types=test262` is present, `wpt` includes Test262 tests from `tc39/test262`. +* **CI Integration:** CI systems enable Test262 runs by adding `--test-types=test262` to their `wpt run` command. ## Alternatives Considered ### Storing Test262 Metadata in the Manifest -Storing the parsed YAML metadata from Test262 files directly within the `MANIFEST.json` entry for each test was considered. This approach would offer significant performance benefits by avoiding the need to re-read and re-parse the YAML frontmatter from each `.js` file at runtime when a test is served. However, for the initial implementation, this optimization is being deferred to prioritize a simpler manifest generation process. The metadata will instead be parsed dynamically by `wptserve` handlers at runtime. +Storing parsed YAML metadata from Test262 files directly within `MANIFEST.json` was considered. This is the standard WPT architecture for other dynamic test types (e.g., Workers, Window tests) to optimize runtime performance. + +The current dynamic approach requires two disk I/O operations per test: one for the `wptserve` handler to parse YAML frontmatter, and a second for the browser to fetch the `.js` content. Storing metadata in `MANIFEST.json` would eliminate the first disk I/O and YAML parsing, effectively halving disk I/O operations per test during runtime. + +However, this optimization is deferred for initial implementation simplicity. Metadata will be parsed dynamically by `wptserve` handlers at runtime. ### Pre-generating HTML Wrappers -An alternative to dynamic HTML wrapper generation at serve-time, as done for some other vendored test suites (e.g., WebAssembly), would be to pre-generate static `.js.html` files for each Test262 test during the vendoring process. This approach was considered but rejected for the following reasons: +Pre-generating static `.js.html` files for each Test262 test (as done for some other vendored suites like WebAssembly) was considered but rejected due to: -* **Repository Bloat:** Test262 contains over 50,000 `.js` test files. Pre-generating a corresponding HTML wrapper for each would double the number of files, significantly increasing repository size and potentially impacting filesystem performance. -* **Complexity of Multiple Contexts:** Test262 tests can be executed in various contexts (e.g., as a raw script, as a module, in strict mode, non-strict mode). A single `.js` file might need to be served with different HTML wrappers depending on the desired execution mode. Pre-generating all possible combinations would lead to a combinatorial explosion of static files and introduce significant complexity in managing and updating them. +* **Complexity of Multiple Contexts:** Test262 tests execute in various contexts (e.g., module, strict). Pre-generating wrappers for all combinations would lead to a combinatorial explosion of static files, adding potentially far more than 50,000 new files and creating a significant management burden. -Dynamic HTML wrapper generation at serve-time, driven by Test262's metadata, offers a more flexible and efficient solution by constructing the appropriate test environment on-the-fly without duplicating files in the repository. +Dynamic HTML wrapper generation at serve-time, driven by Test262's metadata, offers a more flexible and efficient solution without duplicating repository files. ### Using a dedicated `--test262` flag @@ -63,12 +65,13 @@ This was considered to provide a clear separation from WPT's internal test types ## Implementation Considerations & Risks -* **Repository Size:** Vendoring Test262 will significantly increase the size of the WPT repository. While the number of files is large (~50,000), this is a manageable increase on the current repository size. -* **CI Performance:** A full Test262 run is lengthy. It is anticipated that full runs will be scheduled on a nightly or weekly basis, not on every pull request. -* **Upstream Changes:** Care must be taken when updating the vendored Test262 tests to ensure that any changes to the test harness or metadata format are handled correctly in the WPT integration. The CI job that opens update PRs should include a smoketest to catch basic integration issues. -* **Unit Tests for New Code:** Any new Python code developed for Test262 integration (e.g., for manifest parsing or serving logic) will be covered by its own dedicated unit tests, separate from the integration-focused smoketests. +* **Manifest Generation Performance:** Including the 50,000+ Test262 tests in the manifest by default will increase the time required to run `wpt manifest`. This performance cost is accepted to ensure a simpler user experience, where a single, default manifest contains all tests. +* **Repository Size:** Vendoring Test262 will significantly increase WPT repository size (~50,000 files), a manageable increase. +* **CI Performance:** Full Test262 runs are lengthy. They will be scheduled during the existing nighly builds, not on every pull request. +* **Upstream Changes:** Updating vendored Test262 tests requires care to ensure changes to the test harness or metadata format are handled correctly. The CI job's update PRs will include a smoketest for basic integration issues. +* **Unit Tests for New Code:** New Python code for Test262 integration (e.g., manifest parsing, serving logic) will have dedicated unit tests, separate from integration smoketests. ## Proof of Concept -A prototype implementation of the Test262 integration exists in the following pull request: +A prototype Test262 integration exists in: [https://github.com/web-platform-tests/wpt/pull/55997](https://github.com/web-platform-tests/wpt/pull/55997)