Mapping between Test262 and web-features

[foolip]

Let's find a way to find the tests for a given web-features identifier in Test262!

Background:

web-features is an effort to describe the whole web platform, including JavaScript, as a set of features. It's part of Baseline, which powers the "Baseline 2022" badge at the top of MDN pages, such as on Array.prototype.at().

Separately from Baseline, we have a mechanism for finding the tests in web-platform-tests that belong to a particular feature:
https://wpt.fyi/results/css/css-transforms?label=master&label=experimental&aligned&q=feature%3Atransforms3d
https://github.com/web-platform-tests/wpt/blob/master/css/css-transforms/WEB_FEATURES.yml

Alternatives:

Test262 already has a features.txt, so rather than mapping web-features to individual tests we should map to one of those features. There are two obvious options:

1. Maintain a mapping from Tests262 features to web-features in Test262, similar to how WPT controls the mapping from WPT to web-features
2. Add a test262 field to web-features accepting one or more Test262 feature identifiers

With one of these in place, it should be possible to list all tests for a given JavaScript features in web-features. This would help web-features maintainers understand the level of interoperability when determining the Baseline status of JavaScript features.

[nicolo-ribaudo]

For ecmascript features, could web-features just reuse the same name as in test262? Test262 tests are written before that implementations ship, so the identifier should always be available when you'll want to include it in web-features.

[foolip]

I think we could a lot of the time, but not always. Our identifiers cannot have periods, and we might have larger groupings. For example, we have "array-flat" where Test262 has "Array.prototype.flat" + "Array.prototype.flatMap".

[mathiasbynens]

IIUC, there’s a few separate requests here, and it’s worth considering them separately:

1. For each Test262 feature, you want the complete list of Test262 tests tagged with that feature. That mapping is something the Test262 project could provide as a JSON build artifact, potentially published for each commit. (Of course, this could be built as a standalone project that pulls in Test262, but ideally it’s done upstream.)

2. Then, web-features could consume that JSON file to figure out the overall test pass rate for each feature.

3. The mapping from web-features identifiers to Test262 feature identifiers is also something web-features could maintain. It could be done at the Test262 level too, but assuming point 1 above is solved in some way, it doesn’t seem necessary.

The biggest missing piece that affects the tc39/test262 project seems to be the first item. Everything else can be built on top of that.

[nicolo-ribaudo]

With https://github.com/bocoup/test262-stream, the first step is trivial:

await new Test262Stream("./test262")
  .filter(test => test.attrs.features?.includes("Array.prototype.flat"))
  .map(test => test.file)
  .toArray();

[mathiasbynens]

With https://github.com/bocoup/test262-stream, the first step is trivial:

await new Test262Stream("./test262")
  .filter(test => test.attrs.features?.includes("Array.prototype.flat"))
  .map(test => test.file)
  .toArray();

TIL about test262-stream — that makes that part easier indeed:

import {default as Test262Stream} from 'test262-stream';

const stream = new Test262Stream('../test262');

const featureIdentifierToTests = new Map();

for await (const test of stream) {
  const filePath = test.file;
  if (!test.attrs.features) continue;
  for (const feature of test.attrs.features) {
    if (featureIdentifierToTests.has(feature)) {
      featureIdentifierToTests.get(feature).add(filePath);
    } else {
      featureIdentifierToTests.set(feature, new Set([filePath]));
    }
  }
}

console.log(featureIdentifierToTests);

Example output:

Map(169) {
  // …
  'Array.prototype.flat' => Set(15) {
    'test/built-ins/Array/prototype/methods-called-as-functions.js',
    'test/built-ins/Array/prototype/flat/array-like-objects.js',
    'test/built-ins/Array/prototype/flat/bound-function-call.js',
    'test/built-ins/Array/prototype/flat/empty-array-elements.js',
    'test/built-ins/Array/prototype/flat/empty-object-elements.js',
    'test/built-ins/Array/prototype/flat/length.js',
    'test/built-ins/Array/prototype/flat/name.js',
    'test/built-ins/Array/prototype/flat/non-numeric-depth-should-not-throw.js',
    'test/built-ins/Array/prototype/flat/non-object-ctor-throws.js',
    'test/built-ins/Array/prototype/flat/null-undefined-elements.js',
    'test/built-ins/Array/prototype/flat/null-undefined-input-throws.js',
    'test/built-ins/Array/prototype/flat/positive-infinity.js',
    'test/built-ins/Array/prototype/flat/prop-desc.js',
    'test/built-ins/Array/prototype/flat/proxy-access-count.js',
    'test/built-ins/Array/prototype/flat/symbol-object-create-null-depth-throws.js'
  },
  'Array.prototype.flatMap' => Set(20) {
    'test/built-ins/Array/prototype/methods-called-as-functions.js',
    'test/built-ins/Array/prototype/flatMap/array-like-objects-nested.js',
    'test/built-ins/Array/prototype/flatMap/array-like-objects-poisoned-length.js',
    'test/built-ins/Array/prototype/flatMap/array-like-objects-typedarrays.js',
    'test/built-ins/Array/prototype/flatMap/array-like-objects.js',
    'test/built-ins/Array/prototype/flatMap/bound-function-argument.js',
    'test/built-ins/Array/prototype/flatMap/length.js',
    'test/built-ins/Array/prototype/flatMap/depth-always-one.js',
    'test/built-ins/Array/prototype/flatMap/name.js',
    'test/built-ins/Array/prototype/flatMap/non-callable-argument-throws.js',
    'test/built-ins/Array/prototype/flatMap/not-a-constructor.js',
    'test/built-ins/Array/prototype/flatMap/prop-desc.js',
    'test/built-ins/Array/prototype/flatMap/proxy-access-count.js',
    'test/built-ins/Array/prototype/flatMap/this-value-ctor-non-object.js',
    'test/built-ins/Array/prototype/flatMap/this-value-ctor-object-species-bad-throws.js',
    'test/built-ins/Array/prototype/flatMap/this-value-ctor-object-species-custom-ctor-poisoned-throws.js',
    'test/built-ins/Array/prototype/flatMap/this-value-ctor-object-species-custom-ctor.js',
    'test/built-ins/Array/prototype/flatMap/this-value-ctor-object-species.js',
    'test/built-ins/Array/prototype/flatMap/this-value-null-undefined-throws.js',
    'test/built-ins/Array/prototype/flatMap/thisArg-argument.js'
  },
  // …
}

[foolip]

That's very nice! It look like I still need a Test262 checkout, so a build step to create these mappings as build artifacts would be great. In WPT we do this as artifacts published for every merged PR. Example:
https://github.com/web-platform-tests/wpt/releases/tag/merge_pr_45598

[ljharb]

Why can't your identifiers have periods?

[foolip]

That's a self-imposed limitation that we never discussed much, but it's the style of caniuse identifiers and we do want web-features to mix well with caniuse.

[ljharb]

@foolip https://caniuse.com/?search=array.prototype.flat seems to work just fine.

[foolip]

https://caniuse.com/?search=flatten also works, but the canonical URL is https://caniuse.com/array-flat, with the identifier "array-flat" coming from this file:
https://github.com/Fyrd/caniuse/blob/main/features-json/array-flat.json