Async Runner with Message Channel #458

flashdesignory · 2024-11-20T00:56:54Z

As discussed in our sync, here is a combination of the AsyncRunner and Message Channel pr.

Relevant code snippet (udpated):

class AsyncRAFTestInvoker extends BaseRAFTestInvoker {
    static mc = new MessageChannel();
    _scheduleCallbacks(resolve) {
        let gotTimer = false;
        let gotMessage = false;
        let gotPromise = false;

        const tryTriggerAsyncCallback = () => {
            if (!gotTimer || !gotMessage || !gotPromise)
                return;

            this._asyncCallback();
            setTimeout(async () => {
                await this._reportCallback();
                resolve();
            }, 0);
        };

        requestAnimationFrame(async () => {
            await this._syncCallback();
            gotPromise = true;
            tryTriggerAsyncCallback();
        });

        requestAnimationFrame(() => {
            setTimeout(() => {
                gotTimer = true;
                tryTriggerAsyncCallback();
            });

            AsyncRAFTestInvoker.mc.port1.addEventListener(
                "message",
                function () {
                    gotMessage = true;
                    tryTriggerAsyncCallback();
                },
                { once: true }
            );
            AsyncRAFTestInvoker.mc.port1.start();
            AsyncRAFTestInvoker.mc.port2.postMessage("speedometer");
        });
    }
}

For testing purposes I kept an explicit type of "async" in the test file, but I opted all default suites into using it.

Initial testing shows that it should also fix the react specific issue.

Other issue solved by this pr: #83

flashdesignory · 2024-11-21T02:15:43Z

resources/test-invoker.mjs

+
+export const ASYNC_TEST_INVOKER_LOOKUP = {
+    __proto__: null,
+    timer: AsyncTimerTestInvoker,


If we don't want an AsyncTimerTestInvoker, I could simply assign the same AsyncRAFTestInvoker..

resources/test-invoker.mjs

smaug---- · 2024-12-11T11:09:16Z

resources/tests.mjs

@@ -125,6 +125,7 @@ Suites.push({
    name: "TodoMVC-JavaScript-ES5",
    url: "resources/todomvc/vanilla-examples/javascript-es5/dist/index.html",
    tags: ["todomvc"],
+    type: "async",


So in practice all the tests are now async?

This can be reverted. For testing I marked all default tests async.
How do we decide which tests should be async? I know React / Next.js had issues with not capturing all work. Those could be candidates to mark async?

Those being async sounds good, and perhaps everything in the future should be async by default.

I'll leave it unchanged (opting every single workload into async steps), until I get more feedback from other folks.

resources/tests.mjs

smaug---- · 2024-12-11T21:32:42Z

Looks reasonable. @julienw do you think you could have this in some public place for testing? I could do some profiling to check if there is anything unexpected happening.

rniwa · 2024-12-11T21:43:38Z

resources/test-invoker.mjs

@@ -24,7 +24,24 @@ export class TimerTestInvoker extends TestInvoker {
    }
 }

-export class RAFTestInvoker extends TestInvoker {


I'd prefer removing the TimerTestInvoker entirely since we've moved to RAF-based measurement in 3 and we haven't seen any issue with it.

rniwa · 2024-12-11T22:44:15Z

Can we split the change to remove measurementMethod into a separate PR? We should also update params.mjs and developer-mode.mjs.

flashdesignory · 2024-12-11T22:47:29Z

Can we split the change to remove measurementMethod into a separate PR? We should also update params.mjs and developer-mode.mjs.

ok, reverted for now. I agree a cleanup pr to remove this would be better.

bgrins · 2024-12-17T17:44:54Z

@smaug---- I pushed this to https://feature-async-runner-message-channel--speedometer-preview.netlify.app/ for anyone who wants to test.

rniwa · 2024-12-18T17:36:42Z

resources/shared/test-runner.mjs

+        return `${this.#suite.name}.${this.#test.name}-sync-end`;
+    }
+
+    get asyncStartLabel() {


Do we really need to add all these getters now that we're only overriding _runSyncStep?

I can revert this.

rniwa · 2024-12-18T17:39:33Z

resources/shared/test-invoker.mjs

@@ -24,7 +24,24 @@ export class TimerTestInvoker extends TestInvoker {
    }
 }

-export class RAFTestInvoker extends TestInvoker {
+class AsyncTimerTestInvoker extends TestInvoker {


Can we instead just delete TimerTestInvoker as we discussed in a separate PR?

We wanted to do this in a follow up pr, correct?
If that changed, i'm happy to remove it in this pr already.

rniwa · 2024-12-18T17:41:31Z

resources/suite-runner.mjs

@@ -75,7 +75,8 @@ export class SuiteRunner {
            if (this.#client?.willRunTest)
                await this.#client.willRunTest(this.#suite, test);

-            const testRunner = new TestRunner(this.#frame, this.#page, this.#params, this.#suite, test, this._recordTestResults);
+            const testRunnerClass = TEST_RUNNER_LOOKUP[this.#suite.type ?? "default"];


Instead of changing the test runner based on the suite type for some suites, can we initially add a toggle to developer menu so that we can try out async runner?

To my understanding the agreement in the meeting was to unconditionally enable the new runner for the preact workloads and add the global developer-mode menu in a separate PR.

correct - below is the list on enabled suites (for now):

TodoMVC-React-Complex-DOM,

TodoMVC-React-Redux,

TodoMVC-Preact-Complex-DOM, (technically not react based, but it's similar)

NewsSite-Next,

React-Stockcharts-SVG

If we're doing that, we need before/after numbers across browsers.

I'll run crossbench for the ones on the list and post results here.

Here are metrics for the next.js workload:

flashdesignory added 15 commits October 30, 2024 17:09

git branch

b594115

initial change

f154a4c

add event

9459c00

add comment

d462764

rollback requestIdleCallback

d037f89

run format

79f8450

undo newssite changes

14f1f6d

remove test test

d08606d

Merge branch 'main' into feature/async-runner

4540937

merge main

a02d7cc

Merge branch 'main' into feature/async-runner

5281af4

Merge branch 'main' into feature/async-runner

d4fa570

cleanup

764b61b

reset news site build files

ace8c3b

initial refactor

147c38f

flashdesignory requested review from bgrins, camillobruni, rniwa and smaug---- November 20, 2024 00:57

flashdesignory commented Nov 21, 2024

View reviewed changes

flashdesignory marked this pull request as ready for review November 21, 2024 02:15

smaug---- reviewed Nov 22, 2024

View reviewed changes

resources/test-invoker.mjs Outdated Show resolved Hide resolved

smaug---- reviewed Nov 22, 2024

View reviewed changes

resources/test-invoker.mjs Outdated Show resolved Hide resolved

smaug---- reviewed Nov 22, 2024

View reviewed changes

resources/test-invoker.mjs Outdated Show resolved Hide resolved

flashdesignory added 2 commits December 10, 2024 12:02

Merge branch 'main' into feature/async-runner-message-channel

92f9024

pr feedback

7cca4e0

smaug---- reviewed Dec 11, 2024

View reviewed changes

resources/tests.mjs Outdated Show resolved Hide resolved

remove async from next.js steps

2b92c1f

flashdesignory requested a review from smaug---- December 11, 2024 13:47

use static Message Channel

acc7522

rniwa reviewed Dec 11, 2024

View reviewed changes

flashdesignory requested a review from rniwa December 11, 2024 22:39

flashdesignory force-pushed the feature/async-runner-message-channel branch from b24aa1c to acc7522 Compare December 11, 2024 22:46

flashdesignory requested a review from julienw December 12, 2024 12:43

flashdesignory mentioned this pull request Dec 17, 2024

Add Async Runner for News Site Next Testing #442

Open

flashdesignory added 5 commits December 17, 2024 18:05

Merge branch 'main' into feature/async-runner-message-channel

c778807

cleanup

211ba6a

cleanup test-runner

81b815e

fix tests

37ffb46

only opt in react based workloads

63910a3

rniwa reviewed Dec 18, 2024

View reviewed changes

remove label getters

6b6f8d4

rniwa added the major change A change with major implications on benchmark results or affect governance policy label Dec 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Async Runner with Message Channel #458

Async Runner with Message Channel #458

flashdesignory commented Nov 20, 2024 •

edited

Loading

flashdesignory Nov 21, 2024

smaug---- Dec 11, 2024

flashdesignory Dec 11, 2024

smaug---- Dec 11, 2024

flashdesignory Dec 11, 2024

smaug---- commented Dec 11, 2024

rniwa Dec 11, 2024

flashdesignory Dec 11, 2024

rniwa commented Dec 11, 2024

flashdesignory commented Dec 11, 2024

bgrins commented Dec 17, 2024

rniwa Dec 18, 2024

flashdesignory Dec 18, 2024

flashdesignory Dec 18, 2024

rniwa Dec 18, 2024

flashdesignory Dec 18, 2024

rniwa Dec 18, 2024

camillobruni Dec 18, 2024

flashdesignory Dec 18, 2024

rniwa Dec 18, 2024

flashdesignory Dec 18, 2024

flashdesignory Dec 18, 2024 •

edited

Loading

Async Runner with Message Channel #458

Are you sure you want to change the base?

Async Runner with Message Channel #458

Conversation

flashdesignory commented Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smaug---- commented Dec 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rniwa commented Dec 11, 2024

flashdesignory commented Dec 11, 2024

bgrins commented Dec 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flashdesignory Dec 18, 2024 • edited Loading

Choose a reason for hiding this comment

flashdesignory commented Nov 20, 2024 •

edited

Loading

flashdesignory Dec 18, 2024 •

edited

Loading