Combined update graph experiments #60800

DanielRosenwasser · 2024-12-17T21:45:53Z

Merges #60754 and #60755 to see the combined effect on addressing #60633.

…dentical config files

…Experiments

DanielRosenwasser · 2024-12-17T21:46:01Z

typescript-bot · 2024-12-17T21:46:05Z

Starting jobs; this comment will be updated as builds start and complete.

Command	Status	Results
`pack this`	✅ Started	✅ Results

typescript-bot · 2024-12-17T21:47:54Z

Hey @DanielRosenwasser, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/164413/artifacts?artifactName=tgz&fileId=7F1016401FADCAF8EABD4FD550DAA32A0C47E4C9963BB5155861BA888CA2485802&fileName=/typescript-5.8.0-insiders.20241217.tgz"
    }
}

and then running npm install.

There is also a playground for this build and an npm module you can use via "typescript": "npm:@typescript-deploys/[email protected]".;

DanielRosenwasser · 2024-12-17T21:51:51Z

@typescript-bot perf test this

typescript-bot · 2024-12-17T21:51:54Z

Starting jobs; this comment will be updated as builds start and complete.

Command	Status	Results
`perf test this`	✅ Started	👀 Results

typescript-bot · 2024-12-17T22:41:01Z

@DanielRosenwasser
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr

_Metric	_baseline	_pr	_Delta	_Best	_Worst	_p-value
_{Compiler-Unions - node (v18.15.0, x64)}
_Errors	₃₄	₃₄	_~	_~	_~	_{p=1.000 n=6}
_Symbols	_62,363	_62,363	_~	_~	_~	_{p=1.000 n=6}
_Types	_50,395	_50,395	_~	_~	_~	_{p=1.000 n=6}
_{Memory used}	_{195,565k (± 0.95%)}	_{196,716k (± 0.09%)}	_~	_196,490k	_196,854k	_{p=0.471 n=6}
_{Parse Time}	_{1.60s (± 2.42%)}	_{1.59s (± 0.52%)}	_~	_1.59s	_1.61s	_{p=0.513 n=6}
_{Bind Time}	_{0.86s (± 0.94%)}	_{0.86s (± 1.36%)}	_~	_0.85s	_0.88s	_{p=0.315 n=6}
_{Check Time}	_{11.78s (± 0.72%)}	_{11.74s (± 0.27%)}	_~	_11.70s	_11.78s	_{p=0.335 n=6}
_{Emit Time}	_{3.27s (± 1.30%)}	_{3.39s (± 3.95%)}	_~	_3.27s	_3.58s	_{p=0.078 n=6}
_{Total Time}	_{17.51s (± 0.65%)}	_{17.59s (± 0.69%)}	_~	_17.48s	_17.79s	_{p=0.172 n=6}
_{angular-1 - node (v18.15.0, x64)}
_Errors	₃₇	₃₇	_~	_~	_~	_{p=1.000 n=6}
_Symbols	_947,936	_947,936	_~	_~	_~	_{p=1.000 n=6}
_Types	_410,955	_410,955	_~	_~	_~	_{p=1.000 n=6}
_{Memory used}	_{1,225,998k (± 0.00%)}	_{1,225,824k (± 0.01%)}	_{-174k (- 0.01%)}	_1,225,707k	_1,225,915k	_{p=0.005 n=6}
_{Parse Time}	_{8.02s (± 0.92%)}	_{8.09s (± 0.82%)}	_{+0.07s (+ 0.85%)}	_8.03s	_8.21s	_{p=0.045 n=6}
_{Bind Time}	_{2.27s (± 0.71%)}	_{2.29s (± 0.66%)}	_{+0.03s (+ 1.18%)}	_2.28s	_2.32s	_{p=0.016 n=6}
_{Check Time}	_{38.21s (± 0.30%)}	_{38.28s (± 0.19%)}	_~	_38.21s	_38.38s	_{p=0.228 n=6}
_{Emit Time}	_{18.27s (± 0.42%)}	_{18.29s (± 0.27%)}	_~	_18.22s	_18.34s	_{p=0.574 n=6}
_{Total Time}	_{66.76s (± 0.30%)}	_{66.96s (± 0.10%)}	_~	_66.87s	_67.02s	_{p=0.065 n=6}
_{mui-docs - node (v18.15.0, x64)}
_Errors	₀	₀	_~	_~	_~	_{p=1.000 n=6}
_Symbols	_2,448,617	_2,448,617	_~	_~	_~	_{p=1.000 n=6}
_Types	_896,282	_896,282	_~	_~	_~	_{p=1.000 n=6}
_{Memory used}	_{2,320,747k (± 0.00%)}	_{2,320,028k (± 0.01%)}	_{-720k (- 0.03%)}	_2,319,804k	_2,320,194k	_{p=0.005 n=6}
_{Parse Time}	_{11.33s (± 0.58%)}	_{11.36s (± 0.38%)}	_~	_11.31s	_11.43s	_{p=0.470 n=6}
_{Bind Time}	_{2.66s (± 0.51%)}	_{2.65s (± 0.88%)}	_~	_2.62s	_2.69s	_{p=0.869 n=6}
_{Check Time}	_{88.72s (± 2.34%)}	_{89.93s (± 1.46%)}	_~	_87.97s	_91.70s	_{p=0.128 n=6}
_{Emit Time}	_{0.34s (± 2.90%)}	_{0.36s (± 3.26%)}	_{🔻+0.02s (+ 5.91%)}	_0.35s	_0.38s	_{p=0.015 n=6}
_{Total Time}	_{103.04s (± 1.98%)}	_{104.30s (± 1.25%)}	_~	_102.40s	_106.03s	_{p=0.128 n=6}
_{self-build-src - node (v18.15.0, x64)}
_Errors	₀	₀	_~	_~	_~	_{p=1.000 n=6}
_Symbols	_1,225,390	_1,225,444	_{+54 (+ 0.00%)}	_~	_~	_{p=0.001 n=6}
_Types	_266,584	_266,595	_{+11 (+ 0.00%)}	_~	_~	_{p=0.001 n=6}
_{Memory used}	_{2,842,573k (±13.19%)}	_{2,842,940k (±13.21%)}	_~	_2,357,394k	_3,086,316k	_{p=0.810 n=6}
_{Parse Time}	_{6.78s (± 1.66%)}	_{6.69s (± 2.08%)}	_~	_6.50s	_6.80s	_{p=0.261 n=6}
_{Bind Time}	_{2.13s (± 1.09%)}	_{2.15s (± 1.55%)}	_~	_2.10s	_2.19s	_{p=0.470 n=6}
_{Check Time}	_{42.75s (± 0.51%)}	_{42.72s (± 0.69%)}	_~	_42.28s	_43.05s	_{p=1.000 n=6}
_{Emit Time}	_{3.46s (± 3.10%)}	_{3.44s (± 1.45%)}	_~	_3.39s	_3.51s	_{p=0.748 n=6}
_{Total Time}	_{55.13s (± 0.71%)}	_{55.01s (± 0.74%)}	_~	_54.36s	_55.43s	_{p=0.575 n=6}
_{self-build-src-public-api - node (v18.15.0, x64)}
_Errors	₀	₀	_~	_~	_~	_{p=1.000 n=6}
_Symbols	_1,225,390	_1,225,444	_{+54 (+ 0.00%)}	_~	_~	_{p=0.001 n=6}
_Types	_266,584	_266,595	_{+11 (+ 0.00%)}	_~	_~	_{p=0.001 n=6}
_{Memory used}	_{3,150,446k (± 0.01%)}	_{3,029,884k (± 9.78%)}	_~	_2,424,404k	_3,151,951k	_{p=0.936 n=6}
_{Parse Time}	_{7.03s (± 0.62%)}	_{6.99s (± 2.01%)}	_~	_6.75s	_7.14s	_{p=0.873 n=6}
_{Bind Time}	_{2.13s (± 0.71%)}	_{2.13s (± 1.55%)}	_~	_2.10s	_2.19s	_{p=0.629 n=6}
_{Check Time}	_{43.01s (± 0.36%)}	_{42.83s (± 0.49%)}	_~	_42.43s	_42.99s	_{p=0.173 n=6}
_{Emit Time}	_{3.45s (± 2.03%)}	_{3.46s (± 2.60%)}	_~	_3.33s	_3.57s	_{p=0.810 n=6}
_{Total Time}	_{55.63s (± 0.33%)}	_{55.40s (± 0.58%)}	_~	_54.80s	_55.72s	_{p=0.230 n=6}
_{self-compiler - node (v18.15.0, x64)}
_Errors	₀	₀	_~	_~	_~	_{p=1.000 n=6}
_Symbols	_262,278	_262,299	_{+21 (+ 0.01%)}	_~	_~	_{p=0.001 n=6}
_Types	_106,628	_106,636	_{+8 (+ 0.01%)}	_~	_~	_{p=0.001 n=6}
_{Memory used}	_{439,852k (± 0.01%)}	_{440,116k (± 0.01%)}	_{+264k (+ 0.06%)}	_440,076k	_440,159k	_{p=0.005 n=6}
_{Parse Time}	_{2.88s (± 0.36%)}	_{2.89s (± 0.86%)}	_~	_2.86s	_2.91s	_{p=0.461 n=6}
_{Bind Time}	_1.10s	_1.10s	_~	_~	_~	_{p=1.000 n=6}
_{Check Time}	_{15.77s (± 0.12%)}	_{15.73s (± 0.20%)}	_{-0.04s (- 0.29%)}	_15.69s	_15.77s	_{p=0.028 n=6}
_{Emit Time}	_{1.31s (± 0.92%)}	_{1.30s (± 1.13%)}	_~	_1.29s	_1.33s	_{p=0.133 n=6}
_{Total Time}	_{21.06s (± 0.16%)}	_{21.02s (± 0.18%)}	_~	_20.97s	_21.07s	_{p=0.126 n=6}
_{ts-pre-modules - node (v18.15.0, x64)}
_Errors	₇₀	₇₀	_~	_~	_~	_{p=1.000 n=6}
_Symbols	_226,062	_226,062	_~	_~	_~	_{p=1.000 n=6}
_Types	_94,488	_94,488	_~	_~	_~	_{p=1.000 n=6}
_{Memory used}	_{371,577k (± 0.01%)}	_{371,598k (± 0.02%)}	_~	_371,525k	_371,699k	_{p=0.748 n=6}
_{Parse Time}	_{2.90s (± 1.33%)}	_{2.89s (± 1.22%)}	_~	_2.86s	_2.94s	_{p=0.686 n=6}
_{Bind Time}	_{1.58s (± 0.65%)}	_{1.60s (± 1.42%)}	_{+0.02s (+ 1.48%)}	_1.58s	_1.64s	_{p=0.040 n=6}
_{Check Time}	_{16.46s (± 0.44%)}	_{16.51s (± 0.24%)}	_~	_16.44s	_16.55s	_{p=0.146 n=6}
_{Emit Time}	_0.00s	_0.00s	_~	_~	_~	_{p=1.000 n=6}
_{Total Time}	_{20.94s (± 0.49%)}	_{21.00s (± 0.33%)}	_~	_20.92s	_21.09s	_{p=0.199 n=6}
_{vscode - node (v18.15.0, x64)}
_Errors	₃	₃	_~	_~	_~	_{p=1.000 n=6}
_Symbols	_3,220,282	_3,220,282	_~	_~	_~	_{p=1.000 n=6}
_Types	_1,107,900	_1,107,900	_~	_~	_~	_{p=1.000 n=6}
_{Memory used}	_{3,286,953k (± 0.01%)}	_{3,286,265k (± 0.01%)}	_{-688k (- 0.02%)}	_3,286,012k	_3,286,473k	_{p=0.005 n=6}
_{Parse Time}	_{14.09s (± 0.21%)}	_{14.06s (± 0.29%)}	_~	_14.00s	_14.11s	_{p=0.124 n=6}
_{Bind Time}	_{4.49s (± 0.44%)}	_{4.48s (± 0.39%)}	_~	_4.46s	_4.51s	_{p=0.224 n=6}
_{Check Time}	_{88.78s (± 1.40%)}	_{86.66s (± 0.41%)}	_{-2.12s (- 2.39%)}	_86.26s	_87.10s	_{p=0.031 n=6}
_{Emit Time}	_{27.70s (± 1.82%)}	_{28.49s (± 0.41%)}	_{+0.79s (+ 2.86%)}	_28.35s	_28.69s	_{p=0.005 n=6}
_{Total Time}	_{135.06s (± 0.67%)}	_{133.69s (± 0.33%)}	_{-1.37s (- 1.02%)}	_133.19s	_134.34s	_{p=0.037 n=6}
_{webpack - node (v18.15.0, x64)}
_Errors	₀	₀	_~	_~	_~	_{p=1.000 n=6}
_Symbols	_291,463	_291,463	_~	_~	_~	_{p=1.000 n=6}
_Types	_118,920	_118,920	_~	_~	_~	_{p=1.000 n=6}
_{Memory used}	_{445,314k (± 0.05%)}	_{445,167k (± 0.03%)}	_~	_444,998k	_445,349k	_{p=0.230 n=6}
_{Parse Time}	_{4.11s (± 0.81%)}	_{4.11s (± 1.36%)}	_~	_4.03s	_4.17s	_{p=1.000 n=6}
_{Bind Time}	_{1.77s (± 0.94%)}	_{1.76s (± 1.06%)}	_~	_1.74s	_1.79s	_{p=0.684 n=6}
_{Check Time}	_{18.89s (± 0.31%)}	_{18.77s (± 0.75%)}	_~	_18.64s	_19.04s	_{p=0.065 n=6}
_{Emit Time}	_0.00s	_0.00s	_~	_~	_~	_{p=1.000 n=6}
_{Total Time}	_{24.77s (± 0.36%)}	_{24.64s (± 0.69%)}	_~	_24.42s	_24.94s	_{p=0.172 n=6}
_{xstate-main - node (v18.15.0, x64)}
_Errors	₅	₅	_~	_~	_~	_{p=1.000 n=6}
_Symbols	_552,233	_552,233	_~	_~	_~	_{p=1.000 n=6}
_Types	_184,971	_184,971	_~	_~	_~	_{p=1.000 n=6}
_{Memory used}	_{492,394k (± 0.04%)}	_{492,235k (± 0.03%)}	_~	_492,009k	_492,369k	_{p=0.066 n=6}
_{Parse Time}	_{3.43s (± 0.65%)}	_{3.43s (± 0.85%)}	_~	_3.40s	_3.47s	_{p=1.000 n=6}
_{Bind Time}	_{1.18s (± 0.54%)}	_{1.19s (± 0.98%)}	_~	_1.18s	_1.21s	_{p=0.070 n=6}
_{Check Time}	_{19.58s (± 0.97%)}	_{19.57s (± 1.04%)}	_~	_19.39s	_19.94s	_{p=0.872 n=6}
_{Emit Time}	_0.00s	_0.00s	_~	_~	_~	_{p=1.000 n=6}
_{Total Time}	_{24.20s (± 0.77%)}	_{24.19s (± 0.83%)}	_~	_24.02s	_24.56s	_{p=1.000 n=6}

System info unknown

Hosts

node (v18.15.0, x64)

Scenarios

Compiler-Unions - node (v18.15.0, x64)
angular-1 - node (v18.15.0, x64)
mui-docs - node (v18.15.0, x64)
self-build-src - node (v18.15.0, x64)
self-build-src-public-api - node (v18.15.0, x64)
self-compiler - node (v18.15.0, x64)
ts-pre-modules - node (v18.15.0, x64)
vscode - node (v18.15.0, x64)
webpack - node (v18.15.0, x64)
xstate-main - node (v18.15.0, x64)

_Benchmark	_Name	_Iterations
_Current	_pr	₆
_Baseline	_baseline	₆

tsserver

Comparison Report - baseline..pr

_Metric	_baseline	_pr	_Delta	_Best	_Worst	_p-value
_{Compiler-UnionsTSServer - node (v18.15.0, x64)}
_{Req 1 - updateOpen}	_{2,821ms (± 1.15%)}	_{2,839ms (± 0.82%)}	_~	_2,799ms	_2,857ms	_{p=0.471 n=6}
_{Req 2 - geterr}	_{7,086ms (±10.93%)}	_{6,613ms (± 9.15%)}	_~	_6,340ms	_7,849ms	_{p=0.128 n=6}
_{Req 3 - references}	_{386ms (± 8.51%)}	_{350ms (± 2.53%)}	_~	_343ms	_367ms	_{p=0.128 n=6}
_{Req 4 - navto}	_{357ms (± 5.27%)}	_{315ms (±13.60%)}	_~	_273ms	_377ms	_{p=0.108 n=6}
_{Req 5 - completionInfo count}	_1,357	_1,357	_~	_~	_~	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{125ms (± 9.16%)}	_{126ms (±12.01%)}	_~	_104ms	_140ms	_{p=0.936 n=6}
_{CompilerTSServer - node (v18.15.0, x64)}
_{Req 1 - updateOpen}	_{3,152ms (± 9.20%)}	_{2,988ms (± 0.50%)}	_~	_2,972ms	_3,009ms	_{p=0.470 n=6}
_{Req 2 - geterr}	_{5,582ms (±10.27%)}	_{5,169ms (±11.64%)}	_~	_4,776ms	_5,968ms	_{p=0.093 n=6}
_{Req 3 - references}	_{425ms (± 9.74%)}	_{433ms (± 7.65%)}	_~	_370ms	_465ms	_{p=1.000 n=6}
_{Req 4 - navto}	_{375ms (± 1.75%)}	_{379ms (± 3.50%)}	_~	_355ms	_390ms	_{p=0.230 n=6}
_{Req 5 - completionInfo count}	_1,519	_1,519	_~	_~	_~	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{117ms (± 9.47%)}	_{116ms (± 7.86%)}	_~	_106ms	_128ms	_{p=0.936 n=6}
_{xstate-main-1-tsserver - node (v18.15.0, x64)}
_{Req 1 - updateOpen}	_{6,507ms (± 0.41%)}	_{6,803ms (± 6.07%)}	_{🔻+296ms (+ 4.55%)}	_6,482ms	_7,345ms	_{p=0.045 n=6}
_{Req 2 - geterr}	_{1,552ms (±11.52%)}	_{1,613ms (±11.06%)}	_~	_1,377ms	_1,757ms	_{p=0.298 n=6}
_{Req 3 - references}	_{120ms (±12.89%)}	_{110ms (± 7.72%)}	_~	_93ms	_115ms	_{p=0.227 n=6}
_{Req 4 - navto}	_{611ms (± 2.70%)}	_{602ms (± 2.61%)}	_~	_582ms	_630ms	_{p=0.226 n=6}
_{Req 5 - completionInfo count}	_3,450	_3,450	_~	_~	_~	_{p=1.000 n=6}
_{Req 5 - completionInfo}	_{1,324ms (± 1.66%)}	_{1,344ms (± 2.03%)}	_~	_1,300ms	_1,366ms	_{p=0.199 n=6}

System info unknown

Hosts

node (v18.15.0, x64)

Scenarios

CompilerTSServer - node (v18.15.0, x64)
Compiler-UnionsTSServer - node (v18.15.0, x64)
xstate-main-1-tsserver - node (v18.15.0, x64)

_Benchmark	_Name	_Iterations
_Current	_pr	₆
_Baseline	_baseline	₆

startup

Comparison Report - baseline..pr

_Metric	_baseline	_pr	_Delta	_Best	_Worst	_p-value
_{tsc-startup - node (v18.15.0, x64)}
_{Execution time}	_{184.78ms (± 0.17%)}	_{184.67ms (± 0.19%)}	_{-0.12ms (- 0.06%)}	_182.44ms	_188.12ms	_{p=0.000 n=600}
_{tsserver-startup - node (v18.15.0, x64)}
_{Execution time}	_{287.76ms (± 0.30%)}	_{287.80ms (± 0.29%)}	_~	_280.42ms	_291.49ms	_{p=0.301 n=600}
_{tsserverlibrary-startup - node (v18.15.0, x64)}
_{Execution time}	_{272.28ms (± 0.30%)}	_{272.22ms (± 0.32%)}	_~	_264.95ms	_284.87ms	_{p=0.092 n=600}
_{typescript-startup - node (v18.15.0, x64)}
_{Execution time}	_{272.79ms (± 0.34%)}	_{272.62ms (± 0.32%)}	_{-0.16ms (- 0.06%)}	_265.00ms	_278.66ms	_{p=0.000 n=600}

System info unknown

Hosts

node (v18.15.0, x64)

Scenarios

tsc-startup - node (v18.15.0, x64)
tsserver-startup - node (v18.15.0, x64)
tsserverlibrary-startup - node (v18.15.0, x64)
typescript-startup - node (v18.15.0, x64)

_Benchmark	_Name	_Iterations
_Current	_pr	₆
_Baseline	_baseline	₆

Developer Information:

Download Benchmarks

andrewbranch and others added 5 commits December 12, 2024 16:20

Reuse config diagnostics between programs with reused structure and i…

2432b01

…dentical config files

Avoid allocation for normalized absolute paths.

45070c4

Remove debug change

88149ba

Indent.

209073e

Merge branch 'avoidNormalizingAndAbsoluting' into combinedUpdateGraph…

3f08187

…Experiments

typescript-bot assigned DanielRosenwasser Dec 17, 2024

typescript-bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels Dec 17, 2024

DanielRosenwasser mentioned this pull request Dec 17, 2024

UpdateProjectIfDirty is slow on large projects #60633

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Combined update graph experiments #60800

Combined update graph experiments #60800

DanielRosenwasser commented Dec 17, 2024

DanielRosenwasser commented Dec 17, 2024

typescript-bot commented Dec 17, 2024 •

edited

Loading

typescript-bot commented Dec 17, 2024 •

edited

Loading

DanielRosenwasser commented Dec 17, 2024

typescript-bot commented Dec 17, 2024 •

edited

Loading

typescript-bot commented Dec 17, 2024

tsc

tsserver

startup

Combined update graph experiments #60800

Are you sure you want to change the base?

Combined update graph experiments #60800

Conversation

DanielRosenwasser commented Dec 17, 2024

DanielRosenwasser commented Dec 17, 2024

typescript-bot commented Dec 17, 2024 • edited Loading

typescript-bot commented Dec 17, 2024 • edited Loading

DanielRosenwasser commented Dec 17, 2024

typescript-bot commented Dec 17, 2024 • edited Loading

typescript-bot commented Dec 17, 2024

tsc

tsserver

startup

typescript-bot commented Dec 17, 2024 •

edited

Loading

typescript-bot commented Dec 17, 2024 •

edited

Loading

typescript-bot commented Dec 17, 2024 •

edited

Loading