Canonical test ordering - lexicographic

[RenechCDDA]

Summary

Infrastructure "Tests now run in a guaranteed order (lexicographic) by default (can be overriden with cli argument)"

Purpose of change

• Fix vehicle_turret test #78567 (comment) exposed some good questions. What is the ordering in how our tests are run? Apparently it's partly determined by linking order, which makes reproducing some test fails impossible without using the same link order (e.g. being on the same commit and using the same build method).

This is an unnecessary impediment when we're trying to diagnose test failures.

Describe the solution

Run the tests in lexicographical order instead.

Note that this works across subsets of tests. As an example I used the tests from #78524 which are DECLARED in the order
NPC rules (avoid doors)
NPC rules (close doors)
NPC rules (avoid locks)

By default that is the order they are run in (declared order). Running the tests with the arguments of "[npc_rules]" --wait-for-keypress exit --order lex instead gets us the following order:
NPC rules (avoid doors)
NPC rules (avoid locks)
NPC rules (close doors)

So the desired order(lexicographical) remains consistent even when additional arguments are provided, or subsets of tests are run.

Describe alternatives you've considered

We could run in the provided rand order instead. I initially thought this would be useful if we needed to specify subsets of tests to run, but lexicographical has us covered there already.

Since the random order was made subset stable, we promise that given the same random seed, the order of test cases will be the same across different platforms, as long as the tests were compiled against identical version of Catch2. We reserve the right to change the relative order of tests cases between Catch2 versions, but it is unlikely to happen often.

I thought random order might also be useful if we needed to shuffle the order of test cases for some reason, but the more I thought about it I failed to find any 'some reason'.

Testing

Works on my machine

Would like to see github CI all green before this is considered mergeable

Additional context

There is a second, mostly-unrelated commit on this branch, intentionally.

It enables --wait-for-keypress exit on MSVC builds, because it is really annoying that the default is to close the tests after being run.

This should only be the case when a human being manually runs the test (and only when using MSVC). It should NOT run when github tests does its auto magic, otherwise they'll get stuck waiting for a keypress that never comes.

If it turns out to be problematic... well that's why it's in a separate commit, we can just get rid of it.

[PatrikLundell]

The only reason I can see to run tests in a random order would be to find poorly designed tests that rely on conditions they aren't setting up properly, but then we'd run into the problem of tracking the cause down when it happens. I'd say it's better not to detect such errors or encounter them deterministically, especially since the tests aren't intended to test that the tests are good, but that the tested code does what it's intended to do.

[RenechCDDA]

The only reason I can see to run tests in a random order would be to find poorly designed tests that rely on conditions they aren't setting up properly, but then we'd run into the problem of tracking the cause down when it happens. I'd say it's better not to detect such errors or encounter them deterministically, especially since the tests aren't intended to test that the tests are good, but that the tested code does what it's intended to do.

Agree on all points! :)

[RenechCDDA]

...Oh goodness it looks like we're gonna have some fixing to do

[mqrause]

There is a second, mostly-unrelated commit on this branch, intentionally.

It enables --wait-for-keypress exit on MSVC builds, because it is really annoying that the default is to close the tests after being run.

I'm a bit confused by this. I don't think that does anything when you put it there? Plus it doesn't just close for me. I don't think it ever did and I don't remember ever changing some setting. Though it's been too long since I used VS2019 honestly.