v2.321.0 linux-x64 Runner.Listener fails with segfault when emulated on s390x system #3583
Comments
|
CC: @chantra This issue puts Kernel BPF CI in an unfortunate position:
Best case for us: this is just a bug in the application code and it's fixed. I realize that s390x is not an officially supported platform at the moment, although people did request it: #2263 And there have been attempts to add the support for s390x, but they all seem to be out of date:
.NET is officially supported by Canonical for Ubuntu s390x, and .NET platform is cross-arch. So official s390x runner binaries seem at least feasible. It would be great to hear from the Github team what is their stance on s390x support. Are there any plans to implement it? A clear "no" would also be helpful, if that is the answer. Thank you! |
|
FYI You have still ca. 20 days left with the last .net6.0 runner v2.230. The current agent switched to .net8, by v.2.231.0, because of end of live of .net6.0. I do my best to keep my unofficial custom actions runner compatible that has native binaries out of box (aka https://github.com/christopherhx/github-act-runner), but in December it will possible have an outage due to github actions service updates that alters the protocol until I'm affected myself |
|
Did you try emulating the arm64 binary? I mean it's a .net8 framework issue not application code |
Hi. Thank you for suggestion. I've tried arm64 binary, and it does not crash with segfault. So now we at least have a workaround. Still, would be great to get a comment on the issue from the GitHub team. |
Github Actions Runner.Listener x86_64 binary version 2.321.0 has been crashing with segfault on startup when emulated on s390x [1]. Ilya pointed out, that the root cause of this crash is most likely a qemu issue [2]. .NET 8 turns on "write xor execute" feature [3] by default. And GH actions runner 2.320.0->2.321.0 update includes an upgrade to .NET 8 [4]. Ilya suggested opting out of this feature in our environment to mitigate the failure. I tested this with manually built runner immage on one for s390x BPF CI hosts: the runner doesn't crash. [1] actions/runner#3583 [2] https://gitlab.com/qemu-project/qemu/-/issues/2600 [3] https://devblogs.microsoft.com/dotnet/announcing-net-6-preview-7/#runtime:-w^x-(write-xor-execute)-support-for-all-platforms-and-architectures [4] actions/runner#3500 Suggested-by: Ilya Leoshkevich <[email protected]> Signed-off-by: Ihor Solodrai <[email protected]>
Github Actions Runner.Listener x86_64 binary version 2.321.0 has been crashing with segfault on startup when emulated on s390x [1]. Ilya pointed out, that the root cause of this crash is most likely a qemu issue [2]. .NET 8 turns on "write xor execute" feature [3] by default. And GH actions runner 2.320.0->2.321.0 update includes an upgrade to .NET 8 [4]. Ilya suggested opting out of this feature in our environment to mitigate the failure. I tested this with manually built runner immage on one for s390x BPF CI hosts: the runner doesn't crash. [1] actions/runner#3583 [2] https://gitlab.com/qemu-project/qemu/-/issues/2600 [3] https://devblogs.microsoft.com/dotnet/announcing-net-6-preview-7/#runtime:-w^x-(write-xor-execute)-support-for-all-platforms-and-architectures [4] actions/runner#3500 Suggested-by: Ilya Leoshkevich <[email protected]> Signed-off-by: Ihor Solodrai <[email protected]>
|
For anyone interested, the root cause seems to be a qemu issue, tiggered by .NET 8 upgrade: https://gitlab.com/qemu-project/qemu/-/issues/2600 Details about this particular failure here: kernel-patches/runner#61 |
Describe the bug
v2.321.0 linux-x64 Runner.Listener fails with segfault on startup when running emulated on s390x system.
To Reproduce
The docker image is
ghcr.io/kernel-patches/runner:main-noble-s390x@sha256:0e7347243a42f87e06520ab71dbe44250a7e59ba45f569b16c9cf001d2f7f236Dockerfile
We get this failure on a Linux s390x host with binfmt configured to execute x86_64 binaries via qemu.
Expected behavior
The setup with binfmt has been used successfully for quite a while by Linux Kernel BPF CI.
Revert to v2.320.0 mitigates the issue, which suggests that the cause was introduced recently.
Runner Version and Platform
What's not working?
The runner fails to start.
Job Log Output
Runner and Worker's Diagnostic Logs
Unfortunately these are lost, because unhealthy runners are automatically restarted.
The text was updated successfully, but these errors were encountered: