Skip to content

RISC-V runner segfault with cache #1139

Description

@tgross35

The run at https://github.com/rust-lang/compiler-builtins/actions/runs/23674147811/job/68973834693?pr=1138 failed with the following:

Image

I don't think it's happened consistently but fyi in case you have any ideas @yuzibo.

Activity

  1. yuzibo commented on Mar 28, 2026

    @yuzibo
    Contributor

    Yeah, this is very odd.

    I do not think this is one real bug for caching compiler-rt on the runner, as we have many jobs already, including in my repo forked, this is the first time to see it. But I do not have evidence either.

    Or was another memory-intensive job running at the same time? I suspect this. BTW, I will do my best to avoid running other jobs on the host.

  2. tgross35 commented on Mar 28, 2026

    @tgross35
    MemberAuthor

    I'll just close this since I don't think there is something actionable. Just something to keep an eye on if it happens again.

    I don't think there is any problem running other jobs. Shouldn't segfault either way of course :)

  3. tgross35 commented on Mar 30, 2026

    @tgross35
    MemberAuthor

    Something similar with a different cache point, this time SIGILL https://github.com/rust-lang/compiler-builtins/actions/runs/23731213183/job/69125229566

    Image
  4. yuzibo commented on Mar 30, 2026

    @yuzibo
    Contributor

    hmm, I will add one new riscv64 hardware as the runner to see if the situation improves.

  5. tgross35 commented on Mar 30, 2026

    @tgross35
    MemberAuthor

    Another one, this time in the post-run cache https://github.com/rust-lang/compiler-builtins/actions/runs/23532708278/job/69228667733

    Image

    Wonder if some cache-related tool is miscompiled for the arch. I'll reopen since this seems to be recurring.

  6. tgross35 commented on Mar 30, 2026

    @tgross35
    MemberAuthor

    It's happening quite often https://github.com/rust-lang/compiler-builtins/actions/runs/23770665652/job/69261142904?pr=1142, https://github.com/rust-lang/compiler-builtins/actions/runs/23767883361/job/69251906199.

    Maybe you could enable core dumps for the runners? To do this I think you can edit /proc/sys/kernel/core_pattern to contain /var/crash/core.%e.%p.%t on the host (can't be done from docker) then add ulimit -c unlimited as one of the first Docker run commands. Then a job to unconditionally upload like the following can work:

        - name: Upload core dumps
          uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
          if: always()
          with:
            name: core-dump
            path: "/var/crash/core.*"

    at least that should tell us what command is actually failing and give us somewhere to look.

  7. yuzibo commented on Mar 31, 2026

    @yuzibo
    Contributor

    I am experimenting with the workaround. It can generate core.* file when core dumped. The system default has systemd-coredump...

    Due to the issue that almost never happened with others' GitHub action including my forked repo or nextest repo with the same action, I may send one PR here when ready.

  8. yuzibo commented on Mar 31, 2026

    @yuzibo
    Contributor

    hmm , from the new one, may I upgrade nodejs to give a try.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions