FEX-Emu on macOS arm64 | avx(2)

[Rastafabisch]

I couldn't find any information on this topic. This lead me here. Is it possible to use FEX on macOS or would that require additional developing. While Rosetta works incredible great it does not feature AVX(2) support, which is required by more and more game titles.
Especially since D3DMetal works very well (more or less a macOS DXVK & VKD3D equivalent) now and DirectX translation isn't the limiting factor anymore this now starts to be the constraint.

In my imagination there are two scenarios to mitigate this: Either FEX could be a drop-in replacement for Rosetta, or the AVX(2) translation parts could be an addition to Rosetta.

Is either of this feasible?

EDIT:
So far the only use-case, that comes to mind so far is wine. I tried some windows games like Uncharted 4 but they fail complaining about missing instructions.

[Sonicadvance1]

We still don't have plans to support MacOS, but our AVX/AVX2 emulation falls back to 128-bit ASIMD and uses SVE to supplement if available.

The information below is old
FEX only supports Linux. It doesn't have any plans to support MacOS.
Additionally our AVX emulation requires ARM64 hardware that supports SVE2 with 256-bit wide registers, which Apple Silicon doesn't support.

[Sonicadvance1]

Thank you for the prompt answer and technical explanation.

Just one followup: Does that mean, that while FEX works on Asahi Linux (Linux porting effort to Apple Silicon), AVX(2) emulation does not work there either?

FEX doesn't work on Asahi Linux out of the box. Asahi Linux ships a 16KB page size kernel while FEX requires a 4KB page size kernel.
Workarounds include:

• Use the MicroVM approach to run a 4KB kernel in a lightweight VM.
  • Details here: https://sinrega.org/2023-10-06-using-microvms-for-gaming-on-fedora-asahi/
• Recompile the Asahi kernel for 4KB page size, which is unsupported and only quickly put together for testing by Asahi Lina.
  • Not recommended

Apple Silicon doesn't support SVE/SVE2, so FEX's AVX/2 emulation doesn't work on that hardware.

[Spidy123222]

Thank you for the prompt answer and technical explanation.

Just one followup: Does that mean, that while FEX works on Asahi Linux (Linux porting effort to Apple Silicon), AVX(2) emulation does not work there either?

FEX doesn't work on Asahi Linux out of the box. Asahi Linux ships a 16KB page size kernel while FEX requires a 4KB page size kernel.

Workarounds include:

• Use the MicroVM approach to run a 4KB kernel in a lightweight VM.

  • Details here: https://sinrega.org/2023-10-06-using-microvms-for-gaming-on-fedora-asahi/

• Recompile the Asahi kernel for 4KB page size, which is unsupported and only quickly put together for testing by Asahi Lina.

  • Not recommended

Apple Silicon doesn't support SVE/SVE2, so FEX's AVX/2 emulation doesn't work on that hardware.

From what I saw Apple silicon chips for Mac's use armv8.5-a which has sve and sve2. Did Apple remove it some how and it's not an -a variant chip? Also couldn't NEON be used to emulate the avx instructions.

[Sonicadvance1]

No Apple Silicon device today supports SVE or SVE2. This includes Apple M1/M2/M3 and and the latest A17.
There are technical reasons why we have decided that AVX emulation requires SVE2 with 256-bit wide registers. If ARM silicon wants to play with 256-bit AVX then they need to match feature parity.

[Torinde]

Apple M4 reportedly has SVE2/SME, but I can find if it is 128-bit or 256-bit? How many execution units?

Related:

• WIP: Working PR for AVX128 #3720
• Support AVX list #581

Does that mean you plan to support AVX (and other related instructions) on 128-bit SVE2?
Would be interesting do document somewhere what requires which, e.g. F16C requires SVE2 or will run lesser ARM CPUs?

[alyssarosenzweig]

In the past few months, our technical evaluation has changed and so implementing AVX on top of 128-bit is in progress. This will work even on Apple M1 (with ASIMD).

[Rastafabisch]

Just in case someone stumbles on this in the future.
Apple now provides AVX(2) support in macOS Sequoia (beta). Remains to be seen, whether it will be back ported to older macOS releases. At the end of this reddit post you can find some preliminary benchmarks.

[shalaga44]

Late to this, but I measured the macOS side of it recently and wrote up the results, in case it saves anyone the work: https://github.com/shalaga44/rosetta-4k-pages

The short version is that the 4 KiB page problem has a harder form on macOS than it does on Asahi. There's no 4K kernel to boot, and you can't get 4 KiB pages in a native ARM64 process at all. On Apple Silicon a process gets 4 KiB pages only if it's an x86-64 binary running under Rosetta, and such a process can't execute ARM64 instructions, even ones written into MAP_JIT memory with dyld bypassed entirely.

Both properties are selected by the same field: the Mach-O cputype checked at exec. I traced it in the kernel, _vm_map_page_shift reads a byte at vm_map + 0x2C (0xC = 4K, 0xE = 16K), and the exec path compares the image's cputype against CPU_TYPE_X86_64 right before the "does not satisfy requirements for 4k page size" error string.

Four ways around it, all closed:

attempt	result
Forge cputype in the Mach-O header	Routes correctly, crash report says "translated": true, then SIGILL on the ARM64 code
Clear P_TRANSLATED after exec	_proc_is_translated has zero callers in the kernel, it's a report, not a switch
dlopen an ARM64 dylib from a translated process	Rejected by dyld policy
mmap + raw call, no loader at all	SIGILL, with the correct bytes verified in memory

So it's hardware execution state, not a software gate. The repo has five runnable proofs (~200 lines of C, no root, no SIP changes) so anyone can check this on their own machine rather than take my word for it.

Two side findings that might be more immediately useful here:

CPUID under-reports AVX under Rosetta. Leaf 1 says AVX = 0 and leaf 7 says AVX2 = 0, while vaddps, vpaddd, vpermd and vperm2i128 all execute and give correct results. Anything gating on feature bits under a translator will reach a confidently wrong conclusion.

Confirming the SVE point from upthread, by execution rather than by spec: ptrue p0.b, add z0.s and sqrdmlah z0.s all SIGILL on an M1 Pro, with a NEON add v0.4s control passing in the same harness. There's no hw.optional.arm.FEAT_SVE sysctl despite 70 other FEAT_ OIDs existing. SVE/SVE2 are optional in ARMv8.x, so ARMv8.5-A compliance doesn't imply them, which is the answer to the question asked upthread. Rosetta's own translator contains 0 sve* encoding classes against 10 asimd* ones, so Apple reached the same conclusion.

Tested on macOS 27.0 (26A428), M1 Pro. Undocumented internal behaviour on one machine and one OS build, so corrections welcome.