LLVM 19 and enabling `reference-types` by default

[alexcrichton]

In LLVM 19 the reference-types feature is being enabled by default in llvm/llvm-project#96584. This is having consequences in Rust - rust-lang/rust#128475 - and is requiring documentation to be written - rust-lang/rust#128511 - for how to disable features (docs that should be written anyway, but this update is becoming a strong forcing function).

The main consequence of enabling reference-types is that the table index immediate in the call_indirect instruction is now being encoded as an overlong uleb which is 5 bytes long as 80 80 80 80 00. This overlong encoding of 0 has no semantic difference from when reference-types were disabled but validators and parsers which don't support reference types are rejecting these modules.

I wanted to raise this issue here for awareness to confirm that this is expected fallout. I understand that enabling refernece-types by default is expected, but I wanted to additionally confirm that the consequences of actually emitting a breaking change into all modules using call_indirect relative to parsers that don't support reference-types is expected. This seems like surprising behavior to me where modules that don't use reference-types at all are now required to be executed in runtimes that support the reference-types proposal.

Or, alternatively, if there's an easy-ish way to shrink the leb encoding here (e.g. by assuming there's <= 127 tables and continuing to use a single byte) I think that'd personally be best to generate MVP modules as much as possible and only emit newer features when explicitly requested.

cc @sbc100, @aheejin, @tlively

[sbc100]

Yes we were aware of this slightly strange fallout of enabled reference-types.

For example we had (within Google) reports from folks using wamr that led to bytecodealliance/wasm-micro-runtime#3726.

Its a rather odd side effect of including multi-table as part of reference-types.

Using a single byte relocation for the call_indirect immediate would be a fairly big change since it would be the first relocation that was anything but 5 bytes. Are you sure its worth this effort? I don't think the goal of llvm or wasm-ld should be to produce MVP output unless --cpu=mvp is used.

Isn't this just a one time cost for all tools to update their defaults to match llvm? At least for binaryen and emscripten we try to keep all our defaults in line with llvm's defaults so everyone is on the same page. I imagine each time llvm changes its defaults there will be such one-time costs for all downstream tools, no?

[sbc100]

Also, just in case you were not aware, the linker has --compress-relocations flag that can be used to compress all these 5-byte LEBS.

Also, any optimized build that goes through a tool like wasm-opt will also no longer have these non-canonical LEBs. Just in case that helps anyone who might come accross this issue.

[dschuff]

+cc @sunfishcode
Yeah, generally speaking I'm sympathetic to this... for example when this change landed we had to do an update in our CI because the version of node we were using didn't support retypes out of the box.
I think it's actually worth separating the question of enabling features by default generally from reference types specifically.

For features generally, I don't actually agree that we want to only generate MVP modules unless newer features are requested. Reference types is maybe a special case (since most C/Rust style code doesn't benefit from it) but a better example is nontrapping float-to-int conversion. Enabling this by default will make basically nontrivial program (at least C programs, not sure about Rust) a bit smaller and maybe faster, and there's really no downside to enabling it. Given that many users will not want to go through the list of all features and try to reason out which ones they explicitly enable (and therefore many will just use the default plus whatever is actually necessary) I think we should, as a general practice, enable most features by default once support has reached some threshold. Exactly what the criteria should be (for support levels in the wild, benefits of the feature, etc) could be debatable, but I think the "default" practice should be to enable things unless there's a reason not to.

Reference types specifically could maybe be a special case. It's true that C/Rust style code normally doesn't need it. You could imagine a solution like the one you mentioned; this seems sort of analogous to the small vs large code models supported on some architectures, where there are different size limitations on the resulting code in exchange for more efficient code generation in some cases. 127 tables does seem like enough for most purposes. Probably we just didn't think about this when we made the original design (maybe @pmatos remembers?)

[tlively]

FWIW, I don't think we expected the call_indirect issue before we made the change, but I don't think we would have done much differently if we had anticipated it (except perhaps communicate more broadly ahead of the change). We certainly would have wanted to make the change eventually for the reasons @dschuff gave, and reference types have been standard for long enough now that I don't think waiting longer would have materially changed the outcome.

I'm nervous about the idea of capping the number of supported tables at 127. That's certainly sufficient today, but we've had all kinds of ideas about using separate tables for separate function signatures for various purposes and I wouldn't want to create a new problem for those ideas to run into in the future.

[dschuff]

Also it's possible that we may eventually want to enable exception handling by default for C++ (or even for Rust panics?), and that will require reference types.

[alexcrichton]

That's a good point @dschuff and I should clarify that I do not want to shackle LLVM or anyone else to the wasm MVP, I think it's very good to enable new proposals by default as things move forward. I can perhaps clarify my stance though in that if a selected operation is MVP compatible I think it's beneficial to have it encoded in a way that's MVP compatible. For example a nontrapping float-to-int conversion is not MVP compatible so there's nothing that can be done about that. For call_indirect in this case though there is something that can be done which is to encode the leb differently. Put another way I do think it's a nice property that the "AST of a wasm module" (or fragment) is always encoded in the oldest supported encoding for better compatibility.

Isn't this just a one time cost for all tools to update their defaults to match llvm?

The main exception to this I can think of is build configuration. We support building Wasmtime without a gc feature turned on (a Rust-level Cargo feature, not to be confused with the wasm feature name) where if you disable Wasmtime's gc comiple-time Cargo feature it compiles out support for reference-types, function-references, and gc. This is done to create more minially-sized builds of Wasmtime. Previously that would accept modules from LLVM compiled from C/Rust but that no longer works due to this issue. The reference-types proposal must be enabled to accept these modules which means it must be enabled at compile time, bringing in more runtime dependencies.

Now this itself can be split into a number of different possible solutions:

• One solution is encoding the leb differently (I'll say more on this below)
• One possibility is to "invent" a wasm proposal which is just multi-table and not externref from reference-types. Then Wasmtime in non-gc mode would still reject modules with externref but would support modules from LLVM with overlong lebs in call_indirect
• Wasmtime could be updated in non-gc mode to itself always enable reference-types-the-proposal but reject instances of externref.

None of these seem like the obvious solution per-se, but I also suspect that Wasmtime isn't the only runtime that may want to indefinitely provide a build mode where GC and/or externref isn't supported.

---

For the leb/relocation limits, my rough idea is that a new 1-byte relocation would be added for table indices. I realize that would be quite different from what's implemented today, though. I forgot though that LLD supported --compress-relocations. If that's supported then does that mean LLD could support growing relocations as well? For example could a relocation start as a single byte but if needed LLD could grow it up to 5 bytes? That would solve the issue of not actually limiting to 127 tables while still benefiting the most common use case of there's only a single table.

---

Also it's possible that we may eventually want to enable exception handling by default for C++ (or even for Rust panics?), and that will require reference types.

Good point! (and agreed that Rust will want to enable it by default one day too)

IIRC though the exceptions proposal doesn't require at least the GC/externref part of reference-types, so that sort of also points towards the split I was mentioning above of supporting a mode in Wasmtime of reference-types-but-no-externref or something like that.

[fitzgen]

I agree with the sentiment that this is a specific case, and shouldn't generalize to every time any proposal is enabled by default in LLVM.

I think what makes this case more complicated is that the user's source and LLVM's emitted modules aren't actually leveraging any new features unlocked by the newly-default-enabled proposal. The only results in this particular case are worse code size and less portability.

If LLVM were taking advantage of newly-unlocked features and using externrefs in the emitted module, the calculus would be very different. (For example, enabling C++ exceptions and Rust unwinding by default; but also I'd also expect that if I built a plain C program that didn't use setjmp/longjmp then I wouldn't have anything in the wasm binary that depended on exceptions).

Similarly, I would expect that in the far future when the custom-page-sizes proposal has been finished and merged into the core spec, wasm-ld wouldn't add the binary-encoding equivalent of (pagesize 64KiB) to every module it links together that uses the default page size. I'd expect it to simply rely on the defaults when possible, and only emit the annotation when a non-default page size is used.

[sbc100]

wasm-ld wouldn't add the binary-encoding equivalent of (pagesize 64KiB) to every module it links together that uses the default page size

I agree with this, but in this case it is not the linker that is encoding the call_indirect immediate but the compiler. When reference types are enabled the compiler doesn't know if some other translation unit will add additional tables so it has to do the conservative thing and add a relocation. All the linker does is apply the relocations that the compiler generates.

[sbc100]

I don't think the code size argument applies here since optimized binary should already be running though something like wasm-opt or built with --compress-relocations. Binaries that don't do this are already bloated with non-canonical LEBs.

[sbc100]

I think adding a new single-byte relocation would be easier than trying to do something like compress-relocations but in reverse. --compress-relocations breaks debug info, so it only works on release builds.

The single-byte relocation type might not be so bad. If anyone tries to use more than 127 tables the linker could emit a meaningful error such as "obj.o does not support more than 127 tables. Please recompile with with -msomething`.

We could also use the same default for multi-memory where the number of memory relocations will likely be larger by an order of magnitude and the number of memories likely always very small.

In order words, I'm coming around to the single-byte relocation idea..

[alexcrichton]

Oh I didn't realize --compress-relocations would break debuginfo. That's a strong reason to not switch everything to 1-byte-but-can-expand relocations. I think it would be reasonable to have a -msomething flag for "use a bigger relocation here" and I think that could perhaps be turned on by default for a hypothetical future with a table-per-function-type too.

[sbc100]

I assume the idea would be to backport this change to the LLVM 19 branch?

[alexcrichton]

If it's reasonable enough to do so that'd be great, but if it's large enough and/or risky enough I don't think it's the end of the world to update this in LLVM 20 instead

[tlively]

In general, it would also be a good idea for engines to support encoding changes introduced by new proposals, even if they are configured not to support the new features and semantics of those proposals.

[sbc100]

In general, it would also be a good idea for engines to support encoding changes introduced by new proposals, even if they are configured not to support the new features and semantics of those proposals.

I was wondering about that.. this would be an nice solution, but can you explain why you think its a good idea?

I suppose this would also involve updating the core spec tests to match pending/new proposals in these respects?