Eliminate createRequire/require from EXPORT_ES6 output#26384
Eliminate createRequire/require from EXPORT_ES6 output#26384vogel76 wants to merge 23 commits intoemscripten-core:mainfrom
createRequire/require from EXPORT_ES6 output#26384Conversation
sbc100
left a comment
There was a problem hiding this comment.
Wow, thanks for working on this! Nice work.
| Module['wasm'] = fs.readFileSync(new URL('{{{ TARGET_BASENAME }}}.wasm', import.meta.url)); | ||
| #endif | ||
| #endif | ||
| #else |
There was a problem hiding this comment.
This seems like a lot of extra complexity/duplication. Is there some way to avoid it maybe?
createRequire/require from EXPORT_ES6 output
0ec991e to
6b74d2a
Compare
|
@sbc100 I hope your remarks have been addressed in added fixup commits. Once you accept it, I will do autosquash rebase to leave only actual commits in clean history. |
6b74d2a to
8f585b7
Compare
|
Is it possible to write a test for this? When you say "This breaks bundlers", do you know why our current bundler tests in test_browser.py are working? (see |
|
Re-commiting, we always squash all changes in the emscripten repo. If you think this change can be split into to commits then please send a two separate PRs. This is just how we do things in emscripten. It helps keep the tests passing on every commit (for the benefit of bisectors). |
| var fs = require('node:fs'); | ||
| var fs = {{{ makeNodeImport('node:fs') }}}; | ||
| #if WASM == 2 | ||
| if (globalThis.WebAssembly) Module['wasm'] = fs.readFileSync(__dirname + '/{{{ TARGET_BASENAME }}}.wasm'); |
There was a problem hiding this comment.
Does this code (which uses __dirname) simply not work with EXPORT_EST today?
If not, this seems like maybe a separate fix that we could land in isolation. e.g. Fix for EXPORT_ES6 + MINIMAL_RUNTIME + ???
There was a problem hiding this comment.
The __dirname usage in shell_minimal.js indeed doesn't work with EXPORT_ES6 today — it's the same class of issue as require() not being available in ESM.
The fix uses makeNodeFilePath() which switches between:
- ESM:
new URL('file.wasm', import.meta.url) - CJS:
__dirname + '/file.wasm'
I kept it in this PR because the changes are on the same lines as the require() → makeNodeImport() replacements — splitting them would create merge conflicts between the two PRs for no benefit. But happy to separate if you prefer.
There was a problem hiding this comment.
If shell_minimal.js already doesn't work it seems fine to leave it out of this change. shell_minimal.js is not ever used by default, and compatibility with all settings is not important here IMHO.
Sure - will add such tests. The sake of such work are maintenance problems of our Hive blockchain interfacing library: wax which compiles blockchain protocol C++ code into wasm and uses it at TS/JS level. We have set of tests related to bundlers and different frameworks where we got problems while loading wasm etc. https://gitlab.syncad.com/hive/wax/-/tree/develop/examples/ts?ref_type=heads |
8f585b7 to
6c822be
Compare
Deeper analysis gave conclusions why Existing tests don't catch the require() problem. The real breakage happens when:
I extended test to cover more usecases altough adding tests covering whole frameworks seems to heavy for your project. Our wax library covers that... |
I'm curious about this case. I'd love to add a test to Are you saying that rollup doesn't actually break (since it marks node builtins as external by default)? If so, is rollup just not effected by this bug? Is vite effected? Could we add a test to test_other.py alongside |
Here are new tests targeted to main where they are expected to fail: They shall pass here ofc. Once you accept it, I will cherry-pick them to this PR |
|
Those tests look great thanks! |
2270aac to
d068717
Compare
sbc100
left a comment
There was a problem hiding this comment.
(sorry I forgot the send my pending comment)
d068717 to
8b2fb1c
Compare
8b2fb1c to
23863c1
Compare
|
Changes in this update:
Known remaining issues:
|
30c1f96 to
14fdc3b
Compare
|
@sbc100 could you take a look for test-bun and test-deno failures? Probably I will need some help in getting local env. to reproduce problems. Please point me some doc or scripts how to setup it. Best if you have some docker image to run them in... |
4528d9a to
c4a3967
Compare
| #if ENVIRONMENT_MAY_BE_NODE && EXPORT_ES6 | ||
| // In ESM mode, require() is not natively available. When SOCKFS is used, | ||
| // we need require() to lazily load the 'ws' npm package for WebSocket | ||
| // support on Node.js. Set up a createRequire-based polyfill. |
There was a problem hiding this comment.
Why do we still need createRequire in this case?
There was a problem hiding this comment.
sync context requires createRequire, can't use await import()
There was a problem hiding this comment.
Can you explain more here? How/why does libsocksf require "sync context" whereas other places in the codebase don't?
Do we need to update how we use the ws package maybe?
There was a problem hiding this comment.
The sync context at connect()/listen() is real (sync syscall callbacks across the JS↔Wasm boundary — no await without ASYNCIFY). More importantly, ws is a user-installed peer-dep: many programs link libsockets transitively without ever calling WebSocket ops, so loading ws at init would regress those programs vs. the CJS build. createRequire gives us a sync require() scoped to import.meta.url; the inner await import('node:module') is a cheap Node builtin that can't fail. The actual ws load stays lazy — same behavior as the CJS build. A bigger refactor (async-friendly WS library, or asyncify the socket syscall layer) could remove createRequire, but that's out of scope here.
You can analyze failures by change addressing your request:
2642a5d
904f790 to
c1332e0
Compare
When EXPORT_ES6 is enabled, the generated JS used createRequire() to
polyfill require(), which breaks bundlers (webpack, Rollup, esbuild)
and Electron's renderer process. Since EXPORT_ES6 requires MODULARIZE,
the module body is wrapped in an async function where await is valid.
- shell.js: Remove createRequire block entirely. Use await import()
for worker_threads, fs, path, url, util. Replace __dirname with
import.meta.url for path resolution.
- shell_minimal.js: Same pattern for worker_threads and fs. Replace
__dirname with new URL(..., import.meta.url) for wasm file loading.
- runtime_debug.js: Skip local require() for fs/util when EXPORT_ES6,
reuse outer-scope variables from shell.js instead.
- runtime_common.js: Guard perf_hooks require() with EXPORT_ES6
alternative.
- preamble.js: Hoist await import('node:v8') above instantiateSync()
for NODE_CODE_CACHING since await can't be used inside sync functions.
Library functions run in synchronous context where await is unavailable.
Define top-level library symbols that use await import() at module init
time, then reference them via __deps from synchronous functions.
- Add libnode_imports.js with shared $nodeOs symbol, register in
modules.mjs when EXPORT_ES6 is enabled.
- libatomic.js, libwasm_worker.js: Use $nodeOs for os.cpus().length
instead of require('node:os').
- libwasi.js: Define $nodeCrypto for crypto.randomFillSync in
$initRandomFill. Combine conditional __deps to avoid override.
- libcore.js: Define $nodeChildProcess for _emscripten_system.
- libnodepath.js: Switch $nodePath initializer to await import().
- libsockfs.js: Define $nodeWs ((await import('ws')).default) for
WebSocket constructor in connect() and Server in listen().
Bundlers (webpack, rollup, vite, esbuild) and frameworks (Next.js, Nuxt) cannot resolve CommonJS require() calls inside ES modules. This test statically verifies that EXPORT_ES6 output uses `await import()` instead of `require()` for Node.js built-in modules, and that the `createRequire` polyfill pattern is not present. Parameterized for default, node-only, and pthreads configurations to cover the various code paths that import Node.js built-ins (fs, path, url, util, worker_threads).
…ire() Add two tests that verify EXPORT_ES6 output is valid ESM and works with bundlers: - test_webpack_esm_output_clean: Compiles with EXPORT_ES6 and default environment (web+node), then builds with webpack. On main, webpack hard-fails because it cannot resolve 'node:module' (used by emscripten's createRequire polyfill). This breaks any webpack/Next.js/Nuxt project. - test_vite_esm_output_clean: Compiles with EXPORT_ES6 and default environment, then builds with vite. On main, vite externalizes 'node:module' for browser compatibility, emitting a warning. The resulting bundle contains code referencing unavailable node modules. These tests are expected to fail on main and pass after eliminating require() from EXPORT_ES6 output.
The previous fix moved await import() for node:fs and node:util outside dbg() to module scope. This broke test_dbg because dbg() can be called from --pre-js before those module-scope imports execute. Use lazy initialization instead: declare dbg_node_fs/dbg_node_utils early but leave them undefined. Initialize them in shell.js after fs and utils are loaded (reusing the same imports). dbg() checks if the modules are available and falls back to console.warn if not. This handles all cases: - dbg() from --pre-js (before init): uses console.warn - dbg() after init on Node.js with pthreads: uses fs.writeSync - dbg() in browser/non-node: uses console.warn
In ESM mode (WASM_ESM_INTEGRATION), the runtime uses dynamic import()
instead of require() for node modules. Update the test_environment
assertion to check for 'import(' when in ESM mode, rather than always
expecting 'require('.
The test was using require('path') which doesn't work in ESM mode.
Since ESM output (.mjs) wraps the module in an async context, we can
use top-level await with dynamic import.
Changed from:
require('path')['isAbsolute'](scriptDirectory)
To:
var nodePath = await import('node:path');
nodePath.isAbsolute(scriptDirectory)
This properly tests the Node.js path.isAbsolute() function while being
compatible with ESM module format. The CJS variant (test_locate_file_abspath)
continues to use require() as appropriate for CommonJS.
Closure compiler runs before the MODULARIZE async wrapper is applied,
so it sees `await import('node:xyz')` outside an async function and
fails to parse it.
Work around this by:
1. Before Closure: Replace `await import('node:xyz')` with placeholder
variables like `__EMSCRIPTEN_PRIVATE_AWAIT_IMPORT_xyz__`
2. Generate externs for the placeholders so Closure doesn't error on
undeclared variables
3. After Closure: Restore placeholders to `(await import('node:xyz'))`
with parentheses to handle cases where Closure inlines the variable
into expressions like `placeholder.method()`
This follows the same pattern as the existing
`__EMSCRIPTEN_PRIVATE_MODULE_EXPORT_NAME_SUBSTITUTION__` mechanism.
Instead of skipping EM_ASM tests that use CJS require() in ESM modes, add a createRequire-based polyfill (available since Node 12.2.0) that makes require() available in ESM output. The polyfill is only included when the build targets ESM (EXPORT_ES6, MODULARIZE=instance, or WASM_ESM_INTEGRATION). - Add test/require_polyfill.js using createRequire from 'module' - Add is_esm() and add_require_polyfill() helpers to test/common.py - Remove @no_modularize_instance skips from test_fs_nodefs_rw and test_fs_nodefs_home, enabling them in ESM test modes
When EXPORT_ES6 is enabled, makeNodeImport() generates top-level await import() expressions. Babel's transpile step was configured with sourceType 'script', which cannot parse top-level await. This caused test_node_prefix_transpile to fail at compilation. Use sourceType 'module' when EXPORT_ES6 is active so Babel can correctly parse and transpile the ESM output.
runtime_debug.js references dbg_node_fs and dbg_node_utils when ENVIRONMENT_MAY_BE_NODE with PTHREADS or WASM_WORKERS, but these variables were only declared in shell.js, not in shell_minimal.js. This caused Closure compiler to fail with JSC_UNDEFINED_VARIABLE for MINIMAL_RUNTIME builds with threading and --closure=1. Add the declarations for both paths: - PTHREADS: inside the existing if(ENVIRONMENT_IS_NODE) block that already imports fs - WASM_WORKERS without PTHREADS: in a new conditional block
NODERAWFS postset used require('node:tty') directly, which breaks
in ESM output (.mjs) where require is not available. Extract it into
a $nodeTTY library symbol using makeNodeImport, matching the pattern
used by $nodePath and other node module imports. This fixes all
_rawfs test failures in test-esm-integration and test-modularize-instance.
1236377 to
23d0a46
Compare
…terns conflict
The name $child_process collides with Closure compiler's node externs
(third_party/closure-compiler/node-externs/child_process.js) which declares
`var child_process = {};`, causing JSC_VAR_MULTIPLY_DECLARED_ERROR when
INCLUDE_FULL_LIBRARY is used with --closure=1.
|
@sbc100 what prevents this PR from merge? I suppose all your remarks have been addressed and I fixed them to match your needs or explained used solution. I'd like to have this problem solved soon as our software can be also simplified after this fix. |
Thanks! I do want to land this PR, but there might be a couple more things to address. For example, I'd like to look into #23730 and see if landing that first makes sense. I'll give this PR another read over now.
Thanks. We always squash commit all PRs anyway when land them. |
| } | ||
|
|
||
| function makeNodeImport(module, guard = true) { | ||
| assert(ENVIRONMENT_MAY_BE_NODE, 'makeNodeImport called when environment can never be node'); |
There was a problem hiding this comment.
Should we have this assert in makeNodeFilePath too maybe?
| # MODULARIZE=instance implies EXPORT_ES6 which triggers ESM output. | ||
| is_esm = self.get_setting('EXPORT_ES6') or self.get_setting('WASM_ESM_INTEGRATION') or self.get_setting('MODULARIZE') == 'instance' | ||
| if is_esm: | ||
| has_node_imports = 'import(' in js |
There was a problem hiding this comment.
This seems like its testing for more than just node imports, but any ES6 import (which might not be a node import I guess.. even though we don't use any today).
How about this:
if 'node' not self.get_setting('ENVIRONMENT'):
has_external_imports = 'require(' in js or 'import(' in js
self.assertFalse(has_external_import, 'we should not have external imports unless targeting node')
fixup! Add Closure compiler support for ESM dynamic imports Route makeNodeImport through the existing EMSCRIPTEN$AWAIT$IMPORT placeholder mechanism (added by parseTools.preprocess() and reversed by fix_js_mangling()) instead of running a separate regex-based pre/post-Closure pass over the entire JS output. The placeholder identifier is already in src/closure-externs/closure-externs.js, so no extra externs need to be generated. Addresses review comment: emscripten-core#26384 (comment)
fixup! Add require() polyfill for EM_ASM tests in ESM modes
Replace the require() polyfill (used to keep EM_ASM blocks calling
require('fs')/require('path') working in ESM mode) with direct
references to library symbols via EM_JS_DEPS, as suggested by the
maintainer.
- Add new $nodeFs library symbol in libcore.js (alongside $nodeOs,
$nodeChildProcess) so EM_ASM can reference 'nodeFs' directly.
- Update test_nodefs_home.c and test_nodefs_rw.c to declare their
dependency via EM_JS_DEPS(deps, ...) and use nodePath/nodeFs in
place of require().
- Link -lnodepath.js into test_fs_nodefs_home so $nodePath is
resolvable (libnodepath.js is otherwise only auto-included for
NODERAWFS).
- Drop test/require_polyfill.js and the add_require_polyfill helper.
- Reuse the existing is_esm() helper in test_no_node_imports.
Addresses review comment:
emscripten-core#26384 (comment)
…iles fixup! Replace require() with library symbols in EXPORT_ES6 library files Drop the createRequire-based \$nodeRequire polyfill in libsockfs.js and load the 'ws' package directly with await import() at module-init time, as the commit message of c85c69e originally described. Reviewer question (r3075800765): "How/why does libsockfs require 'sync context' whereas other places in the codebase don't? Do we need to update how we use the 'ws' package maybe?" Answer: the call sites at lines 230 (createPeer) and 540 (listen) run as synchronous callbacks from socket syscalls across the JS<->Wasm boundary, so 'await import()' cannot be used at those sites. The createRequire polyfill sidesteps that by exposing a synchronous require() inside an ESM module. But synchronous use is only needed at the call site, not at the point of loading. We can resolve the module asynchronously once at module-init time (where top-level await is available in the MODULARIZE=instance wrapper) and cache it, then reference the cached value synchronously later. 'ws' is a CJS package so the WebSocket class is on the .default export; we bind that to \$nodeWs. This removes the only remaining use of createRequire() and import('node:module') from the PR, and uses the same pattern as \$nodeFs / \$nodePath / \$nodeOs for every other node import. Trade-off: if SOCKFS is linked but the 'ws' package is not installed, the failure now surfaces at module init instead of at first socket use. In practice SOCKFS implies a hard dependency on 'ws', so failing fast is preferable.
…iles
fixup! Replace require() with library symbols in EXPORT_ES6 library files
Revert the eager $nodeWs((await import('ws')).default) approach from the
prior fixup (2642a5d) and restore the createRequire-based $nodeRequire
polyfill.
Why reverting: the eager approach resolved 'ws' at module-init time,
which breaks any program that links SOCKFS but never calls socket ops.
The test runner hit this on instance.test_dlfcn_handle_alloc and 18
other tests that pull in SOCKFS transitively:
Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'ws' imported
from /tmp/emtest_.../main.mjs
The createRequire pattern preserves the original CJS-build semantics:
require() is wrapped but not invoked until the first socket connect()
or listen() call, at which point the user has opted in to actually
using WebSockets.
Proper reply to the review question (r3075800765) -- "do we need to
update how we use the 'ws' package?":
The sync context in libsockfs is real -- connect()/listen() are
synchronous callbacks invoked from socket syscalls across the
JS<->Wasm boundary. We cannot 'await import' at those sites
without ASYNCIFY, which is out of scope for this PR.
And we cannot pre-load 'ws' at module init either, because 'ws'
is a user-installed npm peer-dep; many emscripten programs link
libsockets (it is pulled in by socket()/dlfcn/etc.) without ever
calling WebSocket ops. Failing the whole program at init just
because 'ws' is not installed would be a regression over the CJS
build, which only fails if WebSocket ops are actually invoked.
createRequire is the minimal viable primitive for this: it gives
us a synchronous require() function in an ESM module, scoped via
import.meta.url. The 'await import(node:module)' that produces
it is on a Node builtin, so it is cheap and cannot fail. The
actual ws load stays lazy at the call site, matching CJS behavior.
A larger refactor that replaces 'ws' with an async-friendly
WebSocket library (or that asyncifies the socket syscall layer)
could remove createRequire, but that is a separate change and
out of scope here.
…ime files
fixup! Replace require() with await import() in EXPORT_ES6 shell/runtime files
Drop the per-call `/* webpackIgnore: true */ /* @vite-ignore */`
directives from makeNodeImport's generated `await import(...)` expression.
Reviewer question (r3075818347):
"I won't love having this vender/bundler-specific comments in the code
like this. Didn't you say that these bundlers should automatically
ignore things with the 'node:' prefix?"
Modern webpack 5, vite, and rollup auto-externalize `node:`-prefixed
dynamic imports for browser targets. The hints only suppress a
"externalized for browser compatibility" warning, which
test/vite/vite.config.js already whitelists explicitly for node:*
modules. Correctness is unchanged whether the hints are present or not.
Verified locally:
- other.test_esm_no_require (3 variants) passes.
- emcc -sEXPORT_ES6 -sMODULARIZE produces clean
`await import('node:fs')` / `node:path` / `node:url` calls with
no bundler-specific comment blocks in the output.
…ime files fixup! Replace require() with await import() in EXPORT_ES6 shell/runtime files Revert the bundler-hint removal from 548e32d. It was based on a mistaken assumption that webpack auto-handles `node:`-prefixed dynamic imports -- it does not. CI run on 548e32d hit: test_webpack_esm_output_clean: ERROR in node:fs Module build failed: UnhandledSchemeError: Reading from "node:fs" is not handled by plugins (Unhandled scheme). Webpack supports "data:" and "file:" URIs by default. You may need an additional plugin to handle "node:" URIs. Same error for node:path, node:url. All three show up because without `/* webpackIgnore: true */`, webpack attempts to resolve the dynamic import as a module and fails on the unknown scheme. Vite (per test/vite/vite.config.js) does auto-externalize `node:*` for browser -- emitting a warning that the config already whitelists -- so strictly speaking `/* @vite-ignore */` is optional. But we keep both for symmetry and to also silence vite's dynamic-import analysis noise. Updates the inline comment to explain the requirement so a future reader does not attempt the same simplification again. Addresses and supersedes the response to r3075818347 -- the hints cannot be removed unless we add a webpack plugin or drop support for webpack consumers of EXPORT_ES6 output.
When building with
-sEXPORT_ES6, the generated JavaScript currently usescreateRequire(import.meta.url)to polyfillrequire(), then callsrequire()for Node.js built-in modules. This breaks bundlers (webpack, Rollup, esbuild, Vite) and Electron's renderer process, forcing users to post-process emscripten output withsedor custom plugins to strip out the offending calls.This PR replaces all
require()calls withawait import()whenEXPORT_ES6is enabled, eliminating the need forcreateRequireentirely.Why this is safe
EXPORT_ES6requiresMODULARIZE(enforced intools/link.py).MODULARIZEwraps the module body in anasync function, soawaitis valid at the top level of the generated code.await import(...)is transformed to anEMSCRIPTEN$AWAIT$IMPORTplaceholder during preprocessing (parseTools.mjs:83) and restored during linking (link.py:2136). This mechanism is already used elsewhere in the codebase.(await import('node:fs')).readFileSyncworks identically torequire('node:fs').readFileSync. For CJS npm packages likews,.defaultgives themodule.exportsvalue.#if EXPORT_ES6/#elsepreprocessor conditionals. The existingrequire()code path is untouched.Approach
The changes split naturally into two categories:
Shell and runtime files (top-level async context where
awaitworks directly):src/shell.js— Remove thecreateRequireblock. Useawait import()forworker_threads,fs,path,url,util.src/shell_minimal.js— Same pattern forworker_threadsandfs. Replace__dirnamewithnew URL(..., import.meta.url)for wasm file loading.src/runtime_debug.js— Skip localrequire()forfs/utilwhenEXPORT_ES6; reuse outer-scope variables fromshell.js.src/runtime_common.js— Guardperf_hooksrequire()withEXPORT_ES6alternative.src/preamble.js— Hoistawait import('node:v8')aboveinstantiateSync()forNODE_CODE_CACHING(can't useawaitinside a sync function).Library files (synchronous function bodies where
awaitis unavailable):Library functions execute in synchronous context, so
await import()cannot be called inline. Instead, we define library symbols initialized at module top level (async context) and reference them via__deps:$nodeOsnode:oslibatomic.js,libwasm_worker.js$nodeCryptonode:cryptolibwasi.js($initRandomFill)$nodeChildProcessnode:child_processlibcore.js(_emscripten_system)$nodeWsws(.default)libsockfs.js(WebSocket connect + listen)$nodePathnode:pathlibnodepath.js(NODERAWFS)Shared symbols live in a new
src/lib/libnode_imports.js, registered insrc/modules.mjswhenEXPORT_ES6is enabled.Intentionally skipped
src/lib/libembind_gen.js— Build-time code running in Node.js directly, not in emitted runtime JS.src/closure-externs/closure-externs.js—createRequireextern kept; still needed for the non-ES6 path.What changes in the output
Before (EXPORT_ES6):
After (EXPORT_ES6):
The non-ES6 path remains unchanged:
Files modified
13 files modified, 1 new file — 123 insertions, 12 deletions (source files only).
Testing
All existing ESM tests pass:
test_esm(default, node, pthreads variants)test_esm_worker(default, node variants)test_esm_worker_single_filetest_esm_closuretest_esm_implies_modularizetest_esm_requires_modularizetest_pthread_export_es6(default, trusted variants)Additionally verified:
emcc -sEXPORT_ES6 -sMODULARIZEoutput contains zerocreateRequireorrequire(occurrencesemcc -sEXPORT_ES6 -sMODULARIZE -pthreadoutput contains zerocreateRequireorrequire(occurrencesrequire()as before (no regression)Related issues
EXPORT_ES6output containsrequire()which breaks bundlersEXPORT_ES6use only ES6 semantics? #20503 —createRequirebreaks Electron rendererrequire()require()in ESM output breaks RollupRelated PRs
nodebuilt-in module imports withnode:. #18235 — Earlier attempt to address this (partial)