Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 6a01d39

Browse filesBrowse files
mhdawsonBethGriggs
mhdawson
authored and
BethGriggs
committed
src: add support for externally shared js builtins
Refs: #44000 - add infra to support externally shared js builtins in support of distos that want to externalize deps that include JS/WASM instead of native code - add support for externalizing - cjs_module_lexer/lexer - cjs_module_lexer/dist/lexer - undici/undici Signed-off-by: Michael Dawson <mdawson@devrus.com> PR-URL: #44376 Backport-PR-URL: #45867 Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
1 parent 962a747 commit 6a01d39
Copy full SHA for 6a01d39

File tree

Expand file treeCollapse file tree

6 files changed

+229
-3
lines changed
Open diff view settings
Filter options
Expand file treeCollapse file tree

6 files changed

+229
-3
lines changed
Open diff view settings
Collapse file

‎BUILDING.md‎

Copy file name to clipboardExpand all lines: BUILDING.md
+38Lines changed: 38 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -1062,6 +1062,44 @@ To make `./myModule.js` available via `require('myModule')` and
10621062
> .\vcbuild link-module './myModule.js' link-module './myModule2.js'
10631063
```
10641064

1065+
## Building to use shared dependencies at runtime
1066+
1067+
By default Node.js is built so that all dependencies are bundled into
1068+
the Node.js binary itself. This provides a single binary that includes
1069+
the correct versions of all dependencies on which it depends.
1070+
1071+
Some Node.js distributions, however, prefer to manage dependencies.
1072+
A number of `configure` options are provided to support this use case.
1073+
1074+
* For dependencies with native code, the first set of options allow
1075+
Node.js to be built so that it uses a shared library
1076+
at runtime instead of building and including the dependency
1077+
in the Node.js binary itself. These options are in the
1078+
`Shared libraries` section of the `configure` help
1079+
(run `./configure --help` to get the complete list).
1080+
They provide the ability to enable the use of a shared library,
1081+
to set the name of the shared library, and to set the paths that
1082+
contain the include and shared library files.
1083+
1084+
* For dependencies with JavaScript code (including WASM), the second
1085+
set of options allow the Node.js binary to be built so that it loads
1086+
the JavaScript for dependencies at runtime instead of being built into
1087+
the Node.js binary itself. These options are in the `Shared builtins`
1088+
section of the `configure` help
1089+
(run `./configure --help` to get the complete list). They
1090+
provide the ability to set the path to an external JavaScript file
1091+
for the dependency to be used at runtime.
1092+
1093+
It is the responsibility of any distribution
1094+
shipping with these options to:
1095+
1096+
* ensure that the shared dependencies available at runtime
1097+
match what is expected by the Node.js binary. A
1098+
mismatch may result in crashes or unexpected behavior.
1099+
* fully test that Node.js operates as expected with the
1100+
external dependencies. There may be little or no test coverage
1101+
within the Node.js project CI for these non-default options.
1102+
10651103
## Note for downstream distributors of Node.js
10661104

10671105
The Node.js ecosystem is reliant on ABI compatibility within a major release.
Collapse file

‎configure.py‎

Copy file name to clipboardExpand all lines: configure.py
+31Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,11 @@
5757
with open ('tools/icu/icu_versions.json') as f:
5858
icu_versions = json.load(f)
5959

60+
shareable_builtins = {'cjs_module_lexer/lexer': 'deps/cjs-module-lexer/lexer.js',
61+
'cjs_module_lexer/dist/lexer': 'deps/cjs-module-lexer/dist/lexer.js',
62+
'undici/undici': 'deps/undici/undici.js'
63+
}
64+
6065
# create option groups
6166
shared_optgroup = parser.add_argument_group("Shared libraries",
6267
"Flags that allows you to control whether you want to build against "
@@ -70,6 +75,9 @@
7075
"library you want to build against.")
7176
http2_optgroup = parser.add_argument_group("HTTP2",
7277
"Flags that allows you to control HTTP2 features in Node.js")
78+
shared_builtin_optgroup = parser.add_argument_group("Shared builtins",
79+
"Flags that allows you to control whether you want to build against "
80+
"internal builtins or shared files.")
7381

7482
# Options should be in alphabetical order but keep --prefix at the top,
7583
# that's arguably the one people will be looking for most.
@@ -427,6 +435,16 @@
427435

428436
parser.add_argument_group(shared_optgroup)
429437

438+
for builtin in shareable_builtins:
439+
builtin_id = 'shared_builtin_' + builtin + '_path'
440+
shared_builtin_optgroup.add_argument('--shared-builtin-' + builtin + '-path',
441+
action='store',
442+
dest='node_shared_builtin_' + builtin.replace('/', '_') + '_path',
443+
help='Path to shared file for ' + builtin + ' builtin. '
444+
'Will be used instead of bundled version at runtime')
445+
446+
parser.add_argument_group(shared_builtin_optgroup)
447+
430448
static_optgroup.add_argument('--static-zoslib-gyp',
431449
action='store',
432450
dest='static_zoslib_gyp',
@@ -2006,6 +2024,19 @@ def make_bin_override():
20062024
configure_inspector(output)
20072025
configure_section_file(output)
20082026

2027+
# configure shareable builtins
2028+
output['variables']['node_builtin_shareable_builtins'] = []
2029+
for builtin in shareable_builtins:
2030+
builtin_id = 'node_shared_builtin_' + builtin.replace('/', '_') + '_path'
2031+
if getattr(options, builtin_id):
2032+
if options.with_intl == 'none':
2033+
option_name = '--shared-builtin-' + builtin + '-path'
2034+
error(option_name + ' is incompatible with --with-intl=none' )
2035+
else:
2036+
output['defines'] += [builtin_id.upper() + '=' + getattr(options, builtin_id)]
2037+
else:
2038+
output['variables']['node_builtin_shareable_builtins'] += [shareable_builtins[builtin]]
2039+
20092040
# Forward OSS-Fuzz settings
20102041
output['variables']['ossfuzz'] = b(options.ossfuzz)
20112042

Collapse file

‎doc/contributing/maintaining-dependencies.md‎

Copy file name to clipboard
+106Lines changed: 106 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Maintaining Dependencies
2+
3+
Node.js depends on additional components beyond the Node.js code
4+
itself. These dependencies provide both native and JavaScript code
5+
and are built together with the code under the `src` and `lib`
6+
directories to create the Node.js binaries.
7+
8+
All dependencies are located within the `deps` directory.
9+
10+
Any code which meets one or more of these conditions should
11+
be managed as a dependency:
12+
13+
* originates in an upstream project and is maintained
14+
in that upstream project.
15+
* is not built from the `preferred form of the work for
16+
making modifications to it` (see
17+
[GNU GPL v2, section 3.](https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html)
18+
when `make node` is run. A good example is
19+
WASM code generated from C (the preferred form).
20+
Typically generation is only supported on a subset of platforms, needs
21+
additional tools, and is pre-built outside of the `make node`
22+
step and then committed as a WASM binary in the directory
23+
for the dependency under the `deps` directory.
24+
25+
By default all dependencies are bundled into the Node.js
26+
binary, however, `configure` options should be available to
27+
use an externalized version at runtime when:
28+
29+
* the dependency provides native code and is available as
30+
a shared library in one or more of the common Node.js
31+
distributions.
32+
* the dependency provides JavaScript and is not built
33+
from the `preferred form of the work for making modifications
34+
to it` when `make node` is run.
35+
36+
Many distributions use externalized dependencies for one or
37+
more of these reasons:
38+
39+
1. They have a requirement to build everything that they ship
40+
from the `preferred form of the work for making
41+
modifications to it`. This means that they need to
42+
replace any pre-built components (for example WASM
43+
binaries) with an equivalent that they have built.
44+
2. They manage the dependency separately as it is used by
45+
more applications than just Node.js. Linking against
46+
a shared library allows them to manage updates and
47+
CVE fixes against the library instead of having to
48+
patch all of the individual applications.
49+
3. They have a system wide configuration for the
50+
dependency that all applications should respect.
51+
52+
## Supporting externalized dependencies with native code.
53+
54+
Support for externalized dependencies with native code for which a
55+
shared library is available can added by:
56+
57+
* adding options to `configure.py`. These are added to the
58+
shared\_optgroup and include an options to:
59+
* enable use of a shared library
60+
* set the name of the shared library
61+
* set the path to the directory with the includes for the
62+
shared library
63+
* set the path to where to find the shared library at
64+
runtime
65+
* add a call to configure\_library() to `configure.py` for the
66+
library at the end of list of existing configure\_library() calls.
67+
If there are additional libraries that are required it is
68+
possible to list more than one with the `pkgname` option.
69+
* in `node.gypi` guard the build for the dependency
70+
with `node_shared_depname` so that is is only built if
71+
the dependency is being bundled into Node.js itself. For example:
72+
73+
```text
74+
[ 'node_shared_brotli=="false"', {
75+
'dependencies': [ 'deps/brotli/brotli.gyp:brotli' ],
76+
}],
77+
```
78+
79+
## Supporting externalizable dependencies with JavaScript codeIZA
80+
81+
Support for an externalizable dependency with JavaScript code
82+
can be added by:
83+
84+
* adding an entry to the `sharable_builtins` map in
85+
`configure.py`. The path should correspond to the file
86+
within the deps directory that is normally bundled into
87+
Node.js. For example `deps/cjs-module-lexer/lexer.js`.
88+
This will add a new option for building with that dependency
89+
externalized. After adding the entry you can see
90+
the new option by running `./configure --help`.
91+
92+
* adding a call to `AddExternalizedBuiltin` to the constructor
93+
for BuildinLoader in `src/node_builtins.cc` for the
94+
dependency using the `NODE_SHARED_BUILTLIN` #define generated for
95+
the dependency. After running `./configure` with the new
96+
option you can find the #define in `config.gypi`. You can cut and
97+
paste one of the existing entries and then update to match the
98+
inport name for the dependency and the #define generated.
99+
100+
## Supporting non-externalized dependencies with JavaScript code
101+
102+
If the dependency consists of JavaScript in the
103+
`preferred form of the work for making modifications to it`, it
104+
can be added as a non-externalizable dependency. In this case
105+
simply add the path to the JavaScript file in the `deps_files`
106+
list in the `node.gyp` file.
Collapse file

‎node.gyp‎

Copy file name to clipboardExpand all lines: node.gyp
+1-3Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -50,9 +50,7 @@
5050
'deps/v8/tools/tickprocessor-driver.mjs',
5151
'deps/acorn/acorn/dist/acorn.js',
5252
'deps/acorn/acorn-walk/dist/walk.js',
53-
'deps/cjs-module-lexer/lexer.js',
54-
'deps/cjs-module-lexer/dist/lexer.js',
55-
'deps/undici/undici.js',
53+
'<@(node_builtin_shareable_builtins)',
5654
],
5755
'node_mksnapshot_exec': '<(PRODUCT_DIR)/<(EXECUTABLE_PREFIX)node_mksnapshot<(EXECUTABLE_SUFFIX)',
5856
'mkcodecache_exec': '<(PRODUCT_DIR)/<(EXECUTABLE_PREFIX)mkcodecache<(EXECUTABLE_SUFFIX)',
Collapse file

‎src/node_builtins.cc‎

Copy file name to clipboardExpand all lines: src/node_builtins.cc
+42Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
#include "node_builtins.h"
2+
#include "node_builtins_env.h"
23
#include "debug_utils-inl.h"
34
#include "env-inl.h"
45
#include "node_external_reference.h"
@@ -23,6 +24,24 @@ BuiltinLoader BuiltinLoader::instance_;
2324

2425
BuiltinLoader::BuiltinLoader() : config_(GetConfig()) {
2526
LoadJavaScriptSource();
27+
#if defined(NODE_HAVE_I18N_SUPPORT)
28+
#ifdef NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_LEXER_PATH
29+
AddExternalizedBuiltin(
30+
"internal/deps/cjs-module-lexer/lexer",
31+
STRINGIFY(NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_LEXER_PATH));
32+
#endif // NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_LEXER_PATH
33+
34+
#ifdef NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_DIST_LEXER_PATH
35+
AddExternalizedBuiltin(
36+
"internal/deps/cjs-module-lexer/dist/lexer",
37+
STRINGIFY(NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_DIST_LEXER_PATH));
38+
#endif // NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_DIST_LEXER_PATH
39+
40+
#ifdef NODE_SHARED_BUILTIN_UNDICI_UNDICI_PATH
41+
AddExternalizedBuiltin("internal/deps/undici/undici",
42+
STRINGIFY(NODE_SHARED_BUILTIN_UNDICI_UNDICI_PATH));
43+
#endif // NODE_SHARED_BUILTIN_UNDICI_UNDICI_PATH
44+
#endif // NODE_HAVE_I18N_SUPPORT
2645
}
2746

2847
BuiltinLoader* BuiltinLoader::GetInstance() {
@@ -236,6 +255,29 @@ MaybeLocal<String> BuiltinLoader::LoadBuiltinSource(Isolate* isolate,
236255
#endif // NODE_BUILTIN_MODULES_PATH
237256
}
238257

258+
#if defined(NODE_HAVE_I18N_SUPPORT)
259+
void BuiltinLoader::AddExternalizedBuiltin(const char* id,
260+
const char* filename) {
261+
std::string source;
262+
int r = ReadFileSync(&source, filename);
263+
if (r != 0) {
264+
fprintf(
265+
stderr, "Cannot load externalized builtin: \"%s:%s\".\n", id, filename);
266+
ABORT();
267+
return;
268+
}
269+
270+
icu::UnicodeString utf16 = icu::UnicodeString::fromUTF8(
271+
icu::StringPiece(source.data(), source.length()));
272+
auto source_utf16 = std::make_unique<icu::UnicodeString>(utf16);
273+
BuiltinEnv::Add(id,
274+
UnionBytes(reinterpret_cast<const uint16_t*>((*source_utf16).getBuffer()),
275+
utf16.length()));
276+
// keep source bytes for builtin alive while BuiltinLoader exists
277+
GetInstance()->externalized_source_bytes_.push_back(std::move(source_utf16));
278+
}
279+
#endif // NODE_HAVE_I18N_SUPPORT
280+
239281
// Returns Local<Function> of the compiled module if return_code_cache
240282
// is false (we are only compiling the function).
241283
// Otherwise return a Local<Object> containing the cache.
Collapse file

‎src/node_builtins.h‎

Copy file name to clipboardExpand all lines: src/node_builtins.h
+11Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,14 @@
33

44
#if defined(NODE_WANT_INTERNALS) && NODE_WANT_INTERNALS
55

6+
#include <list>
67
#include <map>
78
#include <memory>
89
#include <set>
910
#include <string>
11+
#if defined(NODE_HAVE_I18N_SUPPORT)
12+
#include <unicode/unistr.h>
13+
#endif // NODE_HAVE_I18N_SUPPORT
1014
#include "node_mutex.h"
1115
#include "node_union_bytes.h"
1216
#include "v8.h"
@@ -76,11 +80,18 @@ class NODE_EXTERN_PRIVATE BuiltinLoader {
7680
const char* id,
7781
Result* result);
7882

83+
#if defined(NODE_HAVE_I18N_SUPPORT)
84+
static void AddExternalizedBuiltin(const char* id, const char* filename);
85+
#endif // NODE_HAVE_I18N_SUPPORT
86+
7987
static BuiltinLoader instance_;
8088
BuiltinCategories builtin_categories_;
8189
BuiltinSourceMap source_;
8290
BuiltinCodeCacheMap code_cache_;
8391
UnionBytes config_;
92+
#if defined(NODE_HAVE_I18N_SUPPORT)
93+
std::list<std::unique_ptr<icu::UnicodeString>> externalized_source_bytes_;
94+
#endif // NODE_HAVE_I18N_SUPPORT
8495

8596
// Used to synchronize access to the code cache map
8697
Mutex code_cache_mutex_;

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.