Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit ca5be26

Browse filesBrowse files
mhdawsonmhdawson
committed
src: add support for externally shared js builtins
Refs: #44000 - add infra to support externally shared js builtins in support of distos that want to externalize deps that include JS/WASM instead of native code - add support for externalizing - cjs_module_lexer/lexer - cjs_module_lexer/dist/lexer - undici/undici Signed-off-by: Michael Dawson <mdawson@devrus.com> PR-URL: #44376 Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
1 parent 36805e8 commit ca5be26
Copy full SHA for ca5be26

File tree

Expand file treeCollapse file tree

6 files changed

+228
-3
lines changed
Open diff view settings
Filter options
Expand file treeCollapse file tree

6 files changed

+228
-3
lines changed
Open diff view settings
Collapse file

‎BUILDING.md‎

Copy file name to clipboardExpand all lines: BUILDING.md
+38Lines changed: 38 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -1053,6 +1053,44 @@ To make `./myModule.js` available via `require('myModule')` and
10531053
> .\vcbuild link-module './myModule.js' link-module './myModule2.js'
10541054
```
10551055

1056+
## Building to use shared dependencies at runtime
1057+
1058+
By default Node.js is built so that all dependencies are bundled into
1059+
the Node.js binary itself. This provides a single binary that includes
1060+
the correct versions of all dependencies on which it depends.
1061+
1062+
Some Node.js distributions, however, prefer to manage dependencies.
1063+
A number of `configure` options are provided to support this use case.
1064+
1065+
* For dependencies with native code, the first set of options allow
1066+
Node.js to be built so that it uses a shared library
1067+
at runtime instead of building and including the dependency
1068+
in the Node.js binary itself. These options are in the
1069+
`Shared libraries` section of the `configure` help
1070+
(run `./configure --help` to get the complete list).
1071+
They provide the ability to enable the use of a shared library,
1072+
to set the name of the shared library, and to set the paths that
1073+
contain the include and shared library files.
1074+
1075+
* For dependencies with JavaScript code (including WASM), the second
1076+
set of options allow the Node.js binary to be built so that it loads
1077+
the JavaScript for dependencies at runtime instead of being built into
1078+
the Node.js binary itself. These options are in the `Shared builtins`
1079+
section of the `configure` help
1080+
(run `./configure --help` to get the complete list). They
1081+
provide the ability to set the path to an external JavaScript file
1082+
for the dependency to be used at runtime.
1083+
1084+
It is the responsibility of any distribution
1085+
shipping with these options to:
1086+
1087+
* ensure that the shared dependencies available at runtime
1088+
match what is expected by the Node.js binary. A
1089+
mismatch may result in crashes or unexpected behavior.
1090+
* fully test that Node.js operates as expected with the
1091+
external dependencies. There may be little or no test coverage
1092+
within the Node.js project CI for these non-default options.
1093+
10561094
## Note for downstream distributors of Node.js
10571095

10581096
The Node.js ecosystem is reliant on ABI compatibility within a major release.
Collapse file

‎configure.py‎

Copy file name to clipboardExpand all lines: configure.py
+31Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,11 @@
5757
with open ('tools/icu/icu_versions.json') as f:
5858
icu_versions = json.load(f)
5959

60+
shareable_builtins = {'cjs_module_lexer/lexer': 'deps/cjs-module-lexer/lexer.js',
61+
'cjs_module_lexer/dist/lexer': 'deps/cjs-module-lexer/dist/lexer.js',
62+
'undici/undici': 'deps/undici/undici.js'
63+
}
64+
6065
# create option groups
6166
shared_optgroup = parser.add_argument_group("Shared libraries",
6267
"Flags that allows you to control whether you want to build against "
@@ -70,6 +75,9 @@
7075
"library you want to build against.")
7176
http2_optgroup = parser.add_argument_group("HTTP2",
7277
"Flags that allows you to control HTTP2 features in Node.js")
78+
shared_builtin_optgroup = parser.add_argument_group("Shared builtins",
79+
"Flags that allows you to control whether you want to build against "
80+
"internal builtins or shared files.")
7381

7482
# Options should be in alphabetical order but keep --prefix at the top,
7583
# that's arguably the one people will be looking for most.
@@ -422,6 +430,16 @@
422430

423431
parser.add_argument_group(shared_optgroup)
424432

433+
for builtin in shareable_builtins:
434+
builtin_id = 'shared_builtin_' + builtin + '_path'
435+
shared_builtin_optgroup.add_argument('--shared-builtin-' + builtin + '-path',
436+
action='store',
437+
dest='node_shared_builtin_' + builtin.replace('/', '_') + '_path',
438+
help='Path to shared file for ' + builtin + ' builtin. '
439+
'Will be used instead of bundled version at runtime')
440+
441+
parser.add_argument_group(shared_builtin_optgroup)
442+
425443
static_optgroup.add_argument('--static-zoslib-gyp',
426444
action='store',
427445
dest='static_zoslib_gyp',
@@ -1960,6 +1978,19 @@ def make_bin_override():
19601978
configure_inspector(output)
19611979
configure_section_file(output)
19621980

1981+
# configure shareable builtins
1982+
output['variables']['node_builtin_shareable_builtins'] = []
1983+
for builtin in shareable_builtins:
1984+
builtin_id = 'node_shared_builtin_' + builtin.replace('/', '_') + '_path'
1985+
if getattr(options, builtin_id):
1986+
if options.with_intl == 'none':
1987+
option_name = '--shared-builtin-' + builtin + '-path'
1988+
error(option_name + ' is incompatible with --with-intl=none' )
1989+
else:
1990+
output['defines'] += [builtin_id.upper() + '=' + getattr(options, builtin_id)]
1991+
else:
1992+
output['variables']['node_builtin_shareable_builtins'] += [shareable_builtins[builtin]]
1993+
19631994
# Forward OSS-Fuzz settings
19641995
output['variables']['ossfuzz'] = b(options.ossfuzz)
19651996

Collapse file

‎doc/contributing/maintaining-dependencies.md‎

Copy file name to clipboard
+106Lines changed: 106 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Maintaining Dependencies
2+
3+
Node.js depends on additional components beyond the Node.js code
4+
itself. These dependencies provide both native and JavaScript code
5+
and are built together with the code under the `src` and `lib`
6+
directories to create the Node.js binaries.
7+
8+
All dependencies are located within the `deps` directory.
9+
10+
Any code which meets one or more of these conditions should
11+
be managed as a dependency:
12+
13+
* originates in an upstream project and is maintained
14+
in that upstream project.
15+
* is not built from the `preferred form of the work for
16+
making modifications to it` (see
17+
[GNU GPL v2, section 3.](https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html)
18+
when `make node` is run. A good example is
19+
WASM code generated from C (the preferred form).
20+
Typically generation is only supported on a subset of platforms, needs
21+
additional tools, and is pre-built outside of the `make node`
22+
step and then committed as a WASM binary in the directory
23+
for the dependency under the `deps` directory.
24+
25+
By default all dependencies are bundled into the Node.js
26+
binary, however, `configure` options should be available to
27+
use an externalized version at runtime when:
28+
29+
* the dependency provides native code and is available as
30+
a shared library in one or more of the common Node.js
31+
distributions.
32+
* the dependency provides JavaScript and is not built
33+
from the `preferred form of the work for making modifications
34+
to it` when `make node` is run.
35+
36+
Many distributions use externalized dependencies for one or
37+
more of these reasons:
38+
39+
1. They have a requirement to build everything that they ship
40+
from the `preferred form of the work for making
41+
modifications to it`. This means that they need to
42+
replace any pre-built components (for example WASM
43+
binaries) with an equivalent that they have built.
44+
2. They manage the dependency separately as it is used by
45+
more applications than just Node.js. Linking against
46+
a shared library allows them to manage updates and
47+
CVE fixes against the library instead of having to
48+
patch all of the individual applications.
49+
3. They have a system wide configuration for the
50+
dependency that all applications should respect.
51+
52+
## Supporting externalized dependencies with native code.
53+
54+
Support for externalized dependencies with native code for which a
55+
shared library is available can added by:
56+
57+
* adding options to `configure.py`. These are added to the
58+
shared\_optgroup and include an options to:
59+
* enable use of a shared library
60+
* set the name of the shared library
61+
* set the path to the directory with the includes for the
62+
shared library
63+
* set the path to where to find the shared library at
64+
runtime
65+
* add a call to configure\_library() to `configure.py` for the
66+
library at the end of list of existing configure\_library() calls.
67+
If there are additional libraries that are required it is
68+
possible to list more than one with the `pkgname` option.
69+
* in `node.gypi` guard the build for the dependency
70+
with `node_shared_depname` so that is is only built if
71+
the dependency is being bundled into Node.js itself. For example:
72+
73+
```text
74+
[ 'node_shared_brotli=="false"', {
75+
'dependencies': [ 'deps/brotli/brotli.gyp:brotli' ],
76+
}],
77+
```
78+
79+
## Supporting externalizable dependencies with JavaScript codeIZA
80+
81+
Support for an externalizable dependency with JavaScript code
82+
can be added by:
83+
84+
* adding an entry to the `sharable_builtins` map in
85+
`configure.py`. The path should correspond to the file
86+
within the deps directory that is normally bundled into
87+
Node.js. For example `deps/cjs-module-lexer/lexer.js`.
88+
This will add a new option for building with that dependency
89+
externalized. After adding the entry you can see
90+
the new option by running `./configure --help`.
91+
92+
* adding a call to `AddExternalizedBuiltin` to the constructor
93+
for BuildinLoader in `src/node_builtins.cc` for the
94+
dependency using the `NODE_SHARED_BUILTLIN` #define generated for
95+
the dependency. After running `./configure` with the new
96+
option you can find the #define in `config.gypi`. You can cut and
97+
paste one of the existing entries and then update to match the
98+
inport name for the dependency and the #define generated.
99+
100+
## Supporting non-externalized dependencies with JavaScript code
101+
102+
If the dependency consists of JavaScript in the
103+
`preferred form of the work for making modifications to it`, it
104+
can be added as a non-externalizable dependency. In this case
105+
simply add the path to the JavaScript file in the `deps_files`
106+
list in the `node.gyp` file.
Collapse file

‎node.gyp‎

Copy file name to clipboardExpand all lines: node.gyp
+1-3Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -48,9 +48,7 @@
4848
'deps/v8/tools/tickprocessor-driver.mjs',
4949
'deps/acorn/acorn/dist/acorn.js',
5050
'deps/acorn/acorn-walk/dist/walk.js',
51-
'deps/cjs-module-lexer/lexer.js',
52-
'deps/cjs-module-lexer/dist/lexer.js',
53-
'deps/undici/undici.js',
51+
'<@(node_builtin_shareable_builtins)',
5452
],
5553
'node_mksnapshot_exec': '<(PRODUCT_DIR)/<(EXECUTABLE_PREFIX)node_mksnapshot<(EXECUTABLE_SUFFIX)',
5654
'conditions': [
Collapse file

‎src/node_builtins.cc‎

Copy file name to clipboardExpand all lines: src/node_builtins.cc
+41Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,24 @@ BuiltinLoader BuiltinLoader::instance_;
3333

3434
BuiltinLoader::BuiltinLoader() : config_(GetConfig()), has_code_cache_(false) {
3535
LoadJavaScriptSource();
36+
#if defined(NODE_HAVE_I18N_SUPPORT)
37+
#ifdef NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_LEXER_PATH
38+
AddExternalizedBuiltin(
39+
"internal/deps/cjs-module-lexer/lexer",
40+
STRINGIFY(NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_LEXER_PATH));
41+
#endif // NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_LEXER_PATH
42+
43+
#ifdef NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_DIST_LEXER_PATH
44+
AddExternalizedBuiltin(
45+
"internal/deps/cjs-module-lexer/dist/lexer",
46+
STRINGIFY(NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_DIST_LEXER_PATH));
47+
#endif // NODE_SHARED_BUILTIN_CJS_MODULE_LEXER_DIST_LEXER_PATH
48+
49+
#ifdef NODE_SHARED_BUILTIN_UNDICI_UNDICI_PATH
50+
AddExternalizedBuiltin("internal/deps/undici/undici",
51+
STRINGIFY(NODE_SHARED_BUILTIN_UNDICI_UNDICI_PATH));
52+
#endif // NODE_SHARED_BUILTIN_UNDICI_UNDICI_PATH
53+
#endif // NODE_HAVE_I18N_SUPPORT
3654
}
3755

3856
BuiltinLoader* BuiltinLoader::GetInstance() {
@@ -220,6 +238,29 @@ MaybeLocal<String> BuiltinLoader::LoadBuiltinSource(Isolate* isolate,
220238
#endif // NODE_BUILTIN_MODULES_PATH
221239
}
222240

241+
#if defined(NODE_HAVE_I18N_SUPPORT)
242+
void BuiltinLoader::AddExternalizedBuiltin(const char* id,
243+
const char* filename) {
244+
std::string source;
245+
int r = ReadFileSync(&source, filename);
246+
if (r != 0) {
247+
fprintf(
248+
stderr, "Cannot load externalized builtin: \"%s:%s\".\n", id, filename);
249+
ABORT();
250+
return;
251+
}
252+
253+
icu::UnicodeString utf16 = icu::UnicodeString::fromUTF8(
254+
icu::StringPiece(source.data(), source.length()));
255+
auto source_utf16 = std::make_unique<icu::UnicodeString>(utf16);
256+
Add(id,
257+
UnionBytes(reinterpret_cast<const uint16_t*>((*source_utf16).getBuffer()),
258+
utf16.length()));
259+
// keep source bytes for builtin alive while BuiltinLoader exists
260+
GetInstance()->externalized_source_bytes_.push_back(std::move(source_utf16));
261+
}
262+
#endif // NODE_HAVE_I18N_SUPPORT
263+
223264
// Returns Local<Function> of the compiled module if return_code_cache
224265
// is false (we are only compiling the function).
225266
// Otherwise return a Local<Object> containing the cache.
Collapse file

‎src/node_builtins.h‎

Copy file name to clipboardExpand all lines: src/node_builtins.h
+11Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,14 @@
33

44
#if defined(NODE_WANT_INTERNALS) && NODE_WANT_INTERNALS
55

6+
#include <list>
67
#include <map>
78
#include <memory>
89
#include <set>
910
#include <string>
11+
#if defined(NODE_HAVE_I18N_SUPPORT)
12+
#include <unicode/unistr.h>
13+
#endif // NODE_HAVE_I18N_SUPPORT
1014
#include <vector>
1115
#include "node_mutex.h"
1216
#include "node_union_bytes.h"
@@ -132,11 +136,18 @@ class NODE_EXTERN_PRIVATE BuiltinLoader {
132136
static void HasCachedBuiltins(
133137
const v8::FunctionCallbackInfo<v8::Value>& args);
134138

139+
#if defined(NODE_HAVE_I18N_SUPPORT)
140+
static void AddExternalizedBuiltin(const char* id, const char* filename);
141+
#endif // NODE_HAVE_I18N_SUPPORT
142+
135143
static BuiltinLoader instance_;
136144
BuiltinCategories builtin_categories_;
137145
BuiltinSourceMap source_;
138146
BuiltinCodeCacheMap code_cache_;
139147
UnionBytes config_;
148+
#if defined(NODE_HAVE_I18N_SUPPORT)
149+
std::list<std::unique_ptr<icu::UnicodeString>> externalized_source_bytes_;
150+
#endif // NODE_HAVE_I18N_SUPPORT
140151

141152
// Used to synchronize access to the code cache map
142153
Mutex code_cache_mutex_;

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.