Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 4c317ce

Browse filesBrowse files
addaleaxtargos
addaleax
authored and
targos
committed
child_process,cluster: allow using V8 serialization API
Add an `serialization` option that allows child process IPC to use the (typically more powerful) V8 serialization API. Fixes: #10965 PR-URL: #30162 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: David Carlier <devnexen@gmail.com> Reviewed-By: Michaël Zasso <targos@protonmail.com>
1 parent 56188fe commit 4c317ce
Copy full SHA for 4c317ce

File tree

Expand file treeCollapse file tree

13 files changed

+304
-39
lines changed
Open diff view settings
Filter options
Expand file treeCollapse file tree

13 files changed

+304
-39
lines changed
Open diff view settings
Collapse file

‎benchmark/cluster/echo.js‎

Copy file name to clipboardExpand all lines: benchmark/cluster/echo.js
+10-1Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,16 +7,25 @@ if (cluster.isMaster) {
77
workers: [1],
88
payload: ['string', 'object'],
99
sendsPerBroadcast: [1, 10],
10+
serialization: ['json', 'advanced'],
1011
n: [1e5]
1112
});
1213

13-
function main({ n, workers, sendsPerBroadcast, payload }) {
14+
function main({
15+
n,
16+
workers,
17+
sendsPerBroadcast,
18+
payload,
19+
serialization
20+
}) {
1421
const expectedPerBroadcast = sendsPerBroadcast * workers;
1522
var readies = 0;
1623
var broadcasts = 0;
1724
var msgCount = 0;
1825
var data;
1926

27+
cluster.settings.serialization = serialization;
28+
2029
switch (payload) {
2130
case 'string':
2231
data = 'hello world!';
Collapse file

‎doc/api/child_process.md‎

Copy file name to clipboardExpand all lines: doc/api/child_process.md
+39Lines changed: 39 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -321,6 +321,9 @@ arbitrary command execution.**
321321
<!-- YAML
322322
added: v0.5.0
323323
changes:
324+
- version: REPLACEME
325+
pr-url: https://github.com/nodejs/node/pull/30162
326+
description: The `serialization` option is supported now.
324327
- version: v8.0.0
325328
pr-url: https://github.com/nodejs/node/pull/10866
326329
description: The `stdio` option can now be a string.
@@ -340,6 +343,9 @@ changes:
340343
* `execPath` {string} Executable used to create the child process.
341344
* `execArgv` {string[]} List of string arguments passed to the executable.
342345
**Default:** `process.execArgv`.
346+
* `serialization` {string} Specify the kind of serialization used for sending
347+
messages between processes. Possible values are `'json'` and `'advanced'`.
348+
See [Advanced Serialization][] for more details. **Default:** `'json'`.
343349
* `silent` {boolean} If `true`, stdin, stdout, and stderr of the child will be
344350
piped to the parent, otherwise they will be inherited from the parent, see
345351
the `'pipe'` and `'inherit'` options for [`child_process.spawn()`][]'s
@@ -386,6 +392,9 @@ The `shell` option available in [`child_process.spawn()`][] is not supported by
386392
<!-- YAML
387393
added: v0.1.90
388394
changes:
395+
- version: REPLACEME
396+
pr-url: https://github.com/nodejs/node/pull/30162
397+
description: The `serialization` option is supported now.
389398
- version: v8.8.0
390399
pr-url: https://github.com/nodejs/node/pull/15380
391400
description: The `windowsHide` option is supported now.
@@ -411,6 +420,9 @@ changes:
411420
[`options.detached`][]).
412421
* `uid` {number} Sets the user identity of the process (see setuid(2)).
413422
* `gid` {number} Sets the group identity of the process (see setgid(2)).
423+
* `serialization` {string} Specify the kind of serialization used for sending
424+
messages between processes. Possible values are `'json'` and `'advanced'`.
425+
See [Advanced Serialization][] for more details. **Default:** `'json'`.
414426
* `shell` {boolean|string} If `true`, runs `command` inside of a shell. Uses
415427
`'/bin/sh'` on Unix, and `process.env.ComSpec` on Windows. A different
416428
shell can be specified as a string. See [Shell Requirements][] and
@@ -998,6 +1010,11 @@ The `'message'` event is triggered when a child process uses
9981010
The message goes through serialization and parsing. The resulting
9991011
message might not be the same as what is originally sent.
10001012

1013+
If the `serialization` option was set to `'advanced'` used when spawning the
1014+
child process, the `message` argument can contain data that JSON is not able
1015+
to represent.
1016+
See [Advanced Serialization][] for more details.
1017+
10011018
### `subprocess.channel`
10021019
<!-- YAML
10031020
added: v7.1.0
@@ -1474,6 +1491,26 @@ the same requirement. Thus, in `child_process` functions where a shell can be
14741491
spawned, `'cmd.exe'` is used as a fallback if `process.env.ComSpec` is
14751492
unavailable.
14761493

1494+
## Advanced Serialization
1495+
<!-- YAML
1496+
added: REPLACEME
1497+
-->
1498+
1499+
Child processes support a serialization mechanism for IPC that is based on the
1500+
[serialization API of the `v8` module][v8.serdes], based on the
1501+
[HTML structured clone algorithm][]. This is generally more powerful and
1502+
supports more built-in JavaScript object types, such as `BigInt`, `Map`
1503+
and `Set`, `ArrayBuffer` and `TypedArray`, `Buffer`, `Error`, `RegExp` etc.
1504+
1505+
However, this format is not a full superset of JSON, and e.g. properties set on
1506+
objects of such built-in types will not be passed on through the serialization
1507+
step. Additionally, performance may not be equivalent to that of JSON, depending
1508+
on the structure of the passed data.
1509+
Therefore, this feature requires opting in by setting the
1510+
`serialization` option to `'advanced'` when calling [`child_process.spawn()`][]
1511+
or [`child_process.fork()`][].
1512+
1513+
[Advanced Serialization]: #child_process_advanced_serialization
14771514
[`'disconnect'`]: process.html#process_event_disconnect
14781515
[`'error'`]: #child_process_event_error
14791516
[`'exit'`]: #child_process_event_exit
@@ -1507,5 +1544,7 @@ unavailable.
15071544
[`subprocess.stdout`]: #child_process_subprocess_stdout
15081545
[`util.promisify()`]: util.html#util_util_promisify_original
15091546
[Default Windows Shell]: #child_process_default_windows_shell
1547+
[HTML structured clone algorithm]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm
15101548
[Shell Requirements]: #child_process_shell_requirements
15111549
[synchronous counterparts]: #child_process_synchronous_process_creation
1550+
[v8.serdes]: v8.html#v8_serialization_api
Collapse file

‎doc/api/cluster.md‎

Copy file name to clipboardExpand all lines: doc/api/cluster.md
+8Lines changed: 8 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -724,6 +724,9 @@ values are `'rr'` and `'none'`.
724724
<!-- YAML
725725
added: v0.7.1
726726
changes:
727+
- version: REPLACEME
728+
pr-url: https://github.com/nodejs/node/pull/30162
729+
description: The `serialization` option is supported now.
727730
- version: v9.5.0
728731
pr-url: https://github.com/nodejs/node/pull/18399
729732
description: The `cwd` option is supported now.
@@ -746,6 +749,10 @@ changes:
746749
**Default:** `process.argv.slice(2)`.
747750
* `cwd` {string} Current working directory of the worker process. **Default:**
748751
`undefined` (inherits from parent process).
752+
* `serialization` {string} Specify the kind of serialization used for sending
753+
messages between processes. Possible values are `'json'` and `'advanced'`.
754+
See [Advanced Serialization for `child_process`][] for more details.
755+
**Default:** `false`.
749756
* `silent` {boolean} Whether or not to send output to parent's stdio.
750757
**Default:** `false`.
751758
* `stdio` {Array} Configures the stdio of forked processes. Because the
@@ -874,4 +881,5 @@ socket.on('data', (id) => {
874881
[`process` event: `'message'`]: process.html#process_event_message
875882
[`server.close()`]: net.html#net_event_close
876883
[`worker.exitedAfterDisconnect`]: #cluster_worker_exitedafterdisconnect
884+
[Advanced Serialization for `child_process`]: child_process.html#child_process_advanced_serialization
877885
[Child Process module]: child_process.html#child_process_child_process_fork_modulepath_args_options
Collapse file

‎doc/api/process.md‎

Copy file name to clipboardExpand all lines: doc/api/process.md
+6Lines changed: 6 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,11 @@ the child process.
119119
The message goes through serialization and parsing. The resulting message might
120120
not be the same as what is originally sent.
121121

122+
If the `serialization` option was set to `advanced` used when spawning the
123+
process, the `message` argument can contain data that JSON is not able
124+
to represent.
125+
See [Advanced Serialization for `child_process`][] for more details.
126+
122127
### Event: `'multipleResolves'`
123128
<!-- YAML
124129
added: v10.12.0
@@ -2457,6 +2462,7 @@ cases:
24572462
[`require.resolve()`]: modules.html#modules_require_resolve_request_options
24582463
[`subprocess.kill()`]: child_process.html#child_process_subprocess_kill_signal
24592464
[`v8.setFlagsFromString()`]: v8.html#v8_v8_setflagsfromstring_flags
2465+
[Advanced Serialization for `child_process`]: child_process.html#child_process_advanced_serialization
24602466
[Android building]: https://github.com/nodejs/node/blob/master/BUILDING.md#androidandroid-based-devices-eg-firefox-os
24612467
[Child Process]: child_process.html
24622468
[Cluster]: cluster.html
Collapse file

‎lib/child_process.js‎

Copy file name to clipboardExpand all lines: lib/child_process.js
+4-3Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -108,12 +108,12 @@ function fork(modulePath /* , args, options */) {
108108
return spawn(options.execPath, args, options);
109109
}
110110

111-
function _forkChild(fd) {
111+
function _forkChild(fd, serializationMode) {
112112
// set process.send()
113113
const p = new Pipe(PipeConstants.IPC);
114114
p.open(fd);
115115
p.unref();
116-
const control = setupChannel(process, p);
116+
const control = setupChannel(process, p, serializationMode);
117117
process.on('newListener', function onNewListener(name) {
118118
if (name === 'message' || name === 'disconnect') control.ref();
119119
});
@@ -547,7 +547,8 @@ function spawn(file, args, options) {
547547
envPairs: opts.envPairs,
548548
stdio: options.stdio,
549549
uid: options.uid,
550-
gid: options.gid
550+
gid: options.gid,
551+
serialization: options.serialization,
551552
});
552553

553554
return child;
Collapse file

‎lib/internal/bootstrap/pre_execution.js‎

Copy file name to clipboardExpand all lines: lib/internal/bootstrap/pre_execution.js
+5-1Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -326,7 +326,11 @@ function setupChildProcessIpcChannel() {
326326
// Make sure it's not accidentally inherited by child processes.
327327
delete process.env.NODE_CHANNEL_FD;
328328

329-
require('child_process')._forkChild(fd);
329+
const serializationMode =
330+
process.env.NODE_CHANNEL_SERIALIZATION_MODE || 'json';
331+
delete process.env.NODE_CHANNEL_SERIALIZATION_MODE;
332+
333+
require('child_process')._forkChild(fd, serializationMode);
330334
assert(process.send);
331335
}
332336
}
Collapse file

‎lib/internal/child_process.js‎

Copy file name to clipboardExpand all lines: lib/internal/child_process.js
+27-32Lines changed: 27 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
'use strict';
22

3-
const { JSON, Object } = primordials;
3+
const { Object } = primordials;
44

55
const {
66
errnoException,
@@ -55,8 +55,6 @@ const {
5555

5656
const { SocketListSend, SocketListReceive } = SocketList;
5757

58-
// Lazy loaded for startup performance.
59-
let StringDecoder;
6058
// Lazy loaded for startup performance and to allow monkey patching of
6159
// internalBinding('http_parser').HTTPParser.
6260
let freeParser;
@@ -343,6 +341,15 @@ ChildProcess.prototype.spawn = function(options) {
343341
const ipcFd = stdio.ipcFd;
344342
stdio = options.stdio = stdio.stdio;
345343

344+
if (options.serialization !== undefined &&
345+
options.serialization !== 'json' &&
346+
options.serialization !== 'advanced') {
347+
throw new ERR_INVALID_OPT_VALUE('options.serialization',
348+
options.serialization);
349+
}
350+
351+
const serialization = options.serialization || 'json';
352+
346353
if (ipc !== undefined) {
347354
// Let child process know about opened IPC channel
348355
if (options.envPairs === undefined)
@@ -353,7 +360,8 @@ ChildProcess.prototype.spawn = function(options) {
353360
options.envPairs);
354361
}
355362

356-
options.envPairs.push('NODE_CHANNEL_FD=' + ipcFd);
363+
options.envPairs.push(`NODE_CHANNEL_FD=${ipcFd}`);
364+
options.envPairs.push(`NODE_CHANNEL_SERIALIZATION_MODE=${serialization}`);
357365
}
358366

359367
validateString(options.file, 'options.file');
@@ -446,7 +454,7 @@ ChildProcess.prototype.spawn = function(options) {
446454
this.stdio.push(stdio[i].socket === undefined ? null : stdio[i].socket);
447455

448456
// Add .send() method and start listening for IPC data
449-
if (ipc !== undefined) setupChannel(this, ipc);
457+
if (ipc !== undefined) setupChannel(this, ipc, serialization);
450458

451459
return err;
452460
};
@@ -513,7 +521,8 @@ class Control extends EventEmitter {
513521
}
514522
}
515523

516-
function setupChannel(target, channel) {
524+
let serialization;
525+
function setupChannel(target, channel, serializationMode) {
517526
target.channel = channel;
518527

519528
// _channel can be deprecated in version 8
@@ -528,12 +537,16 @@ function setupChannel(target, channel) {
528537

529538
const control = new Control(channel);
530539

531-
if (StringDecoder === undefined)
532-
StringDecoder = require('string_decoder').StringDecoder;
533-
const decoder = new StringDecoder('utf8');
534-
var jsonBuffer = '';
535-
var pendingHandle = null;
536-
channel.buffering = false;
540+
if (serialization === undefined)
541+
serialization = require('internal/child_process/serialization');
542+
const {
543+
initMessageChannel,
544+
parseChannelMessages,
545+
writeChannelMessage
546+
} = serialization[serializationMode];
547+
548+
let pendingHandle = null;
549+
initMessageChannel(channel);
537550
channel.pendingHandle = null;
538551
channel.onread = function(arrayBuffer) {
539552
const recvHandle = channel.pendingHandle;
@@ -545,21 +558,7 @@ function setupChannel(target, channel) {
545558
if (recvHandle)
546559
pendingHandle = recvHandle;
547560

548-
// Linebreak is used as a message end sign
549-
var chunks = decoder.write(pool).split('\n');
550-
var numCompleteChunks = chunks.length - 1;
551-
// Last line does not have trailing linebreak
552-
var incompleteChunk = chunks[numCompleteChunks];
553-
if (numCompleteChunks === 0) {
554-
jsonBuffer += incompleteChunk;
555-
this.buffering = jsonBuffer.length !== 0;
556-
return;
557-
}
558-
chunks[0] = jsonBuffer + chunks[0];
559-
560-
for (var i = 0; i < numCompleteChunks; i++) {
561-
var message = JSON.parse(chunks[i]);
562-
561+
for (const message of parseChannelMessages(channel, pool)) {
563562
// There will be at most one NODE_HANDLE message in every chunk we
564563
// read because SCM_RIGHTS messages don't get coalesced. Make sure
565564
// that we deliver the handle with the right message however.
@@ -574,9 +573,6 @@ function setupChannel(target, channel) {
574573
handleMessage(message, undefined, false);
575574
}
576575
}
577-
jsonBuffer = incompleteChunk;
578-
this.buffering = jsonBuffer.length !== 0;
579-
580576
} else {
581577
this.buffering = false;
582578
target.disconnect();
@@ -775,8 +771,7 @@ function setupChannel(target, channel) {
775771

776772
const req = new WriteWrap();
777773

778-
const string = JSON.stringify(message) + '\n';
779-
const err = channel.writeUtf8String(req, string, handle);
774+
const err = writeChannelMessage(channel, req, message, handle);
780775
const wasAsyncWrite = streamBaseState[kLastWriteWasAsync];
781776

782777
if (err === 0) {

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.