Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Commit c9b2885

Browse filesBrowse files
committed
Move typed arrays into a folder and add a readme
1 parent 24e6e64 commit c9b2885
Copy full SHA for c9b2885

File tree

Expand file treeCollapse file tree

3 files changed

+93
-5
lines changed
Filter options
Expand file treeCollapse file tree

3 files changed

+93
-5
lines changed

‎example/typed-arrays/README.md

Copy file name to clipboard
+88Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# MessagePack typed arrays
2+
3+
This is an extension to MessagePack which provides "native" support for JS's TypedArray family.
4+
5+
## Why?
6+
7+
The official JS library can already handle TypedArrays by serialising them as binary data, but this has two disadvantages:
8+
9+
1. You must know, and manually construct, the correct type of array from raw binary data after deserialising.
10+
2. The data is unaligned, which may require copying it into a new array before using it.
11+
12+
Number 2 is the main reason I was inspired to write an extension to handle these types; I didn't want to give up on the possibility of zero-copy decoding.
13+
14+
## Spec
15+
16+
`data` has some internal layout which looks like the following:
17+
18+
```
19+
+--------+--------+========+========+
20+
| artype |AAAAAAAA| align | vals |
21+
+--------+--------+========+========+
22+
```
23+
24+
Where:
25+
26+
- `artype` is an identifier for the type of array that is stored
27+
- `AAAAAAAA` is an 8-bit unsigned integer
28+
- `align` is a number of bytes equal to the value of `AAAAAAAA`, all of which contain 0
29+
- `vals` is the binary content of the TypedArray
30+
31+
The value of `AAAAAAAA`, and therefore the number of bytes in the `align` segment, is determined so that `cont` begins on a byte offset from the _beginning of the encoded MessagePack object_ which correctly aligns `cont` for efficient access.
32+
33+
If `AAAAAAAA` is 0, then there are no `align` bytes, and `vals` begins immediately after.
34+
35+
Note that the length of `data`, and therefore the value of `YYYYYYYY_YYYYYYYY` includes _all_ of `artype`, `AAAAAAAA`, `align` and `vals`.
36+
37+
## Example
38+
39+
A Float32Array containing 10 values will have a `data` size starting at 42 bytes if there is no alignment:
40+
41+
- 1 byte of `artype` = `0x??`
42+
- 1 byte of `AAAAAAAA` = 0
43+
- 0 bytes of `align`
44+
- 40 bytes of `vals`
45+
46+
A Float32Array should be aligned on 4-byte boundaries, so there may need to be up to 3 bytes of padding.
47+
In that case, the total size of `data` woulb become so this may increase to 45 bytes:
48+
49+
- 1 byte of `artype` = `0x??`
50+
- 1 byte of `AAAAAAAA` = 3
51+
- 3 bytes of `align`
52+
- 40 bytes of `vals`
53+
54+
Since the extension array is wrapped with its own header, there is some additional structure before this content.
55+
56+
See the [MessagePack spec for extensions](https://github.com/msgpack/msgpack/blob/master/spec.md#ext-format-family).
57+
The content of a TypedArray object is inserted after the extension header.
58+
For example, an extension where the size of the encoded array is up to (2^8)-1 bytes will be laid out like this:
59+
60+
```
61+
+--------+--------+--------+========+
62+
| 0xc7 |XXXXXXXX| type | data |
63+
+--------+--------+--------+========+
64+
```
65+
66+
Where:
67+
68+
- `0xc8` is the `ext 16` header
69+
- `XXXXXXXX` is a 8-bit unsigned integer which represents the length of `data` in bytes
70+
- `type` is the extension type number 0-127
71+
72+
So to put the entire example of a 10-entry Float32Array together, it would be represented as:
73+
74+
```
75+
+--------+--------+--------+--------+--------+========+========+
76+
| 0xc7 | 0x2D | type | 0x?? | 0x03 |3 zeros | vals |
77+
+--------+--------+--------+--------+--------+========+========+
78+
```
79+
80+
Where:
81+
82+
- `0xc7` is the MessagePack type for `ext 8`
83+
- `0x2D` is 45, the length of the TypedArray payload described above
84+
- `type` is the extension type number
85+
- `0x??` is the `artype` number for Float32Array
86+
- `0x03` is the number of alignment bytes
87+
- 3 zeros are required for alignment
88+
- `vals` contains the actual floating-point data

‎example/typed-array-plugin-example.ts renamed to ‎example/typed-arrays/example.ts

Copy file name to clipboardExpand all lines: example/typed-arrays/example.ts
+3-3Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
// ts-node example/typed-array-plugin-example.ts
1+
// ts-node example/typed-arrays/example.ts
22

3-
import { encode, decode, ExtensionCodec } from "../src";
4-
import { typedArrays } from "../example/typed-arrays-plugin";
3+
import { encode, decode, ExtensionCodec } from "../../src";
4+
import { typedArrays } from "./plugin";
55

66
const extensionCodec = new ExtensionCodec();
77
extensionCodec.registerPlugin(typedArrays({ type: 1 }));

‎example/typed-arrays-plugin.ts renamed to ‎example/typed-arrays/plugin.ts

Copy file name to clipboardExpand all lines: example/typed-arrays/plugin.ts
+2-2Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
import type { Encoder } from "../src/Encoder";
2-
import { ensureUint8Array } from "../src/utils/typedArrays";
1+
import type { Encoder } from "../../src/Encoder";
2+
import { ensureUint8Array } from "../../src/utils/typedArrays";
33

44
export function typedArrays<C>({type}: {type: number}) {
55
const TypedArray = Object.getPrototypeOf(Int8Array);

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.