Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 3af44ee

Browse filesBrowse files
thisalihassanaduh95
thisalihassan
authored and
aduh95
committed
sqlite: add serialize() and deserialize()
Add database.serialize() and database.deserialize() methods to DatabaseSync, wrapping the sqlite3_serialize and sqlite3_deserialize C APIs. These allow extracting an in-memory database as a Uint8Array and loading one back, enabling snapshots, cloning, and transfer of databases without filesystem I/O. Refs: #62575 PR-URL: #62579 Refs: #62575 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Edy Silva <edigleyssonsilva@gmail.com>
1 parent 57e59ea commit 3af44ee
Copy full SHA for 3af44ee

4 files changed

+526Lines changed: 526 additions & 0 deletions

File tree

Expand file treeCollapse file tree
Open diff view settings
Filter options
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎doc/api/sqlite.md‎

Copy file name to clipboardExpand all lines: doc/api/sqlite.md
+86Lines changed: 86 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -531,6 +531,89 @@ Opens the database specified in the `path` argument of the `DatabaseSync`
531531
constructor. This method should only be used when the database is not opened via
532532
the constructor. An exception is thrown if the database is already open.
533533

534+
### `database.serialize([dbName])`
535+
536+
<!-- YAML
537+
added: REPLACEME
538+
-->
539+
540+
* `dbName` {string} Name of the database to serialize. This can be `'main'`
541+
(the default primary database) or any other database that has been added with
542+
[`ATTACH DATABASE`][]. **Default:** `'main'`.
543+
* Returns: {Uint8Array} A binary representation of the database.
544+
545+
Serializes the database into a binary representation, returned as a
546+
`Uint8Array`. This is useful for saving, cloning, or transferring an in-memory
547+
database. This method is a wrapper around [`sqlite3_serialize()`][].
548+
549+
```mjs
550+
import { DatabaseSync } from 'node:sqlite';
551+
552+
const db = new DatabaseSync(':memory:');
553+
db.exec('CREATE TABLE t(key INTEGER PRIMARY KEY, value TEXT)');
554+
db.exec("INSERT INTO t VALUES (1, 'hello')");
555+
const buffer = db.serialize();
556+
console.log(buffer.length); // Prints the byte length of the database
557+
```
558+
559+
```cjs
560+
const { DatabaseSync } = require('node:sqlite');
561+
562+
const db = new DatabaseSync(':memory:');
563+
db.exec('CREATE TABLE t(key INTEGER PRIMARY KEY, value TEXT)');
564+
db.exec("INSERT INTO t VALUES (1, 'hello')");
565+
const buffer = db.serialize();
566+
console.log(buffer.length); // Prints the byte length of the database
567+
```
568+
569+
### `database.deserialize(buffer[, options])`
570+
571+
<!-- YAML
572+
added: REPLACEME
573+
-->
574+
575+
* `buffer` {Uint8Array} A binary representation of a database, such as the
576+
output of [`database.serialize()`][].
577+
* `options` {Object} Optional configuration for the deserialization.
578+
* `dbName` {string} Name of the database to deserialize into.
579+
**Default:** `'main'`.
580+
581+
Loads a serialized database into this connection, replacing the current
582+
database. The deserialized database is writable. Existing prepared statements
583+
are finalized before deserialization is attempted, even if the operation
584+
subsequently fails. This method is a wrapper around
585+
[`sqlite3_deserialize()`][].
586+
587+
```mjs
588+
import { DatabaseSync } from 'node:sqlite';
589+
590+
const original = new DatabaseSync(':memory:');
591+
original.exec('CREATE TABLE t(key INTEGER PRIMARY KEY, value TEXT)');
592+
original.exec("INSERT INTO t VALUES (1, 'hello')");
593+
const buffer = original.serialize();
594+
original.close();
595+
596+
const clone = new DatabaseSync(':memory:');
597+
clone.deserialize(buffer);
598+
console.log(clone.prepare('SELECT value FROM t').get());
599+
// Prints: { value: 'hello' }
600+
```
601+
602+
```cjs
603+
const { DatabaseSync } = require('node:sqlite');
604+
605+
const original = new DatabaseSync(':memory:');
606+
original.exec('CREATE TABLE t(key INTEGER PRIMARY KEY, value TEXT)');
607+
original.exec("INSERT INTO t VALUES (1, 'hello')");
608+
const buffer = original.serialize();
609+
original.close();
610+
611+
const clone = new DatabaseSync(':memory:');
612+
clone.deserialize(buffer);
613+
console.log(clone.prepare('SELECT value FROM t').get());
614+
// Prints: { value: 'hello' }
615+
```
616+
534617
### `database.prepare(sql[, options])`
535618

536619
<!-- YAML
@@ -1545,6 +1628,7 @@ callback function to indicate what type of operation is being authorized.
15451628
[`SQLTagStore`]: #class-sqltagstore
15461629
[`database.applyChangeset()`]: #databaseapplychangesetchangeset-options
15471630
[`database.createTagStore()`]: #databasecreatetagstoremaxsize
1631+
[`database.serialize()`]: #databaseserializedbname
15481632
[`database.setAuthorizer()`]: #databasesetauthorizercallback
15491633
[`sqlite3_backup_finish()`]: https://www.sqlite.org/c3ref/backup_finish.html#sqlite3backupfinish
15501634
[`sqlite3_backup_init()`]: https://www.sqlite.org/c3ref/backup_finish.html#sqlite3backupinit
@@ -1559,12 +1643,14 @@ callback function to indicate what type of operation is being authorized.
15591643
[`sqlite3_create_function_v2()`]: https://www.sqlite.org/c3ref/create_function.html
15601644
[`sqlite3_create_window_function()`]: https://www.sqlite.org/c3ref/create_function.html
15611645
[`sqlite3_db_filename()`]: https://sqlite.org/c3ref/db_filename.html
1646+
[`sqlite3_deserialize()`]: https://sqlite.org/c3ref/deserialize.html
15621647
[`sqlite3_exec()`]: https://www.sqlite.org/c3ref/exec.html
15631648
[`sqlite3_expanded_sql()`]: https://www.sqlite.org/c3ref/expanded_sql.html
15641649
[`sqlite3_get_autocommit()`]: https://sqlite.org/c3ref/get_autocommit.html
15651650
[`sqlite3_last_insert_rowid()`]: https://www.sqlite.org/c3ref/last_insert_rowid.html
15661651
[`sqlite3_load_extension()`]: https://www.sqlite.org/c3ref/load_extension.html
15671652
[`sqlite3_prepare_v2()`]: https://www.sqlite.org/c3ref/prepare.html
1653+
[`sqlite3_serialize()`]: https://sqlite.org/c3ref/serialize.html
15681654
[`sqlite3_set_authorizer()`]: https://sqlite.org/c3ref/set_authorizer.html
15691655
[`sqlite3_sql()`]: https://www.sqlite.org/c3ref/expanded_sql.html
15701656
[`sqlite3changeset_apply()`]: https://www.sqlite.org/session/sqlite3changeset_apply.html
Collapse file

‎src/node_sqlite.cc‎

Copy file name to clipboardExpand all lines: src/node_sqlite.cc
+133Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ namespace sqlite {
2323
using v8::Array;
2424
using v8::ArrayBuffer;
2525
using v8::BackingStoreInitializationMode;
26+
using v8::BackingStoreOnFailureMode;
2627
using v8::BigInt;
2728
using v8::Boolean;
2829
using v8::ConstructorBehavior;
@@ -1719,6 +1720,136 @@ void DatabaseSync::Location(const FunctionCallbackInfo<Value>& args) {
17191720
}
17201721
}
17211722

1723+
void DatabaseSync::Serialize(const FunctionCallbackInfo<Value>& args) {
1724+
DatabaseSync* db;
1725+
ASSIGN_OR_RETURN_UNWRAP(&db, args.This());
1726+
Environment* env = Environment::GetCurrent(args);
1727+
THROW_AND_RETURN_ON_BAD_STATE(env, !db->IsOpen(), "database is not open");
1728+
1729+
std::string db_name = "main";
1730+
if (!args[0]->IsUndefined()) {
1731+
if (!args[0]->IsString()) {
1732+
THROW_ERR_INVALID_ARG_TYPE(env->isolate(),
1733+
"The \"dbName\" argument must be a string.");
1734+
return;
1735+
}
1736+
db_name = Utf8Value(env->isolate(), args[0].As<String>()).ToString();
1737+
}
1738+
1739+
sqlite3_int64 size = 0;
1740+
unsigned char* data =
1741+
sqlite3_serialize(db->connection_, db_name.c_str(), &size, 0);
1742+
1743+
if (data == nullptr) {
1744+
if (size == 0) {
1745+
Local<ArrayBuffer> ab = ArrayBuffer::New(env->isolate(), 0);
1746+
args.GetReturnValue().Set(Uint8Array::New(ab, 0, 0));
1747+
return;
1748+
}
1749+
THROW_ERR_SQLITE_ERROR(env->isolate(), db);
1750+
return;
1751+
}
1752+
1753+
// V8 sandbox forbids external backing stores so allocate inside the
1754+
// sandbox and copy. Without sandbox wrap the output directly using
1755+
// sqlite3_free as the destructor to avoid the copy.
1756+
#ifdef V8_ENABLE_SANDBOX
1757+
auto free_data = OnScopeLeave([&] { sqlite3_free(data); });
1758+
auto store = ArrayBuffer::NewBackingStore(
1759+
env->isolate(),
1760+
size,
1761+
BackingStoreInitializationMode::kUninitialized,
1762+
BackingStoreOnFailureMode::kReturnNull);
1763+
if (!store) {
1764+
THROW_ERR_MEMORY_ALLOCATION_FAILED(env);
1765+
return;
1766+
}
1767+
memcpy(store->Data(), data, size);
1768+
Local<ArrayBuffer> ab = ArrayBuffer::New(env->isolate(), std::move(store));
1769+
#else
1770+
auto store = ArrayBuffer::NewBackingStore(
1771+
data, size, [](void* ptr, size_t, void*) { sqlite3_free(ptr); }, nullptr);
1772+
Local<ArrayBuffer> ab = ArrayBuffer::New(env->isolate(), std::move(store));
1773+
#endif
1774+
1775+
args.GetReturnValue().Set(Uint8Array::New(ab, 0, size));
1776+
}
1777+
1778+
void DatabaseSync::Deserialize(const FunctionCallbackInfo<Value>& args) {
1779+
DatabaseSync* db;
1780+
ASSIGN_OR_RETURN_UNWRAP(&db, args.This());
1781+
Environment* env = Environment::GetCurrent(args);
1782+
THROW_AND_RETURN_ON_BAD_STATE(env, !db->IsOpen(), "database is not open");
1783+
1784+
if (!args[0]->IsUint8Array()) {
1785+
THROW_ERR_INVALID_ARG_TYPE(env->isolate(),
1786+
"The \"buffer\" argument must be a Uint8Array.");
1787+
return;
1788+
}
1789+
1790+
Local<Uint8Array> input = args[0].As<Uint8Array>();
1791+
size_t byte_length = input->ByteLength();
1792+
1793+
if (byte_length == 0) {
1794+
THROW_ERR_INVALID_ARG_VALUE(env,
1795+
"The \"buffer\" argument must not be empty.");
1796+
return;
1797+
}
1798+
1799+
std::string db_name = "main";
1800+
if (args.Length() > 1 && !args[1]->IsUndefined()) {
1801+
if (!args[1]->IsObject()) {
1802+
THROW_ERR_INVALID_ARG_TYPE(env->isolate(),
1803+
"The \"options\" argument must be an object.");
1804+
return;
1805+
}
1806+
1807+
Local<Object> options = args[1].As<Object>();
1808+
Local<String> db_name_key = FIXED_ONE_BYTE_STRING(env->isolate(), "dbName");
1809+
Local<Value> db_name_value;
1810+
if (!options->Get(env->context(), db_name_key).ToLocal(&db_name_value)) {
1811+
return;
1812+
}
1813+
1814+
if (!db_name_value->IsUndefined()) {
1815+
if (!db_name_value->IsString()) {
1816+
THROW_ERR_INVALID_ARG_TYPE(
1817+
env->isolate(),
1818+
"The \"options.dbName\" argument must be a string.");
1819+
return;
1820+
}
1821+
db_name =
1822+
Utf8Value(env->isolate(), db_name_value.As<String>()).ToString();
1823+
}
1824+
}
1825+
1826+
// sqlite3_malloc64 is required because SQLITE_DESERIALIZE_FREEONCLOSE
1827+
// transfers ownership to SQLite, which calls sqlite3_free() on close.
1828+
// See: https://www.sqlite.org/c3ref/deserialize.html
1829+
unsigned char* buf =
1830+
static_cast<unsigned char*>(sqlite3_malloc64(byte_length));
1831+
if (buf == nullptr) {
1832+
THROW_ERR_MEMORY_ALLOCATION_FAILED(env);
1833+
return;
1834+
}
1835+
1836+
input->CopyContents(buf, byte_length);
1837+
1838+
db->FinalizeStatements();
1839+
1840+
int r = sqlite3_deserialize(
1841+
db->connection_,
1842+
db_name.c_str(),
1843+
buf,
1844+
byte_length,
1845+
byte_length,
1846+
SQLITE_DESERIALIZE_FREEONCLOSE | SQLITE_DESERIALIZE_RESIZEABLE);
1847+
if (r != SQLITE_OK) {
1848+
THROW_ERR_SQLITE_ERROR(env->isolate(), db);
1849+
return;
1850+
}
1851+
}
1852+
17221853
void DatabaseSync::AggregateFunction(const FunctionCallbackInfo<Value>& args) {
17231854
DatabaseSync* db;
17241855
ASSIGN_OR_RETURN_UNWRAP(&db, args.This());
@@ -3766,6 +3897,8 @@ static void Initialize(Local<Object> target,
37663897
isolate, db_tmpl, "enableDefensive", DatabaseSync::EnableDefensive);
37673898
SetProtoMethod(
37683899
isolate, db_tmpl, "loadExtension", DatabaseSync::LoadExtension);
3900+
SetProtoMethod(isolate, db_tmpl, "serialize", DatabaseSync::Serialize);
3901+
SetProtoMethod(isolate, db_tmpl, "deserialize", DatabaseSync::Deserialize);
37693902
SetProtoMethod(
37703903
isolate, db_tmpl, "setAuthorizer", DatabaseSync::SetAuthorizer);
37713904
SetSideEffectFreeGetter(isolate,
Collapse file

‎src/node_sqlite.h‎

Copy file name to clipboardExpand all lines: src/node_sqlite.h
+2Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -195,6 +195,8 @@ class DatabaseSync : public BaseObject {
195195
static void EnableDefensive(const v8::FunctionCallbackInfo<v8::Value>& args);
196196
static void LimitsGetter(const v8::FunctionCallbackInfo<v8::Value>& args);
197197
static void LoadExtension(const v8::FunctionCallbackInfo<v8::Value>& args);
198+
static void Serialize(const v8::FunctionCallbackInfo<v8::Value>& args);
199+
static void Deserialize(const v8::FunctionCallbackInfo<v8::Value>& args);
198200
static void SetAuthorizer(const v8::FunctionCallbackInfo<v8::Value>& args);
199201
static int AuthorizerCallback(void* user_data,
200202
int action_code,

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.