Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 3c96ae1

Browse filesBrowse files
thisalihassanaduh95
thisalihassan
authored and
aduh95
committed
sqlite: add serialize() and deserialize()
Add database.serialize() and database.deserialize() methods to DatabaseSync, wrapping the sqlite3_serialize and sqlite3_deserialize C APIs. These allow extracting an in-memory database as a Uint8Array and loading one back, enabling snapshots, cloning, and transfer of databases without filesystem I/O. Refs: #62575 PR-URL: #62579 Refs: #62575 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Edy Silva <edigleyssonsilva@gmail.com>
1 parent 3ed7835 commit 3c96ae1
Copy full SHA for 3c96ae1

4 files changed

+526Lines changed: 526 additions & 0 deletions

File tree

Expand file treeCollapse file tree
Open diff view settings
Filter options
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎doc/api/sqlite.md‎

Copy file name to clipboardExpand all lines: doc/api/sqlite.md
+86Lines changed: 86 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -517,6 +517,89 @@ Opens the database specified in the `path` argument of the `DatabaseSync`
517517
constructor. This method should only be used when the database is not opened via
518518
the constructor. An exception is thrown if the database is already open.
519519

520+
### `database.serialize([dbName])`
521+
522+
<!-- YAML
523+
added: REPLACEME
524+
-->
525+
526+
* `dbName` {string} Name of the database to serialize. This can be `'main'`
527+
(the default primary database) or any other database that has been added with
528+
[`ATTACH DATABASE`][]. **Default:** `'main'`.
529+
* Returns: {Uint8Array} A binary representation of the database.
530+
531+
Serializes the database into a binary representation, returned as a
532+
`Uint8Array`. This is useful for saving, cloning, or transferring an in-memory
533+
database. This method is a wrapper around [`sqlite3_serialize()`][].
534+
535+
```mjs
536+
import { DatabaseSync } from 'node:sqlite';
537+
538+
const db = new DatabaseSync(':memory:');
539+
db.exec('CREATE TABLE t(key INTEGER PRIMARY KEY, value TEXT)');
540+
db.exec("INSERT INTO t VALUES (1, 'hello')");
541+
const buffer = db.serialize();
542+
console.log(buffer.length); // Prints the byte length of the database
543+
```
544+
545+
```cjs
546+
const { DatabaseSync } = require('node:sqlite');
547+
548+
const db = new DatabaseSync(':memory:');
549+
db.exec('CREATE TABLE t(key INTEGER PRIMARY KEY, value TEXT)');
550+
db.exec("INSERT INTO t VALUES (1, 'hello')");
551+
const buffer = db.serialize();
552+
console.log(buffer.length); // Prints the byte length of the database
553+
```
554+
555+
### `database.deserialize(buffer[, options])`
556+
557+
<!-- YAML
558+
added: REPLACEME
559+
-->
560+
561+
* `buffer` {Uint8Array} A binary representation of a database, such as the
562+
output of [`database.serialize()`][].
563+
* `options` {Object} Optional configuration for the deserialization.
564+
* `dbName` {string} Name of the database to deserialize into.
565+
**Default:** `'main'`.
566+
567+
Loads a serialized database into this connection, replacing the current
568+
database. The deserialized database is writable. Existing prepared statements
569+
are finalized before deserialization is attempted, even if the operation
570+
subsequently fails. This method is a wrapper around
571+
[`sqlite3_deserialize()`][].
572+
573+
```mjs
574+
import { DatabaseSync } from 'node:sqlite';
575+
576+
const original = new DatabaseSync(':memory:');
577+
original.exec('CREATE TABLE t(key INTEGER PRIMARY KEY, value TEXT)');
578+
original.exec("INSERT INTO t VALUES (1, 'hello')");
579+
const buffer = original.serialize();
580+
original.close();
581+
582+
const clone = new DatabaseSync(':memory:');
583+
clone.deserialize(buffer);
584+
console.log(clone.prepare('SELECT value FROM t').get());
585+
// Prints: { value: 'hello' }
586+
```
587+
588+
```cjs
589+
const { DatabaseSync } = require('node:sqlite');
590+
591+
const original = new DatabaseSync(':memory:');
592+
original.exec('CREATE TABLE t(key INTEGER PRIMARY KEY, value TEXT)');
593+
original.exec("INSERT INTO t VALUES (1, 'hello')");
594+
const buffer = original.serialize();
595+
original.close();
596+
597+
const clone = new DatabaseSync(':memory:');
598+
clone.deserialize(buffer);
599+
console.log(clone.prepare('SELECT value FROM t').get());
600+
// Prints: { value: 'hello' }
601+
```
602+
520603
### `database.prepare(sql[, options])`
521604

522605
<!-- YAML
@@ -1523,6 +1606,7 @@ callback function to indicate what type of operation is being authorized.
15231606
[`SQLTagStore`]: #class-sqltagstore
15241607
[`database.applyChangeset()`]: #databaseapplychangesetchangeset-options
15251608
[`database.createTagStore()`]: #databasecreatetagstoremaxsize
1609+
[`database.serialize()`]: #databaseserializedbname
15261610
[`database.setAuthorizer()`]: #databasesetauthorizercallback
15271611
[`sqlite3_backup_finish()`]: https://www.sqlite.org/c3ref/backup_finish.html#sqlite3backupfinish
15281612
[`sqlite3_backup_init()`]: https://www.sqlite.org/c3ref/backup_finish.html#sqlite3backupinit
@@ -1537,12 +1621,14 @@ callback function to indicate what type of operation is being authorized.
15371621
[`sqlite3_create_function_v2()`]: https://www.sqlite.org/c3ref/create_function.html
15381622
[`sqlite3_create_window_function()`]: https://www.sqlite.org/c3ref/create_function.html
15391623
[`sqlite3_db_filename()`]: https://sqlite.org/c3ref/db_filename.html
1624+
[`sqlite3_deserialize()`]: https://sqlite.org/c3ref/deserialize.html
15401625
[`sqlite3_exec()`]: https://www.sqlite.org/c3ref/exec.html
15411626
[`sqlite3_expanded_sql()`]: https://www.sqlite.org/c3ref/expanded_sql.html
15421627
[`sqlite3_get_autocommit()`]: https://sqlite.org/c3ref/get_autocommit.html
15431628
[`sqlite3_last_insert_rowid()`]: https://www.sqlite.org/c3ref/last_insert_rowid.html
15441629
[`sqlite3_load_extension()`]: https://www.sqlite.org/c3ref/load_extension.html
15451630
[`sqlite3_prepare_v2()`]: https://www.sqlite.org/c3ref/prepare.html
1631+
[`sqlite3_serialize()`]: https://sqlite.org/c3ref/serialize.html
15461632
[`sqlite3_set_authorizer()`]: https://sqlite.org/c3ref/set_authorizer.html
15471633
[`sqlite3_sql()`]: https://www.sqlite.org/c3ref/expanded_sql.html
15481634
[`sqlite3changeset_apply()`]: https://www.sqlite.org/session/sqlite3changeset_apply.html
Collapse file

‎src/node_sqlite.cc‎

Copy file name to clipboardExpand all lines: src/node_sqlite.cc
+133Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ namespace sqlite {
2323
using v8::Array;
2424
using v8::ArrayBuffer;
2525
using v8::BackingStoreInitializationMode;
26+
using v8::BackingStoreOnFailureMode;
2627
using v8::BigInt;
2728
using v8::Boolean;
2829
using v8::ConstructorBehavior;
@@ -1719,6 +1720,136 @@ void DatabaseSync::Location(const FunctionCallbackInfo<Value>& args) {
17191720
}
17201721
}
17211722

1723+
void DatabaseSync::Serialize(const FunctionCallbackInfo<Value>& args) {
1724+
DatabaseSync* db;
1725+
ASSIGN_OR_RETURN_UNWRAP(&db, args.This());
1726+
Environment* env = Environment::GetCurrent(args);
1727+
THROW_AND_RETURN_ON_BAD_STATE(env, !db->IsOpen(), "database is not open");
1728+
1729+
std::string db_name = "main";
1730+
if (!args[0]->IsUndefined()) {
1731+
if (!args[0]->IsString()) {
1732+
THROW_ERR_INVALID_ARG_TYPE(env->isolate(),
1733+
"The \"dbName\" argument must be a string.");
1734+
return;
1735+
}
1736+
db_name = Utf8Value(env->isolate(), args[0].As<String>()).ToString();
1737+
}
1738+
1739+
sqlite3_int64 size = 0;
1740+
unsigned char* data =
1741+
sqlite3_serialize(db->connection_, db_name.c_str(), &size, 0);
1742+
1743+
if (data == nullptr) {
1744+
if (size == 0) {
1745+
Local<ArrayBuffer> ab = ArrayBuffer::New(env->isolate(), 0);
1746+
args.GetReturnValue().Set(Uint8Array::New(ab, 0, 0));
1747+
return;
1748+
}
1749+
THROW_ERR_SQLITE_ERROR(env->isolate(), db);
1750+
return;
1751+
}
1752+
1753+
// V8 sandbox forbids external backing stores so allocate inside the
1754+
// sandbox and copy. Without sandbox wrap the output directly using
1755+
// sqlite3_free as the destructor to avoid the copy.
1756+
#ifdef V8_ENABLE_SANDBOX
1757+
auto free_data = OnScopeLeave([&] { sqlite3_free(data); });
1758+
auto store = ArrayBuffer::NewBackingStore(
1759+
env->isolate(),
1760+
size,
1761+
BackingStoreInitializationMode::kUninitialized,
1762+
BackingStoreOnFailureMode::kReturnNull);
1763+
if (!store) {
1764+
THROW_ERR_MEMORY_ALLOCATION_FAILED(env);
1765+
return;
1766+
}
1767+
memcpy(store->Data(), data, size);
1768+
Local<ArrayBuffer> ab = ArrayBuffer::New(env->isolate(), std::move(store));
1769+
#else
1770+
auto store = ArrayBuffer::NewBackingStore(
1771+
data, size, [](void* ptr, size_t, void*) { sqlite3_free(ptr); }, nullptr);
1772+
Local<ArrayBuffer> ab = ArrayBuffer::New(env->isolate(), std::move(store));
1773+
#endif
1774+
1775+
args.GetReturnValue().Set(Uint8Array::New(ab, 0, size));
1776+
}
1777+
1778+
void DatabaseSync::Deserialize(const FunctionCallbackInfo<Value>& args) {
1779+
DatabaseSync* db;
1780+
ASSIGN_OR_RETURN_UNWRAP(&db, args.This());
1781+
Environment* env = Environment::GetCurrent(args);
1782+
THROW_AND_RETURN_ON_BAD_STATE(env, !db->IsOpen(), "database is not open");
1783+
1784+
if (!args[0]->IsUint8Array()) {
1785+
THROW_ERR_INVALID_ARG_TYPE(env->isolate(),
1786+
"The \"buffer\" argument must be a Uint8Array.");
1787+
return;
1788+
}
1789+
1790+
Local<Uint8Array> input = args[0].As<Uint8Array>();
1791+
size_t byte_length = input->ByteLength();
1792+
1793+
if (byte_length == 0) {
1794+
THROW_ERR_INVALID_ARG_VALUE(env,
1795+
"The \"buffer\" argument must not be empty.");
1796+
return;
1797+
}
1798+
1799+
std::string db_name = "main";
1800+
if (args.Length() > 1 && !args[1]->IsUndefined()) {
1801+
if (!args[1]->IsObject()) {
1802+
THROW_ERR_INVALID_ARG_TYPE(env->isolate(),
1803+
"The \"options\" argument must be an object.");
1804+
return;
1805+
}
1806+
1807+
Local<Object> options = args[1].As<Object>();
1808+
Local<String> db_name_key = FIXED_ONE_BYTE_STRING(env->isolate(), "dbName");
1809+
Local<Value> db_name_value;
1810+
if (!options->Get(env->context(), db_name_key).ToLocal(&db_name_value)) {
1811+
return;
1812+
}
1813+
1814+
if (!db_name_value->IsUndefined()) {
1815+
if (!db_name_value->IsString()) {
1816+
THROW_ERR_INVALID_ARG_TYPE(
1817+
env->isolate(),
1818+
"The \"options.dbName\" argument must be a string.");
1819+
return;
1820+
}
1821+
db_name =
1822+
Utf8Value(env->isolate(), db_name_value.As<String>()).ToString();
1823+
}
1824+
}
1825+
1826+
// sqlite3_malloc64 is required because SQLITE_DESERIALIZE_FREEONCLOSE
1827+
// transfers ownership to SQLite, which calls sqlite3_free() on close.
1828+
// See: https://www.sqlite.org/c3ref/deserialize.html
1829+
unsigned char* buf =
1830+
static_cast<unsigned char*>(sqlite3_malloc64(byte_length));
1831+
if (buf == nullptr) {
1832+
THROW_ERR_MEMORY_ALLOCATION_FAILED(env);
1833+
return;
1834+
}
1835+
1836+
input->CopyContents(buf, byte_length);
1837+
1838+
db->FinalizeStatements();
1839+
1840+
int r = sqlite3_deserialize(
1841+
db->connection_,
1842+
db_name.c_str(),
1843+
buf,
1844+
byte_length,
1845+
byte_length,
1846+
SQLITE_DESERIALIZE_FREEONCLOSE | SQLITE_DESERIALIZE_RESIZEABLE);
1847+
if (r != SQLITE_OK) {
1848+
THROW_ERR_SQLITE_ERROR(env->isolate(), db);
1849+
return;
1850+
}
1851+
}
1852+
17221853
void DatabaseSync::AggregateFunction(const FunctionCallbackInfo<Value>& args) {
17231854
DatabaseSync* db;
17241855
ASSIGN_OR_RETURN_UNWRAP(&db, args.This());
@@ -3766,6 +3897,8 @@ static void Initialize(Local<Object> target,
37663897
isolate, db_tmpl, "enableDefensive", DatabaseSync::EnableDefensive);
37673898
SetProtoMethod(
37683899
isolate, db_tmpl, "loadExtension", DatabaseSync::LoadExtension);
3900+
SetProtoMethod(isolate, db_tmpl, "serialize", DatabaseSync::Serialize);
3901+
SetProtoMethod(isolate, db_tmpl, "deserialize", DatabaseSync::Deserialize);
37693902
SetProtoMethod(
37703903
isolate, db_tmpl, "setAuthorizer", DatabaseSync::SetAuthorizer);
37713904
SetSideEffectFreeGetter(isolate,
Collapse file

‎src/node_sqlite.h‎

Copy file name to clipboardExpand all lines: src/node_sqlite.h
+2Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -195,6 +195,8 @@ class DatabaseSync : public BaseObject {
195195
static void EnableDefensive(const v8::FunctionCallbackInfo<v8::Value>& args);
196196
static void LimitsGetter(const v8::FunctionCallbackInfo<v8::Value>& args);
197197
static void LoadExtension(const v8::FunctionCallbackInfo<v8::Value>& args);
198+
static void Serialize(const v8::FunctionCallbackInfo<v8::Value>& args);
199+
static void Deserialize(const v8::FunctionCallbackInfo<v8::Value>& args);
198200
static void SetAuthorizer(const v8::FunctionCallbackInfo<v8::Value>& args);
199201
static int AuthorizerCallback(void* user_data,
200202
int action_code,

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.