Description
Hi :)
First wanted to thank you, this project is amazing.
We have a polyglot project — valkey-glide.
We use Rust core logic and bind it to various languages to create client-specific implementations.
In the Node.js binding, we use napi-rs for the connection layer.
However, our project differs from the examples I found.
It’s a TypeScript written Node.js project that depends on Rust core.
Instead of publishing a Rust fully built with napi-rs, we build an API package from our core using napi, which is a separate module.
We then import it into TypeScript code, re-expose parts of the API, and publish the package.
Valkey GLIDE Architecture — High-Level Overview
graph TD
%% Main high-level components
User["Application Code"] --> NodePackage["Node.js Package"]
NodePackage --> RustBindings["Rust Bindings Layer"]
RustBindings --> CoreEngine["GLIDE Core Engine"]
%% Main component breakdown
subgraph "Client Layer"
NodePackage
subgraph "TypeScript API"
TS_API["Client API Surface"]
TS_Commands["Commands & Features"]
end
end
subgraph "Bridge Layer"
RustBindings
NAPI["Node-API (N-API)"]
end
subgraph "Core Implementation"
CoreEngine
Protocol["RESP Protocol"]
ConnectionMgmt["Connection Management"]
Clustering["Cluster Support"]
end
%% Main connections
User --> |"import from @valkey/valkey-glide"| TS_API
TS_API --> |"calls"| NAPI
NAPI --> |"invokes"| CoreEngine
%% Cross-dependencies
Logger["Logging System"] --> RustBindings
Logger --> CoreEngine
So, basically, We have a node package that uses a Rust package.
The lower part and logging system are pure Rust.
The middle is napi code that triggers the connection, connects the logger, and passes pointers occasionally.
A UDS (Rust binding layer) connection to the node code is used for interaction.
UDS performs faster in most scenarios, so it’s the current protocol. I plan to retest and hopefully refactor to full FFI in another round.
Currently, the structure is an empty package with index.ts and separate Rust and TS code in optionals. Inside the full package, the separation is to two packages when TS is dependent on napi-rs built code using file dependencies.
This creates issues.
Yarn doesn’t understand when downloading a new package that the file dependency is compared to the package location, not the root location. A workaround is to create an empty rust-client directory in the root directory for installation.
We need to reexport everything to have types in the user-facing empty package by literary exporting each type in the index file. We built a test to analyze that all public types are available in the index file.
NPM started respecting the libc tag after 11, so on Linux, you get two packages (musl and gnu) and twice the JS code. This leads to the most painful part.
Packing for serveless is a nightmare.
You don’t want the TS code not packed, but you do want to extract the .node from the packing.
Both being part of the same package makes it hard, especially when you have two packages.
One needs to be split into two, and one completely removed.
This isn’t good UX or fun to develop.
My goal is to change it to one TS package and one Rust package
(napi-rs).
How can I import the rust package using optionals without a dummy middle layer?
How can I re-export napi-rs types to TS types code that’s exposed to end users, without copying and pasting after every build or using another static analysis tool?
Lastly, how can I work locally when no optionals exist and need a devo override to a local directory?
Sorry for the length and thanks a lot!