Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Poliglot project - a package for a package. Question. #2570

Copy link
Copy link
Closed
@avifenesh

Description

@avifenesh
Issue body actions

Hi :)
First wanted to thank you, this project is amazing.

We have a polyglot project — valkey-glide.
We use Rust core logic and bind it to various languages to create client-specific implementations.

In the Node.js binding, we use napi-rs for the connection layer.

However, our project differs from the examples I found.
It’s a TypeScript written Node.js project that depends on Rust core.
Instead of publishing a Rust fully built with napi-rs, we build an API package from our core using napi, which is a separate module.
We then import it into TypeScript code, re-expose parts of the API, and publish the package.

Valkey GLIDE Architecture — High-Level Overview

graph TD
    %% Main high-level components
    User["Application Code"] --> NodePackage["Node.js Package"]
    NodePackage --> RustBindings["Rust Bindings Layer"]
    RustBindings --> CoreEngine["GLIDE Core Engine"]
    
    %% Main component breakdown
    subgraph "Client Layer"
        NodePackage
        
        subgraph "TypeScript API"
            TS_API["Client API Surface"]
            TS_Commands["Commands & Features"]
        end
    end
    
    subgraph "Bridge Layer"
        RustBindings
        NAPI["Node-API (N-API)"]
    end
    
    subgraph "Core Implementation"
        CoreEngine
        Protocol["RESP Protocol"]
        ConnectionMgmt["Connection Management"]
        Clustering["Cluster Support"]
    end
    
    %% Main connections
    User --> |"import from @valkey/valkey-glide"| TS_API
    TS_API --> |"calls"| NAPI
    NAPI --> |"invokes"| CoreEngine
    
    %% Cross-dependencies
    Logger["Logging System"] --> RustBindings
    Logger --> CoreEngine
Loading

So, basically, We have a node package that uses a Rust package.
The lower part and logging system are pure Rust.
The middle is napi code that triggers the connection, connects the logger, and passes pointers occasionally.
A UDS (Rust binding layer) connection to the node code is used for interaction.
UDS performs faster in most scenarios, so it’s the current protocol. I plan to retest and hopefully refactor to full FFI in another round.

Currently, the structure is an empty package with index.ts and separate Rust and TS code in optionals. Inside the full package, the separation is to two packages when TS is dependent on napi-rs built code using file dependencies.

This creates issues.
Yarn doesn’t understand when downloading a new package that the file dependency is compared to the package location, not the root location. A workaround is to create an empty rust-client directory in the root directory for installation.
We need to reexport everything to have types in the user-facing empty package by literary exporting each type in the index file. We built a test to analyze that all public types are available in the index file.
NPM started respecting the libc tag after 11, so on Linux, you get two packages (musl and gnu) and twice the JS code. This leads to the most painful part.
Packing for serveless is a nightmare.
You don’t want the TS code not packed, but you do want to extract the .node from the packing.
Both being part of the same package makes it hard, especially when you have two packages.
One needs to be split into two, and one completely removed.

This isn’t good UX or fun to develop.

My goal is to change it to one TS package and one Rust package
(napi-rs).

How can I import the rust package using optionals without a dummy middle layer?
How can I re-export napi-rs types to TS types code that’s exposed to end users, without copying and pasting after every build or using another static analysis tool?
Lastly, how can I work locally when no optionals exist and need a devo override to a local directory?

Sorry for the length and thanks a lot!

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    Morty Proxy This is a proxified and sanitized view of the page, visit original site.