Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Q: General questions / COVID-19 sanity checking #150

Copy link
Copy link
@ghudo

Description

@ghudo
Issue body actions

Hi @benalexau - you've been so helpful... I was wondering if you could sanity check the below... if not, or if there's a better forum for these types of questions please let me know!

LMDB really does seem to be best-in-class for read focused performance - I'm surprised there's not more people using/discussing it... it's made it challenging to get as far as I've got!

Background: I've been using LMDB for a while with a lot of success - all my keys are incrementing longs (put with MDB_APPEND flag) and my values are binary encoded, compressed (using LZ4-fast) with checksums (using xxHash64).

My questions;

  1. I've been using LMDB a bit like a relational database and creating a separate LMDB file/Env for each entity (i.e. think SQL table). This works very well, but there are rare cases where I have to open transaction open across multiple LMDB file/Env's.

  2. I group things like foreign-key relationships into the same LMDB file/Env as a separate named database to avoid having to always open multiple read or write transactions.

  3. There are cases where I want to partition information - so I have a separate LMDB file/Env for each calendar month or complex user specific databases - so I keep only the most commonly/recently used LMDB file/Env's open to limit the number of environments I have open at a time.

  4. There are rare (and performance critical) cases where I want to upgrade from a read lock to a write lock. In these cases, I first release the read lock, then open the write lock and re-read everything again with this write lock. I've noted from What does Environment maxreaders reached (-30790) actually mean? #65 (comment) that you prefer JVM-based locking. I thought I might get performance gains by using JVM-based locking and setting EnvFlags.MDB_NOLOCK - but didn't... so gave up.

Is there anything obviously wrong with any of the above? Any recommendations?

Thanks in advance ;-)

Reactions are currently unavailable

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionQueries on usage of the library, architectural patterns and alikeQueries on usage of the library, architectural patterns and alike

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      Morty Proxy This is a proxified and sanitized view of the page, visit original site.