Introduce standalone C parser for RBS with arena allocation #2398

st0012 · Apr 10, 2025

RBS C Parser Library Refactoring

This PR refactors the RBS parser and related components into a standalone C library that no longer depends on the Ruby runtime. This architectural change enables direct integration with static analysis tools like Sorbet while potentially improving performance.

Sorbet's RBS support already runs on this new architecture and we haven't discovered any major issues around it.

This work was a collaborative effort by

Key Improvements

Ruby-Independent Implementation: Extracted parser from ext folder into a standalone C library with a clean API, which can now be embedded in non-Ruby tools without Ruby runtime dependency (e.g. Sorbet, JRuby)
Enhanced Memory Management: Implemented arena allocator to efficiently manage parser object lifecycles
Improved Architecture: Clear separation between public API (headers) and implementation
Performance: Potential performance gains from custom memory management and reduced overhead

Enhanced Memory Management

Arena allocator handles all memory for parser objects, including parser itself, lexer, constant pool, strings...etc. When the parser is freed by calling rbs_parser_free, the allocator will free all the objects it allocated. This eliminates the need to manually free individual objects and reduces the risk of memory leaks.

Component Architecture

graph TD
    RubyClient[Ruby Client] --> RubyAPI[Ruby API]
    CClient[C Client] --> CAPI[C API]

    RubyAPI --> CExtension[C Extension]
    CExtension --> CLibrary
    CAPI --> CLibrary

    subgraph CLibrary[C Library]
        subgraph Parser1[Parser Instance 1]
            direction TB
            ConstantPool1[Constant Pool]
            Lexer1[Lexer]
            ArenaAllocator1[Arena Allocator]
        end

        subgraph Parser2[Parser Instance 2]
            direction TB
            ConstantPool2[Constant Pool]
            Lexer2[Lexer]
            ArenaAllocator2[Arena Allocator]
        end
    end

    subgraph "Public API"
        RubyAPI
        CAPI
    end

    %% Parser1 --> ConstantPool1
    %% Parser1 --> Lexer1
    %% Parser1 --> ArenaAllocator1
    %% Parser2 --> ConstantPool2
    %% Parser2 --> Lexer2
    %% Parser2 --> ArenaAllocator2

Initial template for C structs Use allocator in node constructors

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Add linked list implementation Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Type `Class#super_class` field Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Type fields of `RBS::Types::Block` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Type `block` fields Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Type `RBS::Types::Proc#self_type` field Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Refactor `parse_function` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Copy value in `rbs_struct_to_ruby_value` Remove usages of `rbs_loc` from `parser.c` Extract `rbs_location.h` Migrate `RBS::Types::Function::Param` fields Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Type `RBS::Types::UntypedFunction` fields Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Type fields of `RBS::AST::TypeParam` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Type some more fields of `RBS::AST::Members::Attr` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Type fields in `RBS::AST::Members::MethodDefinition` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Type `RBS::AST::Directives::Use::SingleClause#new_name` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Type `RBS::Namespace#absolute` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Temporary handle nil types Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Handle `bool` type Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Type all fields of `RBS::Types::Variable` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Migrate `RBS::TypeName` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Migrate `parse_use_clauses` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Migrate `class_instance_name` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Handle overloads as a rbs_node_list Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Remove more `builds_ruby_object_internally` flags Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Invert `builds_ruby_object_internally` default value Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Introduce `rbs_location_t` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Store C structs instead of Ruby `VALUE`s Introduce +rbs_ast_symbol_t and migrate to it Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Remove ZzzTmpNotImplemented node Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Remove one more instance of EMPTY_ARRAY Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Migrate from VALUE array to rbs_node_list_t Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Migrate `method_params` from taking a VALUE arrays Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Migrate `parse_type_list` from taking a VALUE array Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Forward all C-typed params as-is Get types on constructor params Handle mix of C types and Ruby VALUE Move Ruby object construction into `new` functions Conditionally construct `ruby_value` internally Type Attr* field `ivar_name` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Add `AST::Bool` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Use two less VALUE values Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Use more instance of `bool` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Add Hash implementation Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Use C hash for `check_key_duplication` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Use C hash to represent Record fields Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Migrate `memo` to using a C hash Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Uses C hashes for keyword parameters Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Remove parser call to `todo!` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Remove calls to `rbs_struct_to_ruby_value` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> TMP symbol Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Replace 2 fake nodes by one Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Set fields for `Record::FieldType` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Make comment use a `rbs_ast_comment_t` instead of a `VALUE` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Add `rbs_ast_string_t` Add `rbs_ast_integer_t` Migrate `literal` to store C nodes Remove `cached_ruby_string` Remove useless templating stuff Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Remove `cached_ruby_value` from `rbs_node_list` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Remove `cached_ruby_value` from `rbs_hash` Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Add `rbs_string`, and use it for annotations Add `rbs_ast_symbol_t` to model symbols in the AST Co-Authored-By: Alexander Momchilov <alexander.momchilov@shopify.com>

And rename it to `class_constants` to disambiguate it from `rbs_constant_id`, `rbs_constant_pool`, etc.

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Do not create comments using a VALUE Use a rbs_string instead Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

.idea/.gitignore

.vscode/launch.json

demo.rb

make_signed_ruby.rb

sync_from_prism.rb

trace_rbs.sh

`rbs_node_destroy`, `rbs_hash_free`, `rbs_node_list_free` are only calling each other recursively without any real freeing logic. This is the result of previous efforts to allocate all nodes on the arena. So we don't need these functions anymore. Discovered while working on #41

Co-authored-by: Alexander Momchilov <amomchilov@users.noreply.github.com>

soutaro · May 1, 2025

Thank you all for this contribution! I really appreciate all of you. 👏
The code generally looks great, but to confirm everything works I would like to implement another parsing function by myself. I will try it tomorrow. 🙏

soutaro

I tried to add a new inline annotation at #2443, and confirmed that I feel I can work on this codebase. 👍

soutaro · May 2, 2025

Opened ruby/ruby#13237 to test if the new C code works with Ruby CI compilers. (I should have done the test before merging... 💦 )

Introduce standalone C parser for RBS with arena allocation

amomchilov and others added 30 commits February 25, 2025 18:14

Allocate lexer inside alloc_parser()

bcf8aa2

Add new arena allocator

5f8e833

Initial template for C structs Use allocator in node constructors

Enable unused-result warnings

a51d08c

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Do not test inner type for shitlist

6fcb89e

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Move constants.c/.h to rbs_extension/

f6753f7

And rename it to `class_constants` to disambiguate it from `rbs_constant_id`, `rbs_constant_pool`, etc.

Replace rbs_abort

7110ba4

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Do not raise for syntax errors

b396faf

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Make parse_type_params return a NODISCARD bool

1ab4c32

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Rename syntax_error to set_syntax_error

62e7e79

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Merge aborted and syntax_error

57f9a5e

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Introduce rbs_buffer

0fab17e

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com> Do not create comments using a VALUE Use a rbs_string instead Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Move extension parsing functions out of parser.c

547bb0b

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Remove a few more ruby.h

eecf824

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Use a bare char * for type_name error

ae31edf

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Add rbs_translation_context_t to rbs_loc_to_ruby_location

52f201e

Set locations’ buffer in translation layer

2a67c96

Remove cached_ruby_object from rbs_location_t

b239f86

Dedupe rbs_loc_add_required_child body

3ca4117

misc cleanup

2f136c5

Migrate remaining uses of rbs_unquote_string

aad46c1

Apply rbs_parser_get_current_token()

c998ecf

Delete rbs_other_ruby_value_t

7c72570

Import encoding handling from Prism

869a1a2

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Store encoding in the lexer state

0984a4a

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Migrate lexerstate string to rbs_string

08b4a7a

Remove useless includes

efa6646

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Remove last ruby/encoding dependencies from parser.h

89b4e4b

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Move lexer out of the extension directory

c87a436

Signed-off-by: Alexandre Terrasa <alexandre.terrasa@shopify.com>

Merge branch 'master' into c-api

da91a88

st0012 commented Apr 10, 2025

View reviewed changes

st0012 added 3 commits April 25, 2025 16:35

Merge branch 'master' into merge-master

afc0f98

Merge branch 'c-api' into merge-master

b5e5032

Morriar mentioned this pull request Apr 25, 2025

[Epic] Pure C API #2133

Open

7 tasks

Remove files that won't be upstreamed (#42)

563e081

st0012 mentioned this pull request Apr 30, 2025

Setup clang format for C code #2437

Merged

st0012 and others added 4 commits April 30, 2025 21:23

Apply suggestions from code review

0a8a4c5

Co-authored-by: Alexander Momchilov <amomchilov@users.noreply.github.com>

Update src/parser.c

a9e7702

Co-authored-by: Alexander Momchilov <amomchilov@users.noreply.github.com>

Update src/parser.c

514745b

Merge branch 'merge-master' into c-api

9e6b6ae

st0012 force-pushed the c-api branch from 21752fb to 9e6b6ae Compare April 30, 2025 14:47

st0012 changed the title ~~[Feedback Wanted] Introduce standalone C parser for RBS with arena allocation~~ Introduce standalone C parser for RBS with arena allocation Apr 30, 2025

st0012 marked this pull request as ready for review April 30, 2025 14:48

soutaro added this to the RBS 4.0 milestone May 1, 2025

Merge branch 'master' into c-api

f032275

soutaro approved these changes May 2, 2025

View reviewed changes

soutaro added this pull request to the merge queue May 2, 2025

Merged via the queue into ruby:master with commit 4ecd51a May 2, 2025
20 checks passed

soutaro mentioned this pull request May 2, 2025

Test with updated RBS gem which comes with new pure C parser ruby/ruby#13237

Draft

vinistock mentioned this pull request Jun 17, 2025

jruby installation broken since 0.17.3 Shopify/ruby-lsp#2292

Open

amomchilov deleted the c-api branch July 2, 2025 13:59

amomchilov mentioned this pull request Sep 26, 2025

Use malloc based allocator #2666

Merged

soutaro added a commit that referenced this pull request Oct 6, 2025

Merge pull request #2398 from Shopify/c-api

713b3ad

Introduce standalone C parser for RBS with arena allocation

soutaro mentioned this pull request Oct 6, 2025

Backport pure-C parser #2671

Merged

soutaro added a commit that referenced this pull request Oct 6, 2025

Merge pull request #2398 from Shopify/c-api

f0efec1

Introduce standalone C parser for RBS with arena allocation

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Introduce standalone C parser for RBS with arena allocation #2398

Introduce standalone C parser for RBS with arena allocation #2398

Uh oh!

st0012 commented Apr 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

soutaro commented May 1, 2025

Uh oh!

soutaro left a comment

Uh oh!

Uh oh!

soutaro commented May 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Search code, repositories, users, issues, pull requests...

Introduce standalone C parser for RBS with arena allocation #2398

Introduce standalone C parser for RBS with arena allocation #2398

Uh oh!

Conversation

st0012 commented Apr 10, 2025

RBS C Parser Library Refactoring

Key Improvements

Enhanced Memory Management

Component Architecture

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

soutaro commented May 1, 2025

Uh oh!

soutaro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

soutaro commented May 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants