Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings
Discussion options

Hello I've recently been using the adbc driver for duckdb and find it convenient, but one huge missing piece that has been vexing me is codegen / static type checking of arrow schemas. While I know that arrow is a columnar storage format, when interacting with it in a client it is in many cases still preferable to model / access it as a traditional domain object for the simple reason that the properties of a given arrow schema will change over time -just like any other data interchange structure.

For most widespread data formats, there are typically tools to take a schema and generate the relevant client code. Flatbuffers has flatc, protobuf has protoc. There are plenty of tools which can take a JSON schema and generate client code, but conspicuously I have never been able to find any equivalent for arrow (atleast for python classes / type annotations) despite it's widespread use.

I am wondering if these capabilities exist somewhere and I have simply missed them or if there is some underlying reason that no such tools exist.

I have a very crude system of wrapper facades that works well enough for static type checking which looks something like below, but I keep thinking somewhere out there with 8.1 billion people on Earth, someone must have built a more automated / polished version of this.

Wrappers

class ListFacade:
    """ A facade for a pyarrow ListScalar """
    def __init__(self, lst: pyarrow.lib.ListScalar):
        self.lst = lst

    def __getitem__(self, index: int):
        val_type = self.lst.type.value_type
        return convert_value(self.lst[index],val_type)

    def __len__(self):
        return len(self.lst)


class StructFacade:
    """ A facade for a pyarrow StructScalar """
    def __init__(self, struct: pyarrow.lib.StructScalar):
        self.struct = struct

    def __getattr__(self, key: str):
        field = self.struct.type[key]
        return convert_value(self.struct[key],field.type)


class RecordFacade:
    """ A facade for a single row in a pyarrow RecordBatch """

    def __init__(self, batch: pyarrow.lib.RecordBatch, row_num: int):
        self.batch = batch
        self.row_num = row_num

    def __getattr__(self, key: str):
        field = self.batch.schema.field(key)
        return convert_value(self.batch.column(key)[self.row_num],field.type)

class BatchReaderFacade[T]:
    """Wrapper for batch reader that yields rows as facades"""
    def __init__(self, reader:pyarrow.lib.RecordBatchReader):
        self.reader = reader

    def rows(self) -> typing.Iterable[T]:
        """ Yields each row in the batch as a RecordFacade that provides attributes for Protocol T"""
        for batch in self.reader:
            for i in range(batch.num_rows):
                yield RecordFacade(batch, i)

## Converter
```python
def convert_value(value,schema:pyarrow.lib.DataType):
    """ Convert a pyarrow value to a Python value or a facade object """
    if pyarrow.types.is_struct(schema):
        return StructFacade(value)
    if pyarrow.types.is_primitive(schema):
        return value.as_py()
    if pyarrow.types.is_list(schema):
        return ListFacade(value)
    raise ValueError(f"Unsupported type: {schema}")

Example domain Protocols (interfaces)

class DatabaseFunctionParam(Protocol):
    name: str
    dtype: str

class DatabaseFunction(Protocol):
    name: str
    kind: str
    return_type: str
    params: list[DatabaseFunctionParam]
    database: str

class DatabaseSchema(Protocol):
    id: int
    name: str
    functions: list[DatabaseFunction]

Query

with adbc_driver_manager.dbapi.connect(
    driver="libduckdb", entrypoint="duckdb_adbc_init"
) as conn:
    with conn.cursor() as cursor:
        cursor.execute("""
        SELECT
          s.oid as id,
          s.schema_name as name,
          s.database_name as database,
          array_agg(
            struct_pack(
              name:=f.function_name,
              kind:=f.function_type,
              return_type:=f.return_type,
              params:=f.parameters.apply(lambda pn,i: struct_pack(name:=pn,dtype:=f.parameter_types[i]))
            ) ORDER BY f.function_name
          ) as functions
        FROM
          duckdb_functions() as f
          JOIN duckdb_schemas() as s ON f.database_name = s.database_name AND f.schema_name = s.schema_name
        GROUP BY
            s.oid,
            s.schema_name,
            s.database_name
        """)
        reader = BatchReaderFacade[DatabaseSchema](cursor.fetch_record_batch())
        for dbSchema in reader.rows():
            print(dbSchema.name)
            for fn in dbSchema.functions:
                print(f"  {fn.name}")
                for param in fn.params:
                    print(f"     {param.name}: {param.dtype}")
You must be logged in to vote

Replies: 1 comment

Comment options

I have wanted something like this for a while, but AFAIK nothing really exists. There are some other experiments along these lines, e.g. https://github.com/simw/pydantic-to-pyarrow for Pydantic, https://github.com/michalc/pgarrow for SQLAlchemy + PostgreSQL

There's also a lack of a textual format for Arrow schemas in the first place (#25078)

You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
None yet
2 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.