Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

#49 Implement Python native write with PyArrow#51

Merged
yuzelin merged 12 commits into
apache:mainapache/paimon-python:mainfrom
chenghuichen:pyarrow-part2chenghuichen/paimon-python:pyarrow-part2Copy head branch name to clipboard
Jul 28, 2025
Merged

#49 Implement Python native write with PyArrow#51
yuzelin merged 12 commits into
apache:mainapache/paimon-python:mainfrom
chenghuichen:pyarrow-part2chenghuichen/paimon-python:pyarrow-part2Copy head branch name to clipboard

Conversation

@chenghuichen
Copy link
Copy Markdown
Contributor

@chenghuichen chenghuichen commented Jun 23, 2025

Purpose
Linked issue: close #49

Tests

  • pypaimon/pynative/tests/test_append_only_reader.py
  • pypaimon/pynative/tests/test_primary_key_reader.py
  • pypaimon/pynative/tests/test_native_write_and_read.py

@chenghuichen
Copy link
Copy Markdown
Contributor Author

chenghuichen commented Jul 13, 2025

Key To-Do Items:

  • Implement end-to-end filtering from the read plan phase to the format reader phase to avoid performance lost in cases with predicates.
  • Optimize the raw performance of the MergeReader.
  • Complete the manifest details for the writer commit phase.

@yuzelin yuzelin merged commit c263bdd into apache:main Jul 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Performance Improvement: Replace Py4J-based Implementation with Native PyArrow

2 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.