Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Conversation

@gabeiglio
Copy link
Contributor

@gabeiglio gabeiglio commented Dec 18, 2024

This is a fix for issue #1401. In which table scans needed to infer partition column by following the column projection rules

Fixes #1401

@Fokko Fokko self-requested a review December 18, 2024 20:38
@gabeiglio gabeiglio marked this pull request as ready for review December 19, 2024 15:12
Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a few comments, please take a look! The PR looks great already. Thanks for working on this!

pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
tests/io/test_pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
@kevinjqliu kevinjqliu self-requested a review December 23, 2024 19:04
…tion logic to helper method, changed test to use high-level table scan
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
tests/io/test_pyarrow.py Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
@Fokko Fokko self-requested a review January 13, 2025 12:48
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generally LGTM! I added a few nit comments and some clarifying questions on testing.

thanks for working on this!

tests/io/test_pyarrow.py Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
tests/io/test_pyarrow.py Outdated Show resolved Hide resolved
tests/io/test_pyarrow.py Show resolved Hide resolved
Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like CI caught an interesting case where a new identity partition is added after data files were written. The accessor then cannot find the proper partition record... We need to do something like this

tests/io/test_pyarrow.py Outdated Show resolved Hide resolved
tests/io/test_pyarrow.py Outdated Show resolved Hide resolved
tests/io/test_pyarrow.py Show resolved Hide resolved
Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there!

pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
pyiceberg/io/pyarrow.py Outdated Show resolved Hide resolved
Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks @gabeiglio !!

@kevinjqliu kevinjqliu requested a review from Fokko February 5, 2025 02:16
@Fokko Fokko merged commit d9b1c03 into apache:main Feb 10, 2025
7 checks passed
@Fokko
Copy link
Contributor

Fokko commented Feb 10, 2025

Thanks for working on this @gabeiglio, and thanks for the review @kevinjqliu 🙌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

API table.scan does not conform to Iceberg spec for identity partition columns

3 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.