Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

ENH: Implement pandas.read_iceberg #61383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 37 commits into from
May 14, 2025
+396 −37
Merged

ENH: Implement pandas.read_iceberg #61383

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
aa59971
ENH: Implement pandas.read_iceberg
datapythonista Apr 30, 2025
9ef8e4e
Typo in API index
datapythonista Apr 30, 2025
34792c9
Run iceberg tests
datapythonista Apr 30, 2025
c1b2426
Fix docstring example
datapythonista Apr 30, 2025
6081bfb
Fixes to the docstring
datapythonista Apr 30, 2025
91370db
Creating catalog dynamically
datapythonista Apr 30, 2025
ecf40ff
Fixing tests
datapythonista May 2, 2025
8c1e4dc
Adding debug info
datapythonista May 3, 2025
ee61079
Making pyiceberg tests run in a single cpu
datapythonista May 3, 2025
2debd5f
Debugging CI problems
datapythonista May 3, 2025
e593977
Bump pyiceberg version to 0.8.1
datapythonista May 3, 2025
24c0ceb
Revert "Bump pyiceberg version to 0.8.1"
datapythonista May 3, 2025
5fc738e
Commenting debugging information
datapythonista May 3, 2025
4953745
Bump minimum version to 0.7.1
datapythonista May 4, 2025
17d73e8
Removing debug code
datapythonista May 4, 2025
5f07a49
Allowing older version of gcsfs
datapythonista May 4, 2025
32add5f
Allowing an older version of s3fs
datapythonista May 4, 2025
3b0d7ee
Allowing an older version of fsspec
datapythonista May 5, 2025
7018c11
adding pyiceberg to requirements
datapythonista May 5, 2025
75c24d6
empty
datapythonista May 5, 2025
9c343a5
pre-commit
datapythonista May 5, 2025
9cd2d5c
Updating gcsfs minimum version
datapythonista May 5, 2025
301e988
Print difference in validation of minimum versions
datapythonista May 5, 2025
15f6397
Fix diff print
datapythonista May 5, 2025
f973e61
Fix bug when showing diff
datapythonista May 5, 2025
9110f2c
Merge remote-tracking branch 'upstream/main' into read_iceberg
datapythonista May 5, 2025
c13ce5b
debug validate min versions
datapythonista May 5, 2025
bc4d689
Updating new CI deps, reverting validate min versions script changes
datapythonista May 5, 2025
735c48c
Reverting test data to working version
datapythonista May 5, 2025
761b92e
Making read_iceberg experimental
datapythonista May 12, 2025
c25536c
Merge from main
datapythonista May 12, 2025
0a2e9ea
Using fixture for catalog
datapythonista May 12, 2025
2ef6343
Fix tests after using fixture
datapythonista May 12, 2025
7dd054b
Remove unneeded list when defining single cpu pytest mark
datapythonista May 12, 2025
74a1e65
whatsnew entry
datapythonista May 12, 2025
6917b30
Merge branch 'read_iceberg' of github.com:datapythonista/pandas into …
datapythonista May 12, 2025
dd0b5e4
Merge branch 'main' into read_iceberg
datapythonista May 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Updating gcsfs minimum version
  • Loading branch information
@datapythonista
datapythonista committed May 5, 2025
commit 9cd2d5ca9b639258f5667972ecd0100ade4a4069
2 changes: 1 addition & 1 deletion 2 ci/deps/actions-310-minimum_versions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ dependencies:
- fsspec=2023.12.2
mroeschke marked this conversation as resolved.
Show resolved Hide resolved
- html5lib=1.1
- hypothesis=6.84.0
- gcsfs=2024.0.0
- gcsfs=2023.12.2
- jinja2=3.1.3
- lxml=4.9.2
- matplotlib=3.8.3
Expand Down
2 changes: 1 addition & 1 deletion 2 ci/deps/actions-310.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ dependencies:
- fsspec>=2023.12.2
- html5lib>=1.1
- hypothesis>=6.84.0
- gcsfs>=2024.0.0
- gcsfs>=2023.12.2
- jinja2>=3.1.3
- lxml>=4.9.2
- matplotlib>=3.8.3
Expand Down
2 changes: 1 addition & 1 deletion 2 ci/deps/actions-311-downstream_compat.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ dependencies:
- fsspec>=2023.12.2
- html5lib>=1.1
- hypothesis>=6.84.0
- gcsfs>=2024.0.0
- gcsfs>=2023.12.2
- jinja2>=3.1.3
- lxml>=4.9.2
- matplotlib>=3.8.3
Expand Down
2 changes: 1 addition & 1 deletion 2 ci/deps/actions-311.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ dependencies:
- fsspec>=2023.12.2
- html5lib>=1.1
- hypothesis>=6.84.0
- gcsfs>=2024.0.0
- gcsfs>=2023.12.2
- jinja2>=3.1.3
- lxml>=4.9.2
- matplotlib>=3.8.3
Expand Down
2 changes: 1 addition & 1 deletion 2 ci/deps/actions-312.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ dependencies:
- fsspec>=2023.12.2
- html5lib>=1.1
- hypothesis>=6.84.0
- gcsfs>=2024.0.0
- gcsfs>=2023.12.2
- jinja2>=3.1.3
- lxml>=4.9.2
- matplotlib>=3.8.3
Expand Down
2 changes: 1 addition & 1 deletion 2 doc/source/getting_started/install.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -332,7 +332,7 @@ Dependency Minimum Version pip extra
============================================ ================== =============== ==========================================================
`fsspec <https://github.com/fsspec>`__ 2023.12.2 fss, gcp, aws Handling files aside from simple local and HTTP (required
dependency of s3fs, gcsfs).
`gcsfs <https://github.com/fsspec/gcsfs>`__ 2024.0.0 gcp Google Cloud Storage access
`gcsfs <https://github.com/fsspec/gcsfs>`__ 2023.12.2 gcp Google Cloud Storage access
`s3fs <https://github.com/fsspec/s3fs>`__ 2023.12.2 aws Amazon S3 access
============================================ ================== =============== ==========================================================

Expand Down
2 changes: 1 addition & 1 deletion 2 environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ dependencies:
- fsspec>=2023.12.2
- html5lib>=1.1
- hypothesis>=6.84.0
- gcsfs>=2024.0.0
- gcsfs>=2023.12.2
- ipython
- pickleshare # Needed for IPython Sphinx directive in the docs GH#60429
- jinja2>=3.1.3
Expand Down
4 changes: 2 additions & 2 deletions 4 pandas/compat/_optional.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
"fsspec": "2023.12.2",
"html5lib": "1.1",
"hypothesis": "6.84.0",
"gcsfs": "2024.0.0",
"gcsfs": "2023.12.2",
"jinja2": "3.1.3",
"lxml.etree": "4.9.2",
"matplotlib": "3.8.3",
Expand All @@ -40,7 +40,7 @@
"psycopg2": "2.9.6", # (dt dec pq3 ext lo64)
"pymysql": "1.1.0",
"pyarrow": "10.0.1",
"pyiceberg": "0.7.1", # FIXME remove this comment
"pyiceberg": "0.7.1",
"pyreadstat": "1.2.6",
"pytest": "7.3.2",
"python-calamine": "0.1.7",
Expand Down
4 changes: 2 additions & 2 deletions 4 pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ performance = ['bottleneck>=1.3.6', 'numba>=0.59.0', 'numexpr>=2.9.0']
computation = ['scipy>=1.12.0', 'xarray>=2024.1.1']
fss = ['fsspec>=2023.12.2']
aws = ['s3fs>=2023.12.2']
gcp = ['gcsfs>=2024.0.0']
gcp = ['gcsfs>=2023.12.2']
excel = ['odfpy>=1.4.1', 'openpyxl>=3.1.2', 'python-calamine>=0.1.7', 'pyxlsb>=1.0.10', 'xlrd>=2.0.1', 'xlsxwriter>=3.2.0']
parquet = ['pyarrow>=10.0.1']
feather = ['pyarrow>=10.0.1']
Expand All @@ -91,7 +91,7 @@ all = ['adbc-driver-postgresql>=0.10.0',
'bottleneck>=1.3.6',
'fastparquet>=2024.2.0',
'fsspec>=2023.12.2',
'gcsfs>=2024.0.0',
'gcsfs>=2023.12.2',
'html5lib>=1.1',
'hypothesis>=6.84.0',
'jinja2>=3.1.3',
Expand Down
2 changes: 1 addition & 1 deletion 2 requirements-dev.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ fastparquet>=2024.2.0
fsspec>=2023.12.2
html5lib>=1.1
hypothesis>=6.84.0
gcsfs>=2024.0.0
gcsfs>=2023.12.2
ipython
pickleshare
jinja2>=3.1.3
Expand Down
4 changes: 2 additions & 2 deletions 4 scripts/tests/data/deps_minimum.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ timezone = ['tzdata>=2022.1']
computation = ['scipy>=1.7.1', 'xarray>=0.21.0']
fss = ['fsspec>=2023.12.2']
aws = ['s3fs>=2023.12.2']
gcp = ['gcsfs>=2024.0.0']
gcp = ['gcsfs>=2023.12.2']
excel = ['odfpy>=1.4.1', 'openpyxl>=3.0.7', 'python-calamine>=0.1.7', 'pyxlsb>=1.0.8', 'xlrd>=2.0.1', 'xlsxwriter>=1.4.3']
parquet = ['pyarrow>=7.0.0']
feather = ['pyarrow>=7.0.0']
Expand All @@ -83,7 +83,7 @@ all = ['beautifulsoup4>=5.9.3',
'bottleneck>=1.3.2',
'fastparquet>=0.6.3',
'fsspec>=2023.12.2',
'gcsfs>=2024.0.0',
'gcsfs>=2023.12.2',
'html5lib>=1.1',
'hypothesis>=6.34.2',
'jinja2>=3.0.0',
Expand Down
Loading
Morty Proxy This is a proxified and sanitized view of the page, visit original site.