docs: update notebook for JSON subfields support in to_pandas_batches() by shuoweil · Pull Request #2138 · googleapis/python-bigquery-dataframes

shuoweil · Oct 4, 2025

When displaying a DataFrame containing JSON columns (including nested JSON in lists or structs), the anywidget table would fail to render and fall back to the "Computation deferred" message. This was caused by a limitation in PyArrow (apache/arrow#45262).

This branch only updates the relevant notebook. The underlying code fix for handling JSON subfields in STRUCT and ARRAY columns was already merged via PR #2216 (fix: support results with STRUCT and ARRAY columns containing JSON subfields in to_pandas_batches()).

This change simply aligns the notebook with the updated code.

Fixes #<448126500 and 453561268> 🦕

review-notebook-app · Oct 4, 2025

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

tswast

In b/448126500 I suggest investigating the errors that happen when visualizing STRUCT columns, but this PR doesn't test such cases.

tswast · Oct 15, 2025

-        # anywdiget mode uses the same display logic as the "deferred" mode
-        # for faster execution
-        if opts.repr_mode in ("deferred", "anywidget"):


Let's revert this.

tswast

Thanks!

🤖 I have created a release *beep* *boop* --- ## [2.29.0](v2.28.0...v2.29.0) (2025-11-10) ### Features * Add bigframes.bigquery.st_regionstats to join raster data from Earth Engine ([#2228](#2228)) ([10ec52f](10ec52f)) * Add DataFrame.resample and Series.resample ([#2213](#2213)) ([c9ca02c](c9ca02c)) * SQL Cell no longer escapes formatted string values ([#2245](#2245)) ([d2d38f9](d2d38f9)) * Support left_index and right_index for merge ([#2220](#2220)) ([da9ba26](da9ba26)) ### Bug Fixes * Correctly iterate over null struct values in ManagedArrowTable ([#2209](#2209)) ([12e04d5](12e04d5)) * Simplify UnsupportedTypeError message ([#2212](#2212)) ([6c9a18d](6c9a18d)) * Support results with STRUCT and ARRAY columns containing JSON subfields in `to_pandas_batches()` ([#2216](#2216)) ([3d8b17f](3d8b17f)) ### Documentation * Switch API reference docs to pydata theme ([#2237](#2237)) ([9b86dcf](9b86dcf)) * Update notebook for JSON subfields support in to_pandas_batches() ([#2138](#2138)) ([5663d2a](5663d2a)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## [2.29.0](googleapis/python-bigquery-dataframes@v2.28.0...v2.29.0) (2025-11-10) ### Features * Add bigframes.bigquery.st_regionstats to join raster data from Earth Engine ([#2228](googleapis/python-bigquery-dataframes#2228)) ([10ec52f](googleapis/python-bigquery-dataframes@10ec52f)) * Add DataFrame.resample and Series.resample ([#2213](googleapis/python-bigquery-dataframes#2213)) ([c9ca02c](googleapis/python-bigquery-dataframes@c9ca02c)) * SQL Cell no longer escapes formatted string values ([#2245](googleapis/python-bigquery-dataframes#2245)) ([d2d38f9](googleapis/python-bigquery-dataframes@d2d38f9)) * Support left_index and right_index for merge ([#2220](googleapis/python-bigquery-dataframes#2220)) ([da9ba26](googleapis/python-bigquery-dataframes@da9ba26)) ### Bug Fixes * Correctly iterate over null struct values in ManagedArrowTable ([#2209](googleapis/python-bigquery-dataframes#2209)) ([12e04d5](googleapis/python-bigquery-dataframes@12e04d5)) * Simplify UnsupportedTypeError message ([#2212](googleapis/python-bigquery-dataframes#2212)) ([6c9a18d](googleapis/python-bigquery-dataframes@6c9a18d)) * Support results with STRUCT and ARRAY columns containing JSON subfields in `to_pandas_batches()` ([#2216](googleapis/python-bigquery-dataframes#2216)) ([3d8b17f](googleapis/python-bigquery-dataframes@3d8b17f)) ### Documentation * Switch API reference docs to pydata theme ([#2237](googleapis/python-bigquery-dataframes#2237)) ([9b86dcf](googleapis/python-bigquery-dataframes@9b86dcf)) * Update notebook for JSON subfields support in to_pandas_batches() ([#2138](googleapis/python-bigquery-dataframes#2138)) ([5663d2a](googleapis/python-bigquery-dataframes@5663d2a)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

shuoweil requested review from TrevorBergeron and tswast October 4, 2025 07:39

shuoweil self-assigned this Oct 4, 2025

shuoweil requested review from a team October 4, 2025 07:39

product-auto-label Bot added size: s Pull request size is small. api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. labels Oct 4, 2025

product-auto-label Bot added size: m Pull request size is medium. and removed size: s Pull request size is small. labels Oct 4, 2025

tswast reviewed Oct 6, 2025

View reviewed changes

Comment thread bigframes/dataframe.py Outdated

tswast suggested changes Oct 6, 2025

View reviewed changes

shuoweil force-pushed the shuowei-anywidget-col branch 2 times, most recently from 43a938c to 4a33ccf Compare October 9, 2025 06:43

shuoweil requested a review from tswast October 9, 2025 06:45

tswast reviewed Oct 15, 2025

View reviewed changes

Comment thread bigframes/operations/blob.py Outdated

tswast reviewed Oct 15, 2025

View reviewed changes

Comment thread tests/system/large/blob/test_function.py Outdated

tswast reviewed Oct 15, 2025

View reviewed changes

Comment thread bigframes/dataframe.py Outdated

shuoweil force-pushed the shuowei-anywidget-col branch from 30dfa7d to 237c134 Compare October 15, 2025 19:45

shuoweil requested a review from tswast October 15, 2025 20:51

tswast reviewed Oct 15, 2025

View reviewed changes

shuoweil force-pushed the shuowei-anywidget-col branch from f18fd9e to 8601e52 Compare October 20, 2025 08:20

product-auto-label Bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Oct 21, 2025

shuoweil force-pushed the shuowei-anywidget-col branch from 533541a to 860db3e Compare October 21, 2025 01:57

product-auto-label Bot added size: m Pull request size is medium. size: l Pull request size is large. and removed size: l Pull request size is large. size: m Pull request size is medium. labels Oct 21, 2025

shuoweil changed the title ~~perf: Default to interactive display for SQL in anywidget mode~~ feat: Display JSON columns in anywidget mode Oct 21, 2025

product-auto-label Bot removed the size: l Pull request size is large. label Oct 30, 2025

shuoweil changed the title ~~fix: Correctly display DataFrames with JSON columns in anywidget~~ fix: Handle empty DataFrames with nested JSON columns in to_pandas_batches() Oct 30, 2025

shuoweil changed the title ~~fix: Handle empty DataFrames with nested JSON columns in to_pandas_batches()~~ feat: Display JSON columns in anywidget mode fix: Correctly display DataFrames with JSON columns in anywidget Oct 30, 2025

shuoweil changed the title ~~feat: Display JSON columns in anywidget mode fix: Correctly display DataFrames with JSON columns in anywidget~~ feat: Display JSON columns in anywidget mode Oct 30, 2025

shuoweil added 2 commits October 31, 2025 00:12

Improve JSON type handling for to_gbq and to_pandas_batches

4cbaf15

Remove unnecessary comment

2113425

product-auto-label Bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Oct 31, 2025

shuoweil added 2 commits October 31, 2025 20:37

Merge branch 'main' into shuowei-anywidget-col

b75cb8c

Merge branch 'main' into shuowei-anywidget-col

d97cfac

product-auto-label Bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels Nov 4, 2025

Revert bigframes/dtypes.py and mypy.ini to main branch version

e06a9f4

shuoweil changed the title ~~feat: Display JSON columns in anywidget mode~~ docs: sync notebook with JSON subfield fix from #2216 Nov 4, 2025

shuoweil marked this pull request as ready for review November 4, 2025 20:22

shuoweil requested review from TrevorBergeron and tswast November 4, 2025 20:22

shuoweil changed the title ~~docs: sync notebook with JSON subfield fix from #2216~~ docs: update notebook for JSON subfields support in to_pandas_batches() Nov 4, 2025

tswast added the owlbot:run Add this label to trigger the Owlbot post processor. label Nov 4, 2025

gcf-owl-bot Bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Nov 4, 2025

tswast approved these changes Nov 4, 2025

View reviewed changes

tswast enabled auto-merge (squash) November 4, 2025 22:34

tswast merged commit 5663d2a into main Nov 4, 2025
25 checks passed

tswast deleted the shuowei-anywidget-col branch November 4, 2025 22:34

release-please Bot mentioned this pull request Nov 4, 2025

chore(main): release 2.29.0 #2225

Closed

parthea mentioned this pull request Nov 7, 2025

chore: librarian release pull request: 20251107T184449Z #2241

Closed

release-please Bot mentioned this pull request Nov 7, 2025

chore(main): release 2.29.0 #2242

Merged

gkevinzheng mentioned this pull request Nov 13, 2025

chore: librarian release pull request: 20251113T215818Z #2265

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

docs: update notebook for JSON subfields support in to_pandas_batches()#2138

docs: update notebook for JSON subfields support in to_pandas_batches()#2138
tswast merged 59 commits into
maingoogleapis/python-bigquery-dataframes:mainfrom
shuowei-anywidget-colgoogleapis/python-bigquery-dataframes:shuowei-anywidget-colCopy head branch name to clipboard

shuoweil commented Oct 4, 2025 •

edited

Loading

Uh oh!

review-notebook-app Bot commented Oct 4, 2025

Uh oh!

Uh oh!

tswast left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tswast Oct 15, 2025

Uh oh!

tswast left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Search code, repositories, users, issues, pull requests...

Uh oh!

Conversation

shuoweil commented Oct 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

review-notebook-app Bot commented Oct 4, 2025

Uh oh!

Uh oh!

tswast left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tswast Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

tswast left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shuoweil commented Oct 4, 2025 •

edited

Loading