Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

fix: correct the numeric literal dtype #365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 14, 2024
Merged

fix: correct the numeric literal dtype #365

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f81077a
fix: correct the numeric literal dtype
chelsea-lin Feb 1, 2024
cf175be
Merge branch 'main' into main_chelsealin_taxibug
chelsea-lin Feb 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion 20 bigframes/dtypes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -378,11 +378,29 @@ def literal_to_ibis_scalar(
scalar_expr = ibis.literal(literal, ibis_dtypes.float64)
elif scalar_expr.type().is_integer():
scalar_expr = ibis.literal(literal, ibis_dtypes.int64)
elif scalar_expr.type().is_decimal():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would break BIGNUMERIC, I think. We should probably have something that goes to the nearest type. https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#decimal_types

Like is_decimal and precision < 76, then this otherwise set to the precision and scale for BIGNUMERIC.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point! thanks!

precision = scalar_expr.type().precision
scale = scalar_expr.type().scale
if (not precision and not scale) or (
precision and scale and scale <= 9 and precision + (9 - scale) <= 38
):
scalar_expr = ibis.literal(
literal, ibis_dtypes.decimal(precision=38, scale=9)
)
elif precision and scale and scale <= 38 and precision + (38 - scale) <= 76:
scalar_expr = ibis.literal(
literal, ibis_dtypes.decimal(precision=76, scale=38)
)
else:
raise TypeError(
"BigQuery's decimal data type supports a maximum precision of 76 and a maximum scale of 38."
f"Current precision: {precision}. Current scale: {scale}"
)

# TODO(bmil): support other literals that can be coerced to compatible types
if validate and (scalar_expr.type() not in BIGFRAMES_TO_IBIS.values()):
raise ValueError(
f"Literal did not coerce to a supported data type: {literal}. {constants.FEEDBACK_LINK}"
f"Literal did not coerce to a supported data type: {scalar_expr.type()}. {constants.FEEDBACK_LINK}"
)

return scalar_expr
Expand Down
10 changes: 10 additions & 0 deletions 10 tests/system/small/test_series.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1228,6 +1228,16 @@ def test_median(scalars_dfs):
assert pd_min < bf_result < pd_max


def test_numeric_literal(scalars_dfs):
scalars_df, _ = scalars_dfs
col_name = "numeric_col"
assert scalars_df[col_name].dtype == pd.ArrowDtype(pa.decimal128(38, 9))
bf_result = scalars_df[col_name] - scalars_df[col_name].median()
assert bf_result.size == scalars_df[col_name].size
# TODO(b/323387826): The precision increased by 1 unexpectedly.
# assert bf_result.dtype == pd.ArrowDtype(pa.decimal128(38, 9))


def test_repr(scalars_dfs):
scalars_df, scalars_pandas_df = scalars_dfs
if scalars_pandas_df.index.name != "rowindex":
Expand Down
Morty Proxy This is a proxified and sanitized view of the page, visit original site.