Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

BUG/PERF: Series.combine_first converting int64 to float64 #51777

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Series.combine_first to preserve dtype
  • Loading branch information
lukemanley committed Mar 4, 2023
commit d28c39c518f387383c1b166f15627c9b4025024f
2 changes: 1 addition & 1 deletion 2 doc/source/whatsnew/v2.1.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ Groupby/resample/rolling

Reshaping
^^^^^^^^^
-
- Bug in :meth:`Series.combine_first` converting ``int64`` dtype to ``float64`` (:issue:`51764`)
-

Sparse
Expand Down
10 changes: 9 additions & 1 deletion 10 pandas/core/series.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,11 +93,13 @@
from pandas.core.dtypes.cast import (
LossySetitemError,
convert_dtypes,
find_common_type,
maybe_box_native,
maybe_cast_pointwise_result,
)
from pandas.core.dtypes.common import (
is_dict_like,
is_dtype_equal,
is_extension_array_dtype,
is_integer,
is_iterator,
Expand Down Expand Up @@ -3272,7 +3274,13 @@ def combine_first(self, other) -> Series:
if this.dtype.kind == "M" and other.dtype.kind != "M":
other = to_datetime(other)

return this.where(notna(this), other)
combined = this.where(notna(this), other)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this won't break anything, but for the series case there might be a proper fix. You could set a compatible fill_value for the reindex ops. Only disadvantage is, that you'll have to compute notna(this) somehow


if not is_dtype_equal(combined.dtype, self.dtype):
dtype = find_common_type([self.dtype, other.dtype])
combined = combined.astype(dtype, copy=False)

return combined
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a finalise call with self, otherwise we will loose metadata.

This might changed result ordering? Can we do a reindex in the end?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, thanks


def update(self, other: Series | Sequence | Mapping) -> None:
"""
Expand Down
8 changes: 8 additions & 0 deletions 8 pandas/tests/series/methods/test_combine_first.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,3 +112,11 @@ def test_combine_first_timezone_series_with_empty_series(self):
s2 = Series(index=time_index)
result = s1.combine_first(s2)
tm.assert_series_equal(result, s1)

def test_combine_first_preserves_dtype(self):
# GH51764
s1 = Series([4, 5])
s2 = Series([6, 7, 8])
result = s1.combine_first(s2)
expected = Series([4, 5, 8])
tm.assert_series_equal(result, expected)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.