Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

REGR: Comparing timestamp to empty Series raises TypeError starting in 0.25.0 #35548

Copy link
Copy link
Closed
@pganssle

Description

@pganssle
Issue body actions
  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


Minimal reproducing code sample

import pandas as pd

pd.Series() < pd.Timestamp.now()  # Raises TypeError

(Note: This raises a warning because the default dtype is changing to object — which will actually fix the issue — but no warning is raised when you slice a column from an empty DataFrame).

Problem description

Prior to 0.25.0, it was possible to compare a pd.Timestamp to an empty Series (even if that series did not have a dtype of datetime64[ns]), but starting in the 0.25.0 release, this operation now raises TypeError because of mismatched dtypes. Running git bisect, I gather that this change was introduced in GH-26916, but neither the PR or the issue seem to suggest that this change was an explicit goal.

It might be arguable what the correct behavior should be, since it's probably valid that you shouldn't be able to compare a Timestamp to a float64, but it's also obvious what the return value should be: an empty Series of dtype bool.

The reason it would be valuable to allow this behavior is that a lot of people rely on pandas to infer their data types from the input, but no such inference can take place if the data set is empty, in the pre-0.25.0 world, it was possible to design an API like this:

def get_names_in_past(names: List[str], dates: List[datetime]) -> pd.DataFrame:
    df = pd.DataFrame({'names': names, 'dates': dates})
    return df[df['dates'] < pd.Timestamp.now()]

But starting in 0.25.0, when passed an empty list, you'd get a TypeError instead of an empty DataFrame, and you need to special-case the situation where df is empty. There are similar APIs you can imagine where the data are read from a CSV or some other source where the dtype is inferred from the contents that would start failing on empty data.

I think it would be reasonable to short-circuit comparisons with empty Series before the dtypes are compared, to always return an empty boolean Series.

Another reasonable option that I think would be more narrowly-scoped to solve the particular problem of dtype inference without data would be adding a flag to Series to indicate if the dtype is ambiguous. For pd.Series() it would be True and for pd.Series(dtype=float) it would be False. This would be more complicated to implement because you'd need to make sure that it's set correctly when the Series is part of a DataFrame and make sure it's set correctly in .astype, but that would allow you to preserve the error in comparing the wrong types when a type has been explicitly specified while avoiding the need for clunky code that has to explicitly handle empty DataFrames.

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugClosing CandidateMay be closeable, needs more eyeballsMay be closeable, needs more eyeballsDatetimeDatetime data dtypeDatetime data dtypeNumeric OperationsArithmetic, Comparison, and Logical operationsArithmetic, Comparison, and Logical operations

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      Morty Proxy This is a proxified and sanitized view of the page, visit original site.