Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

BUG: hashing datetime64 objects #50960

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 64 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
7761ecd
BUG: hashing datetime64 objects
jbrockmendel Jan 24, 2023
610b0c6
handle cases out of pydatetime bounds
jbrockmendel Jan 24, 2023
0e140ae
Merge branch 'main' into bug-hash-dt64
jbrockmendel Jan 24, 2023
919383c
Merge branch 'main' into bug-hash-dt64
jbrockmendel Jan 24, 2023
ae8c0bb
Merge branch 'main' into bug-hash-dt64
jbrockmendel Jan 25, 2023
92a39eb
troubleshoot CI builds
jbrockmendel Jan 25, 2023
2f67805
troubleshoot CI builds
jbrockmendel Jan 25, 2023
0635f86
troubleshoot CI builds
jbrockmendel Jan 25, 2023
229ab72
troubleshoot CI builds
jbrockmendel Jan 25, 2023
6e96805
troubleshoot CI builds
jbrockmendel Jan 25, 2023
24fda82
Merge branch 'main' into bug-hash-dt64
jbrockmendel Jan 26, 2023
058b666
suggested edits
jbrockmendel Jan 27, 2023
7398991
Merge branch 'main' into bug-hash-dt64
jbrockmendel Jan 31, 2023
3fdf564
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 8, 2023
6e4836e
use sebergs suggestion
jbrockmendel Feb 8, 2023
f55337a
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 9, 2023
a97dfc9
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 9, 2023
1338ca2
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 9, 2023
9fb1987
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 10, 2023
818682c
suggested edits
jbrockmendel Feb 10, 2023
74ab540
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 10, 2023
037ba05
remove unnecessary casts
jbrockmendel Feb 11, 2023
7a4b1ab
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 11, 2023
47e5247
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 11, 2023
dcd09dd
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 11, 2023
32d479b
Merge branch 'bug-hash-dt64' of github.com:jbrockmendel/pandas into b…
jbrockmendel Feb 11, 2023
d47cfd8
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 12, 2023
c091317
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 12, 2023
1ce791e
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 13, 2023
704fb69
suggested edit for PyDateTime_IMPORT
jbrockmendel Feb 13, 2023
6d962b0
Merge branch 'bug-hash-dt64' of github.com:jbrockmendel/pandas into b…
jbrockmendel Feb 13, 2023
f838953
revert delay
jbrockmendel Feb 13, 2023
0d8500a
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 13, 2023
58a29b6
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 14, 2023
95069e0
restore check
jbrockmendel Feb 14, 2023
7362f3e
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 14, 2023
dd08670
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 15, 2023
998a4cc
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 15, 2023
b75730b
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 17, 2023
5c57a5e
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 20, 2023
6b4460f
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 21, 2023
c94609b
add test
jbrockmendel Feb 22, 2023
afe9493
Merge branch 'main' into bug-hash-dt64
jbrockmendel Feb 26, 2023
4fecc97
Merge branch 'main' into bug-hash-dt64
jbrockmendel Mar 3, 2023
b4cc41e
Merge branch 'main' into bug-hash-dt64
jbrockmendel Mar 8, 2023
c620339
Merge branch 'main' into bug-hash-dt64
jbrockmendel Mar 9, 2023
3633653
Merge branch 'main' into bug-hash-dt64
jbrockmendel Mar 11, 2023
c55f182
shot in the dark
jbrockmendel Mar 11, 2023
6e2bbf0
Merge branch 'main' into bug-hash-dt64
jbrockmendel Mar 12, 2023
23c2826
capsule stuff
jbrockmendel Mar 12, 2023
143b3a3
guessing
jbrockmendel Mar 13, 2023
ffb8365
still tryin
jbrockmendel Mar 13, 2023
1fdfd64
Merge branch 'main' into bug-hash-dt64
jbrockmendel Mar 13, 2023
5513721
macro
jbrockmendel Mar 13, 2023
875d6af
revert sources
jbrockmendel Mar 13, 2023
a29a56a
Merge branch 'main' into bug-hash-dt64
jbrockmendel Mar 13, 2023
40e6e17
Move np_datetime64_object_hash to np_datetime.c
jbrockmendel Mar 13, 2023
15a701c
import_pandas_datetime more
jbrockmendel Mar 13, 2023
af25f40
troubleshoot
jbrockmendel Mar 13, 2023
9d5cb46
Merge branch 'main' into bug-hash-dt64
jbrockmendel Mar 14, 2023
d5a031d
Merge branch 'main' into bug-hash-dt64
jbrockmendel Mar 14, 2023
bd7d432
post-merge merges
jbrockmendel Mar 14, 2023
394d86e
frickin guess
jbrockmendel Mar 14, 2023
1766bc3
Merge branch 'main' into bug-hash-dt64
jbrockmendel Mar 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
troubleshoot
  • Loading branch information
jbrockmendel committed Mar 13, 2023
commit af25f40d46eae55a5c9217bab5f18d81d4dc3d33
1 change: 1 addition & 0 deletions 1 pandas/_libs/tslibs/src/datetime/pd_datetime.c
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ static int pandas_datetime_exec(PyObject *module) {
capi->make_iso_8601_timedelta = make_iso_8601_timedelta;
capi->hash_datetime_from_struct = hash_datetime_from_struct;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You also need to define a macro for this later in the file

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#define hash_datetime_from_struct(dts)                                         \
  PandasDateTimeAPI->hash_datetime_from_struct((dts))

didnt seem to do it

capi->np_datetime64_object_hash = np_datetime64_object_hash;
capi->tuple_update_uhash = tuple_update_uhash;

PyObject *capsule = PyCapsule_New(capi, PandasDateTime_CAPSULE_NAME,
pandas_datetime_destructor);
Expand Down
3 changes: 3 additions & 0 deletions 3 pandas/_libs/tslibs/src/datetime/pd_datetime.h
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ typedef struct {
int (*make_iso_8601_timedelta)(pandas_timedeltastruct *, char *, size_t *);
Py_hash_t (*hash_datetime_from_struct)(npy_datetimestruct* dts);
Py_hash_t (*np_datetime64_object_hash)(PyDatetimeScalarObject* key);
Py_uhash_t tuple_update_uhash(Py_uhash_t acc, Py_uhash_t lane);
} PandasDateTime_CAPI;

// The capsule name appears limited to module.attributename; see bpo-32414
Expand Down Expand Up @@ -114,6 +115,8 @@ static PandasDateTime_CAPI *PandasDateTimeAPI = NULL;
PandasDateTimeAPI->hash_datetime_from_struct((dts))
#define np_datetime64_object_hash(dts) \
PandasDateTimeAPI->np_datetime64_object_hash((key))
#define tuple_update_uhash(acc, lane) \
Copy link
Member

@WillAyd WillAyd Mar 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK need to be a bit careful here. Remember that C has no concept of namespaces - if you define tuple_update_hash here and also in khash.h, then whichever code being inserted by the pre-processor will depend upon whichever one of those files was last included, which can be very confusing.

With the capsule, the function that you ultimately include as part of the PandasDateTime_CAPI struct should be defined as static within the pd_datetime.c module, and the macro which resolves to that should have a unique name

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tuple_update_hash is accessed in khash_python.h, is only defined in np_datetime.c AFAICT

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Header files are loosely just copied into whatever they are included into. So you are doing #include khash_python.h with one declaration of tuple_update_uhash and then doing #include pd_datetime.h which has a different notion of what tuple_update_uhash is, your target ultimately gets two different ideas for what tuple_update_uhash means. That's basically what the current build failures are telling you

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do they have a different notion of what tuple_update_uhash is? It's only defined in one place.

Opened #51951 to revert the capsuling.

Copy link
Member

@WillAyd WillAyd Mar 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have a declaration for tuple_update_uhash in khash_python.h then try to define it via a macro in pd_datetime.h

I understand it is frustrating but the implementation here is not correct. Have you checked out any books on C programming? K&R is a nice book for a quick intro - might be worth reading through that or another book before coming back to this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah sorry not khash_python looks like np_datetime

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, so if i remove it from np_datetime.h i get build failure locally error: implicit declaration of function 'tuple_update_uhash' is invalid in C99

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's right. The way the includes happen and the way you have things structured now there you are either providing conflicting declarations for tuple_update_uhash for files that aren't pd_datetime.c OR you are providing no declaration for pd_datetime.c. You would probably be much better served if you had your hashing functions defined as static functions within the capsule rather than your current structure.

I really think you'd be better off getting an intro to C book and learning a bit more about what headers/modules do with respect to the different compilation phases. Of course always good to get hands-on experience, but doing some of the basics first before trying to dive in would probably make the process easier

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will I appreciate your encouragement but I have a long reading list and dayjob responsibilities. Can you just tell me how to fix the [expletive deleted] thing that was working a week ago?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Move your hashing functions to the capsule as static functions

PandasDateTimeAPI->tuple_update_uhash((acc), (lane))
#endif /* !defined(_PANDAS_DATETIME_IMPL) */

#ifdef __cplusplus
Expand Down
Morty Proxy This is a proxified and sanitized view of the page, visit original site.