Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

dict_traverse crash during PGO training on Windows with JIT enabled #144970

Copy link
Copy link
@savannahostrowski

Description

@savannahostrowski
Issue body actions

Bug report

Bug description:

Building CPython at e779777 with PCbuild/build.bat -c Release --pgo --experimental-jit can crash during the PGO training phase. The instrumented interpreter hits an access violation in dict_traverse → visit_decref → _PyObject_IS_GC with ob = 0xffffffffffffffff (-1). The last successful build was at 2f7634c (the prior night's run; I suspect this has to do with #144757).

I caught this as part of the nightly run for the JIT benchmark website (see https://github.com/savannahostrowski/pyperf_bench/actions/runs/22133822860/job/64045676082) on my Windows runner (x86_64, Windows 11). Reproduces under a Windows service account (NETWORK SERVICE); appears to be timing-dependent.

Details 0:000> !analyze -v ******************************************************************************* * * * Exception Analysis * * * *******************************************************************************

KEY_VALUES_STRING: 1

Key  : AV.Type
Value: Read

Key  : Analysis.CPU.mSec
Value: 375

Key  : Analysis.Elapsed.mSec
Value: 577

Key  : Analysis.IO.Other.Mb
Value: 0

Key  : Analysis.IO.Read.Mb
Value: 20

Key  : Analysis.IO.Write.Mb
Value: 56

Key  : Analysis.Init.CPU.mSec
Value: 1718

Key  : Analysis.Init.Elapsed.mSec
Value: 36703

Key  : Analysis.Memory.CommitPeak.Mb
Value: 111

Key  : Analysis.Version.DbgEng
Value: 10.0.29507.1001

Key  : Analysis.Version.Description
Value: 10.2511.5.1 amd64fre

Key  : Analysis.Version.Ext
Value: 1.2511.5.1

Key  : Failure.Bucket
Value: INVALID_POINTER_READ_c0000005_python315.dll!_PyObject_IS_GC

Key  : Failure.Exception.Code
Value: 0xc0000005

Key  : Failure.Exception.IP.Address
Value: 0x7ffc5f3395a0

Key  : Failure.Exception.IP.Module
Value: python315

Key  : Failure.Exception.IP.Offset
Value: 0x395a0

Key  : Failure.Hash
Value: {41ad561c-4f71-20b0-5dcb-a02ca0bc2c99}

Key  : Failure.ProblemClass.Primary
Value: INVALID_POINTER_READ

Key  : Faulting.IP.Type
Value: Paged

Key  : Timeline.OS.Boot.DeltaSec
Value: 405382

Key  : Timeline.Process.Start.DeltaSec
Value: 3

Key  : WER.OS.Branch
Value: ge_release

Key  : WER.OS.Version
Value: 10.0.26100.1

Key  : WER.Process.Version
Value: 3.15.106.1013

FILE_IN_CAB: python.exe.3948.dmp

NTGLOBALFLAG: 0

APPLICATION_VERIFIER_FLAGS: 0

CONTEXT: (.ecxr)
rax=000001b2fdf20130 rbx=ffffffffffffffff rcx=ffffffffffffffff
rdx=000001b2fdf08c00 rsi=0000000000000027 rdi=000001b2fdf08c00
rip=00007ffc5f3395a0 rsp=0000000dd217c490 rbp=000001b2fdf08c00
r8=00007ffc5fbd1a80 r9=ffffffffffffffff r10=00007ffcd2895980
r11=00007ffcd2893870 r12=000001b2fa194d00 r13=00007ffc5fcb4c80
r14=00007ffc5f79c9c0 r15=00007ffc5fce9db8
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
python315!_Py_TYPE_impl [inlined in python315!_PyObject_IS_GC+0x20]:
00007ffc5f3395a0 4c8b4108 mov r8,qword ptr [rcx+8] ds:0000000000000007=????????????????
Resetting default scope

EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ffc5f3395a0 (python315!_Py_TYPE_impl)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 0000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 0000000000000007
Attempt to read from address 0000000000000007

PROCESS_NAME: python.exe

READ_ADDRESS: 0000000000000007

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR: c0000005

EXCEPTION_PARAMETER1: 0000000000000000

EXCEPTION_PARAMETER2: 0000000000000007

STACK_TEXT:
(Inline Function) ---------------- : ---------------- ---------------- ---------------- ---------------- : python315!_Py_TYPE_impl 0000000dd217c490 00007ffc5f79c9f5 : 00007ffc5fc67980 00007ffc5fbd1a80 000001b2fdf08c00 ffffffffffffffff : python315!_PyObject_IS_GC+0x20 0000000dd217c4d0 00007ffc5f683fcb : 0000000000000003 000000000000000 000000000000000 000000000000000 : python315!visit_decref+0x35 0000000dd217c500 00007ffc5f36f42c : 000001b2fdf08bf 0000000dd217c708 0000000dd217c6f8 0000000dd217c6f8 : python315!dict_traverse+0xab 0000000dd217c570 00007ffc5f389fd5 : 000000000000085f 0000000dd217c648 0000000dd217c788 0000000dd217c708 : python315!subtract_refs+0x7c 0000000dd217c5c0 00007ffc5f443b38 : 0000000dd217c6f8 00007ffc5fcb2fa0 00007ffc5fce9db8 000000000000000 : python315!deduce_unreachable+0x65 0000000dd217c600 00007ffc5f44de15 : 00007ffc5fcb4c80 00007ffc5fce9db8 000000000000000 00007ffc5fcb4c80 : python315!gc_collect_region+0x78 0000000dd217c6c0 00007ffc5f44f0e6 : 000000000000000 000000000000000 00007ffc5fcb4c80 000000000000000 : python315!gc_collect_increment+0x285 0000000dd217c760 00007ffc5f44fd6a : 000000000000000 00007ffc5fce9db8 000000000000000 000170b127ba8994 : python315!_PyGC_Collect+0x1a6 0000000dd217c7f0 00007ffc5f4500c0 : 000000000000000 000000000000000 000000000000000 000001b2f94b7c20 : python315!_Py_RunGC+0x4a 0000000dd217c830 00007ffc5f45028e : 000001b2fe56ec51 000001b2fe9d9100 000001b2fdf20951 000000000000002 : python315!_Py_HandlePending+0x140 0000000dd217c870 00007ffc5f766de9 : 000001b2fe56ec51 000001b2fdf0c7b0 0000000000000008 000000000000000 : python315!check_periodics+0x3e 0000000dd217c8b0 00007ffc5f4504e5 : 000000001ab78c8b 000000000000000 0000000dd217cbf0 000000000000002 : python315!_PyEval_EvalFrameDefault+0x68cd 0000000dd217cb4 00007ffc5f450891 : 0000000dd217cbf0 000001b2f94e9f90 000000000000000 0000000dd217cd68 : python315!_PyEval_EvalFrame+0x35 0000000dd217cb80 00007ffc5f673e28 : 000000000000000 000000000000000 000000000000000 00007ffc5fce9db8 : python315!_PyEval_Vector+0x291 0000000dd217cc80 00007ffc5f4ae45e : 000001b2fdeffcc0 000000000000002 000000000000000 000000000000000 : python315!_PyFunction_Vectorcall+0x88 0000000dd217ccc0 00007ffc5f3ea92b : 0000000000000002 0000000dd217ce30 000000000000000 0000000dd217cd68 : python315!_PyObject_VectorcallTstate+0xbe 0000000dd217cd10 00007ffc5f673cf9 : 000001b2fde7bf40 000000000000000 000000000000000 000000000000000 : python315!object_vacall+0x1ab 0000000dd217cdd0 00007ffc5f441d83 : 000000000000000 00007ffc5fca7760 00007ffc5fcaec48 000001b2fdf20630 : python315!PyObject_CallMethodObjArgs+0x109 0000000dd217ce20 00007ffc5f7a5ce4 : 000000000000000 000000000000000 000000000000000 00007ffc5fc99478 : python315!import_find_and_load+0xf3 0000000dd217ceb0 00007ffc5f4429dd : 00007ffc5fcb2fa0 000001b2fe53e1c0 000001b2fe53e1c0 00000001b11c0c7 : python315!PyImport_ImportModuleLevelObject+0x384 0000000dd217cf20 00007ffc5f442b4d : 00007ffc5fcaec48 000000000000000 000000000000000 000000000000000 : python315!_PyEval_ImportNameWithImport+0xed 0000000dd217cfd0 00007ffc5f448104 : 00007ffc5fcaec48 000000000000000 00007ffc5fce9db8 000001b2fe53e1c0 : python315!_PyEval_ImportName+0xed 0000000dd217d030 00007ffc5f763fc1 : 000001b2fde7b640 00007ffc5fce9db8 00007ffc5fc99400 000001b2fe451f90 : python315!_PyEval_LazyImportName+0x124 0000000dd217d120 00007ffc5f4504e5 : 000000001b16c513 000000000000000 0000000dd217d460 000000000000000 : python315!_PyEval_EvalFrameDefault+0x3aa5 0000000dd217d3b0 00007ffc5f450891 : 0000000dd217d460 000001b2fe27f420 000000000000000 000000000000000 : python315!_PyEval_EvalFrame+0x35 0000000dd217d3f0 00007ffc5f450a79 : 000001b2fe0ef9b0 000001b2fe53e1c0 000001b2fe53e1c0 000001b2fea71170 : python315!_PyEval_Vector+0x291 0000000dd217d4f0 00007ffc5f7591a9 : 000001b2fe53e1c0 000000000000000 000001b2fe53e1c0 00000000000000f : python315!PyEval_EvalCode+0x199 0000000dd217d590 00007ffc5f758dc0 : 000000000000000 000000000000000 000001b2f95e0871 00007ffc5f8b1946 : python315!builtin_exec_impl+0x3a9 0000000dd217d640 00007ffc5f6b00f5 : 0000000000000002 000000000000000 000000000000000 000000000000002 : python315!builtin_exec+0x150 0000000dd217d6f 00007ffc5f3e37c8 : 00007ffc5f6b0030 000001b2fdf20ae0 0000000000000002 00007ffc5f3681c1 : python315!cfunction_vectorcall_FASTCALL_KEYWORDS+0xc5 0000000dd217d730 00007ffc5f495c61 : 000001b2fdf20ae0 000001b2fe9d94c0 000001b2fdf20ae1 000000000000000 : python315!_PyVectorcall_Call+0x148 0000000dd217d790 00007ffc5f496b6c : 0000000000000010 000000000000000 000001b2fdf20ae1 000000000000002 : python315!_PyObject_Call+0x81 0000000dd217d7f0 00007ffc5f766d33 : 000001b2fea540c1 000001b2fdf0c7b0 0000000000000008 000000000000000 : python315!PyObject_Call+0x5c 0000000dd217d830 00007ffc5f4504e5 : 000000001ab78c8b 000001b2fe5700c0 0000000dd217db70 000000000000002 : python315!_PyEval_EvalFrameDefault+0x6817 0000000dd217dac0 00007ffc5f450891 : 0000000dd217db70 000001b2f94e9f90 000000000000000 0000000dd217dce8 : python315!_PyEval_EvalFrame+0x35 0000000dd217db00 00007ffc5f673e28 : 000001b2fdf6f480 000001b2fe5700c0 000000000000000 00007ffc5fce9db8 : python315!_PyEval_Vector+0x291 0000000dd217dc00 00007ffc5f4ae45e : 000001b2fdeffcc0 000000000000002 000001b2fdf6f480 000001b2fe5700c0 : python315!_PyFunction_Vectorcall+0x88 0000000dd217dc40 00007ffc5f3ea92b : 0000000000000002 0000000dd217ddb0 000000000000000 0000000dd217dce8 : python315!_PyObject_VectorcallTstate+0xbe 0000000dd217dc90 00007ffc5f673cf9 : 000001b2fde7bf40 000000000000000 000000000000000 000000000000000 : python315!object_vacall+0x1ab 0000000dd217dd5 00007ffc5f441d83 : 000001b2fe5538b0 00007ffc5fca7760 000001b2fe0ef9b0 000001b2fdf20630 : python315!PyObject_CallMethodObjArgs+0x109 0000000dd217dda0 00007ffc5f7a5ce4 : 000000000000000 000000000000000 000000000000000 00007ffc5fc99478 : python315!import_find_and_load+0xf3 0000000dd217de3 00007ffc5f4429dd : 00007ffc5fcb2fa0 000001b2fe1c5dc0 000001b2fe1c5dc0 00000001b11c0c7 : python315!PyImport_ImportModuleLevelObject+0x384 0000000dd217dea0 00007ffc5f442b4d : 000001b2fe0ef9b0 000001b2fea70a90 000001b2fe5538b0 000001b2fea70bf0 : python315!_PyEval_ImportNameWithImport+0xed 0000000dd217df50 00007ffc5f448104 : 000001b2fe0ef9b0 000000000000000 00007ffc5fce9db8 000001b2fe1c5dc0 : python315!_PyEval_ImportName+0xed 0000000dd217dfb0 00007ffc5f763fc1 : 000001b2fde7b640 00007ffc5fce9db8 00007ffc5fc99400 000001b2f951d670 : python315!_PyEval_LazyImportName+0x124 0000000dd217e0a0 00007ffc5f4504e5 : 000000001b16c513 000000000000000 0000000dd217e3e0 000000000000000 : python315!_PyEval_EvalFrameDefault+0x3aa5 0000000dd217e330 00007ffc5f450891 : 0000000dd217e3e0 000001b2fe1dab40 000000000000000 000000000000000 : python315!_PyEval_EvalFrame+0x35 0000000dd217e370 00007ffc5f450a79 : 000001b2fe1c5470 000001b2fe1c5dc0 000001b2fe1c5dc0 000001b2fe4d2770 : python315!_PyEval_Vector+0x291 0000000dd217e47 00007ffc5f7591a9 : 000001b2fe1c5dc0 000000000000000 000001b2fe1c5dc0 00000000000000f : python315!PyEval_EvalCode+0x199 0000000dd217e510 00007ffc5f758dc0 : 000000000000000 000000000000000 000001b2f95e0871 00007ffc5f8b1946 : python315!builtin_exec_impl+0x3a9 0000000dd217e5c0 00007ffc5f6b00f5 : 0000000000000002 000000000000000 000000000000000 000000000000002 : python315!builtin_exec+0x150 0000000dd217e670 00007ffc5f3e37c8 : 00007ffc5f6b0030 000001b2fdf20ae0 0000000000000002 00007ffc5f3681c1 : python315!cfunction_vectorcall_FASTCALL_KEYWORDS+0xc5 0000000dd217e6b0 00007ffc5f495c61 : 000001b2fdf20ae0 000001b2fe5330c0 000001b2fdf20ae1 000000000000000 : python315!_PyVectorcall_Call+0x148 0000000dd217e710 00007ffc5f496b6c : 0000000000000010 000000000000000 000001b2fdf20ae1 000000000000002 : python315!_PyObject_Call+0x81 0000000dd217e770 00007ffc5f766d33 : 000001b2fe528201 000001b2fdf0c7b0 0000000000000008 000000000000000 : python315!PyObject_Call+0x5c 0000000dd217e7b0 00007ffc5f4504e5 : 000000001ab78c8b 5f5f6e69616d5f5f 0000000dd217eaf0 000000000000002 : python315!_PyEval_EvalFrameDefault+0x6817 0000000dd217ea40 00007ffc5f450891 : 0000000dd217eaf0 000001b2f94e9f90 000000000000000 0000000dd217ec68 : python315!_PyEval_EvalFrame+0x35 0000000dd217ea8 00007ffc5f673e28 : 5c5f5f6568636163 5f5f6e69616d5f5f 000000000000000 00007ffc5fce9db8 : python315!_PyEval_Vector+0x291 0000000dd217eb8 00007ffc5f4ae45e : 000001b2fdeffcc0 000000000000002 5c5f5f6568636163 5f5f6e69616d5f5f : python315!_PyFunction_Vectorcall+0x88 0000000dd217ebc0 00007ffc5f3ea92b : 0000000000000002 0000000dd217ed30 000000000000000 0000000dd217ec68 : python315!_PyObject_VectorcallTstate+0xbe 0000000dd217ec10 00007ffc5f673cf9 : 000001b2fde7bf40 000000000000000 000000000000000 000000000000000 : python315!object_vacall+0x1ab 0000000dd217ecd0 00007ffc5f441d83 : 000000000000000 00007ffc5fca7760 000001b2fe1c5470 000001b2fdf20630 : python315!PyObject_CallMethodObjArgs+0x109 0000000dd217ed20 00007ffc5f7a5ce4 : 000000000000000 000000000000000 000000000000000 00007ffc5fc99478 : python315!import_find_and_load+0xf3 0000000dd217edb0 00007ffc5f4429dd : 00007ffc5fcb2fa0 000001b2fe15e840 000001b2fe15e840 00000001b11c0c7 : python315!PyImport_ImportModuleLevelObject+0x384 0000000dd217ee20 00007ffc5f442b4d : 000001b2fe1c5470 000000000000000 000000000000000 000000000000000 : python315!_PyEval_ImportNameWithImport+0xed 0000000dd217eed0 00007ffc5f448104 : 000001b2fe1c5470 000000000000000 00007ffc5fce9db8 000001b2fe15e840 : python315!_PyEval_ImportName+0xed 0000000dd217ef30 00007ffc5f763fc1 : 000001b2f9620228 000001b2fdea86a0 0000000dd217f130 00007ffc5fce9db8 : python315!_PyEval_LazyImportName+0x124 0000000dd217f020 00007ffc5f4504e5 : 000000001b16c513 5f5f6e69616d5f5f 0000000dd217f360 000000000000000 : python315!_PyEval_EvalFrameDefault+0x3aa5 0000000dd217f2b0 00007ffc5f450891 : 0000000dd217f360 000001b2fe151730 000000000000000 000000000000000 : python315!_PyEval_EvalFrame+0x35 0000000dd217f2f0 00007ffc5f450a79 : 00007ffc5fca6408 000001b2fe15e840 000001b2fe15e840 000001b2fe13b320 : python315!_PyEval_Vector+0x291 0000000dd217f3f0 00007ffc5f7591a9 : 000001b2fe15e840 000000000000000 000001b2fe15e840 000001b2fe113580 : python315!PyEval_EvalCode+0x199 0000000dd217f490 00007ffc5f758dc0 : 000000000000000 000000000000000 000001b2f95e0871 00007ffc5f8b1946 : python315!builtin_exec_impl+0x3a9 0000000dd217f540 00007ffc5f6b00f5 : 0000000000000002 5f5f6e69616d5f5f 000000000000000 8000000000000002 : python315!builtin_exec+0x150 0000000dd217f5f0 00007ffc5f4ae45e : 000001b2fdf20ae0 00007ffc5fce9db8 0000000dd217f738 5f5f6e69616d5f5f : python315!cfunction_vectorcall_FASTCALL_KEYWORDS+0xc5 0000000dd217f630 00007ffc5f3eaa96 : 0000000000000005 000001b2fdf20ae0 000001b2f9620190 000000000000010 : python315!_PyObject_VectorcallTstate+0xbe 0000000dd217f680 00007ffc5f3f136f : 0000000000000001 000001b2fdf20ae0 000000000000000 000000000000000 : python315!PyObject_Vectorcall+0x56 0000000dd217f6d0 00007ffc5f769b92 : 000001b2fe15e841 000001b2f9620180 000000000000000 000001b2fde9d3b4 : python315!_Py_VectorCallInstrumentation_StackRefSteal+0xef 0000000dd217f7f 00007ffc5f4504e5 : 000000001ab78c8b 000001b2fe4d2350 0000000dd217fb30 000000000000002 : python315!_PyEval_EvalFrameDefault+0x9676 0000000dd217fa80 00007ffc5f450891 : 0000000dd217fb30 000001b2fe3af7f0 000000000000000 000001b2`fe1bba20 : python315!_PyEval_EvalFrame+0x35

STACK_COMMAND: ~0s; .ecxr ; kb

IP_IN_PAGED_CODE:
python315!_PyObject_IS_GC+20 [C:\actions-runner_work\pyperf_bench\pyperf_bench\cpython\Include\internal\pycore_object.h @ 834]
00007ffc`5f3395a0 4c8b4108 mov r8,qword ptr [rcx+8]

FAULTING_SOURCE_LINE: C:\actions-runner_work\pyperf_bench\pyperf_bench\cpython\Include\internal\pycore_object.h

FAULTING_SOURCE_FILE: C:\actions-runner_work\pyperf_bench\pyperf_bench\cpython\Include\internal\pycore_object.h

FAULTING_SOURCE_LINE_NUMBER: 834

FAULTING_SOURCE_CODE:
309:
310: static inline
311: PyTypeObject* _Py_TYPE_impl(PyObject *ob)
312: {

313: return ob->ob_type;
314: }
315:
316: // bpo-39573: The Py_SET_SIZE() function must be used to set an object size.
317: static inline Py_ssize_t
318: _Py_SIZE_impl(PyObject *ob)

SYMBOL_NAME: python315!_PyObject_IS_GC+20

MODULE_NAME: python315

IMAGE_NAME: python315.dll

FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_python315.dll!_PyObject_IS_GC

OS_VERSION: 10.0.26100.1

BUILDLAB_STR: ge_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

IMAGE_VERSION: 3.15.106.1013

FAILURE_ID_HASH: {41ad561c-4f71-20b0-5dcb-a02ca0bc2c99}

Followup: MachineOwner

CPython versions tested on:

CPython main branch

Operating systems tested on:

Windows

Reactions are currently unavailable

Metadata

Metadata

Labels

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    Morty Proxy This is a proxified and sanitized view of the page, visit original site.