Bug report

Bug description:

OSS-Fuzz has found a heap buffer overflow in _PyTokenizer_ensure_utf8. Link to OSS-Fuzz bug report.

The root cause is that valid_utf8() in Parser/tokenizer/helpers.c checks continuation bytes in reverse order thus reader s[expected] before s[1] on these lines:

When a multi-byte UTF-8 sequence is truncated - such as a 3-byte lead \xEA followed immediately by a null terminator - the backward loop reads past the end of the valid data before encountering the null byte that would stop it.

This is not a security-critical issue.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-144872: fix heap buffer overflow _PyTokenizer_ensure_utf8 #144807
• [3.14] gh-144872: fix heap buffer overflow _PyTokenizer_ensure_utf8 (GH-144807) #145287
• [3.13] gh-144872: fix heap buffer overflow _PyTokenizer_ensure_utf8 (GH-144807) #145441