Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

(3.12+) tokenize exception type is wrong #105390

Copy link
Copy link
Closed
Closed
(3.12+) tokenize exception type is wrong#105390
Copy link
Assignees
pablogsal
Labels
3.12only security fixesonly security fixes3.13bugs and security fixesbugs and security fixestype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
@asottile

Description

@asottile
asottile
opened on Jun 6, 2023
Issue body actions

Bug report

the document exception type for the tokenize.* functions is TokenError -- however it appears it now raises SyntaxError: https://docs.python.org/3.12/library/tokenize.html#tokenize.TokenError

Raised when either a docstring or expression that may be split over several lines is not completed anywhere in the file, for example:

import io
import tokenize

list(tokenize.generate_tokens(io.StringIO('"""').readline))
$ python3.11 t.py
Traceback (most recent call last):
  File "/home/asottile/workspace/switch-microcontroller/t.py", line 4, in <module>
    list(tokenize.generate_tokens(io.StringIO('"""').readline))
  File "/usr/lib/python3.11/tokenize.py", line 465, in _tokenize
    raise TokenError("EOF in multi-line string", strstart)
tokenize.TokenError: ('EOF in multi-line string', (1, 0))

and

#  python3.13 t.py
Traceback (most recent call last):
  File "/y/pyupgrade/t.py", line 4, in <module>
    list(tokenize.generate_tokens(io.StringIO('"""').readline))
  File "/usr/lib/python3.13/tokenize.py", line 526, in _generate_tokens_from_c_tokenizer
    for info in it:
  File "<string>", line 1
    """
    ^
SyntaxError: unterminated triple-quoted string literal (detected at line 1)

Your environment

  • CPython versions tested on: c7bf74b
  • Operating system and architecture: ubuntu 22.04 x86_64

Linked PRs

Reactions are currently unavailable

Metadata

Metadata

Assignees

Labels

3.12only security fixesonly security fixes3.13bugs and security fixesbugs and security fixestype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions
    Morty Proxy This is a proxified and sanitized view of the page, visit original site.