Closed
Description
Bug report
Bug description:
Found when investigating test.test_tokenize.TestRoundtrip.test_random_files failing on #134577
def test(code):
tokens = list(tokenize.tokenize(iter([code]).__next__))
from5 = tokenize.untokenize(tokens)
print("from5 ", from5, eval(from5))
tokens2 = [tok[:2] for tok in tokens]
from2 = tokenize.untokenize(tokens2)
print("from2 ", from2, eval(from2))>>> test(b't"{ {} }"')
from5 b't"{ {} }"' Template(strings=('', ''), interpolations=(Interpolation({}, ' {}', None, ''),))
from2 b't"{{}}"' Template(strings=('{}',), interpolations=())
>>> test(b'f"{ {} }"')
from5 b'f"{ {} }"' {}
from2 b'f"{ {} }"' {}From what I understand, untokenize should round-trip correctly even with only the type and string of tokens.
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Linked PRs
Metadata
Metadata
Assignees
Labels
bugs and security fixesbugs and security fixesnew features, bugs and security fixesnew features, bugs and security fixes(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)An unexpected behavior, bug, or errorAn unexpected behavior, bug, or error