Do not remove tokens before AST json serialization #78610
Conversation
83db54e to
6b63e9b
Compare
|
I wanted to avoid making this serializable, to guarantee that we properly 'lower' a |
|
It doesn't matter much when exactly we force the real token stream creation? Lazy token streams just never survive until that time because all of them are in nonterminals where you have to do other things forcing token creation, like print-reparse checking, anyway. |
|
The I don't think there's a particularly clean way to only panic if we're encoding the |
|
📌 Commit 6b63e9b has been approved by |
Do not remove tokens before AST json serialization `TokenStripper` is error-prone and introduces one more use of `MutVisitor`. It's much simpler to treat serialization as just one more place that wants lazy token stream to turn into a real token stream. Also, no code is better than more code, in general. r? @Aaron1011 (I also merged tests for `TokenStripper` ICEs into one.)
Rollup of 5 pull requests Successful merges: - rust-lang#78606 (Clarify handling of final line ending in str::lines()) - rust-lang#78610 (Do not remove tokens before AST json serialization) - rust-lang#78620 (Trivial fixes to bitwise operator documentation) - rust-lang#78627 (Point out that total_cmp is no strict superset of partial comparison) - rust-lang#78637 (Add fetch_update methods to AtomicBool and AtomicPtr) Failed merges: r? `@ghost`
TokenStripperis error-prone and introduces one more use ofMutVisitor.It's much simpler to treat serialization as just one more place that wants lazy token stream to turn into a real token stream.
Also, no code is better than more code, in general.
r? @Aaron1011
(I also merged tests for
TokenStripperICEs into one.)