Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 1745629

Browse filesBrowse files
JelleZijlstraPrivat33r-dev
JelleZijlstra
authored and
Privat33r-dev
committed
gh-105858: Improve AST node constructors (#105880)
Demonstration: >>> ast.FunctionDef.__annotations__ {'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]} >>> ast.FunctionDef() <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15. <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15. <ast.FunctionDef object at 0x101959460> >>> node = ast.FunctionDef(name="foo", args=ast.arguments()) >>> node.decorator_list [] >>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments()) <stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15. <ast.FunctionDef object at 0x1019581f0>
1 parent beae246 commit 1745629
Copy full SHA for 1745629

File tree

Expand file treeCollapse file tree

10 files changed

+4676
-50
lines changed
Filter options
Expand file treeCollapse file tree

10 files changed

+4676
-50
lines changed

‎Doc/library/ast.rst

Copy file name to clipboardExpand all lines: Doc/library/ast.rst
+14-11Lines changed: 14 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -103,20 +103,15 @@ Node classes
103103
For example, to create and populate an :class:`ast.UnaryOp` node, you could
104104
use ::
105105

106-
node = ast.UnaryOp()
107-
node.op = ast.USub()
108-
node.operand = ast.Constant()
109-
node.operand.value = 5
110-
node.operand.lineno = 0
111-
node.operand.col_offset = 0
112-
node.lineno = 0
113-
node.col_offset = 0
114-
115-
or the more compact ::
116-
117106
node = ast.UnaryOp(ast.USub(), ast.Constant(5, lineno=0, col_offset=0),
118107
lineno=0, col_offset=0)
119108

109+
If a field that is optional in the grammar is omitted from the constructor,
110+
it defaults to ``None``. If a list field is omitted, it defaults to the empty
111+
list. If any other field is omitted, a :exc:`DeprecationWarning` is raised
112+
and the AST node will not have this field. In Python 3.15, this condition will
113+
raise an error.
114+
120115
.. versionchanged:: 3.8
121116

122117
Class :class:`ast.Constant` is now used for all constants.
@@ -140,6 +135,14 @@ Node classes
140135
In the meantime, instantiating them will return an instance of
141136
a different class.
142137

138+
.. deprecated-removed:: 3.13 3.15
139+
140+
Previous versions of Python allowed the creation of AST nodes that were missing
141+
required fields. Similarly, AST node constructors allowed arbitrary keyword
142+
arguments that were set as attributes of the AST node, even if they did not
143+
match any of the fields of the AST node. This behavior is deprecated and will
144+
be removed in Python 3.15.
145+
143146
.. note::
144147
The descriptions of the specific node classes displayed here
145148
were initially adapted from the fantastic `Green Tree

‎Doc/whatsnew/3.13.rst

Copy file name to clipboardExpand all lines: Doc/whatsnew/3.13.rst
+15Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,21 @@ array
206206
ast
207207
---
208208

209+
* The constructors of node types in the :mod:`ast` module are now stricter
210+
in the arguments they accept, and have more intuitive behaviour when
211+
arguments are omitted.
212+
213+
If an optional field on an AST node is not included as an argument when
214+
constructing an instance, the field will now be set to ``None``. Similarly,
215+
if a list field is omitted, that field will now be set to an empty list.
216+
(Previously, in both cases, the attribute would be missing on the newly
217+
constructed AST node instance.)
218+
219+
If other arguments are omitted, a :exc:`DeprecationWarning` is emitted.
220+
This will cause an exception in Python 3.15. Similarly, passing a keyword
221+
argument that does not map to a field on the AST node is now deprecated,
222+
and will raise an exception in Python 3.15.
223+
209224
* :func:`ast.parse` now accepts an optional argument ``optimize``
210225
which is passed on to the :func:`compile` built-in. This makes it
211226
possible to obtain an optimized ``AST``.

‎Include/internal/pycore_global_objects_fini_generated.h

Copy file name to clipboardExpand all lines: Include/internal/pycore_global_objects_fini_generated.h
+1Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Include/internal/pycore_global_strings.h

Copy file name to clipboardExpand all lines: Include/internal/pycore_global_strings.h
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,7 @@ struct _Py_global_strings {
242242
STRUCT_FOR_ID(_check_retval_)
243243
STRUCT_FOR_ID(_dealloc_warn)
244244
STRUCT_FOR_ID(_feature_version)
245+
STRUCT_FOR_ID(_field_types)
245246
STRUCT_FOR_ID(_fields_)
246247
STRUCT_FOR_ID(_finalizing)
247248
STRUCT_FOR_ID(_find_and_load)

‎Include/internal/pycore_runtime_init_generated.h

Copy file name to clipboardExpand all lines: Include/internal/pycore_runtime_init_generated.h
+1Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Include/internal/pycore_unicodeobject_generated.h

Copy file name to clipboardExpand all lines: Include/internal/pycore_unicodeobject_generated.h
+3Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Lib/test/test_ast.py

Copy file name to clipboardExpand all lines: Lib/test/test_ast.py
+76-25Lines changed: 76 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -525,17 +525,38 @@ def test_field_attr_existence(self):
525525
if name == 'Index':
526526
continue
527527
if self._is_ast_node(name, item):
528-
x = item()
528+
x = self._construct_ast_class(item)
529529
if isinstance(x, ast.AST):
530530
self.assertIs(type(x._fields), tuple)
531531

532+
def _construct_ast_class(self, cls):
533+
kwargs = {}
534+
for name, typ in cls.__annotations__.items():
535+
if typ is str:
536+
kwargs[name] = 'capybara'
537+
elif typ is int:
538+
kwargs[name] = 42
539+
elif typ is object:
540+
kwargs[name] = b'capybara'
541+
elif isinstance(typ, type) and issubclass(typ, ast.AST):
542+
kwargs[name] = self._construct_ast_class(typ)
543+
return cls(**kwargs)
544+
532545
def test_arguments(self):
533546
x = ast.arguments()
534547
self.assertEqual(x._fields, ('posonlyargs', 'args', 'vararg', 'kwonlyargs',
535548
'kw_defaults', 'kwarg', 'defaults'))
536-
537-
with self.assertRaises(AttributeError):
538-
x.args
549+
self.assertEqual(x.__annotations__, {
550+
'posonlyargs': list[ast.arg],
551+
'args': list[ast.arg],
552+
'vararg': ast.arg | None,
553+
'kwonlyargs': list[ast.arg],
554+
'kw_defaults': list[ast.expr],
555+
'kwarg': ast.arg | None,
556+
'defaults': list[ast.expr],
557+
})
558+
559+
self.assertEqual(x.args, [])
539560
self.assertIsNone(x.vararg)
540561

541562
x = ast.arguments(*range(1, 8))
@@ -551,7 +572,7 @@ def test_field_attr_writable_deprecated(self):
551572
self.assertEqual(x._fields, 666)
552573

553574
def test_field_attr_writable(self):
554-
x = ast.Constant()
575+
x = ast.Constant(1)
555576
# We can assign to _fields
556577
x._fields = 666
557578
self.assertEqual(x._fields, 666)
@@ -611,15 +632,22 @@ def test_classattrs_deprecated(self):
611632

612633
self.assertEqual([str(w.message) for w in wlog], [
613634
'ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead',
635+
"Constant.__init__ missing 1 required positional argument: 'value'. This will become "
636+
'an error in Python 3.15.',
614637
'Attribute n is deprecated and will be removed in Python 3.14; use value instead',
615638
'ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead',
616639
'Attribute n is deprecated and will be removed in Python 3.14; use value instead',
617640
'ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead',
641+
"Constant.__init__ missing 1 required positional argument: 'value'. This will become "
642+
'an error in Python 3.15.',
618643
'ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead',
619644
'Attribute n is deprecated and will be removed in Python 3.14; use value instead',
620645
'ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead',
621646
'ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead',
622647
'ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead',
648+
"Constant.__init__ got an unexpected keyword argument 'foo'. Support for "
649+
'arbitrary keyword arguments is deprecated and will be removed in Python '
650+
'3.15.',
623651
'ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead',
624652
'Attribute n is deprecated and will be removed in Python 3.14; use value instead',
625653
'ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead',
@@ -636,7 +664,8 @@ def test_classattrs_deprecated(self):
636664
])
637665

638666
def test_classattrs(self):
639-
x = ast.Constant()
667+
with self.assertWarns(DeprecationWarning):
668+
x = ast.Constant()
640669
self.assertEqual(x._fields, ('value', 'kind'))
641670

642671
with self.assertRaises(AttributeError):
@@ -651,7 +680,7 @@ def test_classattrs(self):
651680
with self.assertRaises(AttributeError):
652681
x.foobar
653682

654-
x = ast.Constant(lineno=2)
683+
x = ast.Constant(lineno=2, value=3)
655684
self.assertEqual(x.lineno, 2)
656685

657686
x = ast.Constant(42, lineno=0)
@@ -662,8 +691,9 @@ def test_classattrs(self):
662691
self.assertRaises(TypeError, ast.Constant, 1, None, 2)
663692
self.assertRaises(TypeError, ast.Constant, 1, None, 2, lineno=0)
664693

665-
# Arbitrary keyword arguments are supported
666-
self.assertEqual(ast.Constant(1, foo='bar').foo, 'bar')
694+
# Arbitrary keyword arguments are supported (but deprecated)
695+
with self.assertWarns(DeprecationWarning):
696+
self.assertEqual(ast.Constant(1, foo='bar').foo, 'bar')
667697

668698
with self.assertRaisesRegex(TypeError, "Constant got multiple values for argument 'value'"):
669699
ast.Constant(1, value=2)
@@ -815,11 +845,11 @@ def test_isinstance(self):
815845
assertBytesDeprecated(self.assertNotIsInstance, Constant('42'), Bytes)
816846
assertNameConstantDeprecated(self.assertNotIsInstance, Constant(42), NameConstant)
817847
assertEllipsisDeprecated(self.assertNotIsInstance, Constant(42), Ellipsis)
818-
assertNumDeprecated(self.assertNotIsInstance, Constant(), Num)
819-
assertStrDeprecated(self.assertNotIsInstance, Constant(), Str)
820-
assertBytesDeprecated(self.assertNotIsInstance, Constant(), Bytes)
821-
assertNameConstantDeprecated(self.assertNotIsInstance, Constant(), NameConstant)
822-
assertEllipsisDeprecated(self.assertNotIsInstance, Constant(), Ellipsis)
848+
assertNumDeprecated(self.assertNotIsInstance, Constant(None), Num)
849+
assertStrDeprecated(self.assertNotIsInstance, Constant(None), Str)
850+
assertBytesDeprecated(self.assertNotIsInstance, Constant(None), Bytes)
851+
assertNameConstantDeprecated(self.assertNotIsInstance, Constant(1), NameConstant)
852+
assertEllipsisDeprecated(self.assertNotIsInstance, Constant(None), Ellipsis)
823853

824854
class S(str): pass
825855
with assertStrDeprecated():
@@ -888,8 +918,9 @@ def test_module(self):
888918
self.assertEqual(x.body, body)
889919

890920
def test_nodeclasses(self):
891-
# Zero arguments constructor explicitly allowed
892-
x = ast.BinOp()
921+
# Zero arguments constructor explicitly allowed (but deprecated)
922+
with self.assertWarns(DeprecationWarning):
923+
x = ast.BinOp()
893924
self.assertEqual(x._fields, ('left', 'op', 'right'))
894925

895926
# Random attribute allowed too
@@ -927,8 +958,9 @@ def test_nodeclasses(self):
927958
self.assertEqual(x.right, 3)
928959
self.assertEqual(x.lineno, 0)
929960

930-
# Random kwargs also allowed
931-
x = ast.BinOp(1, 2, 3, foobarbaz=42)
961+
# Random kwargs also allowed (but deprecated)
962+
with self.assertWarns(DeprecationWarning):
963+
x = ast.BinOp(1, 2, 3, foobarbaz=42)
932964
self.assertEqual(x.foobarbaz, 42)
933965

934966
def test_no_fields(self):
@@ -941,8 +973,9 @@ def test_pickling(self):
941973

942974
for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
943975
for ast in (compile(i, "?", "exec", 0x400) for i in exec_tests):
944-
ast2 = pickle.loads(pickle.dumps(ast, protocol))
945-
self.assertEqual(to_tuple(ast2), to_tuple(ast))
976+
with self.subTest(ast=ast, protocol=protocol):
977+
ast2 = pickle.loads(pickle.dumps(ast, protocol))
978+
self.assertEqual(to_tuple(ast2), to_tuple(ast))
946979

947980
def test_invalid_sum(self):
948981
pos = dict(lineno=2, col_offset=3)
@@ -1310,8 +1343,9 @@ def test_copy_location(self):
13101343
'lineno=1, col_offset=4, end_lineno=1, end_col_offset=5), lineno=1, '
13111344
'col_offset=0, end_lineno=1, end_col_offset=5))'
13121345
)
1313-
src = ast.Call(col_offset=1, lineno=1, end_lineno=1, end_col_offset=1)
1314-
new = ast.copy_location(src, ast.Call(col_offset=None, lineno=None))
1346+
func = ast.Name('spam', ast.Load())
1347+
src = ast.Call(col_offset=1, lineno=1, end_lineno=1, end_col_offset=1, func=func)
1348+
new = ast.copy_location(src, ast.Call(col_offset=None, lineno=None, func=func))
13151349
self.assertIsNone(new.end_lineno)
13161350
self.assertIsNone(new.end_col_offset)
13171351
self.assertEqual(new.lineno, 1)
@@ -1570,15 +1604,15 @@ def test_level_as_none(self):
15701604
self.assertIn('sleep', ns)
15711605

15721606
def test_recursion_direct(self):
1573-
e = ast.UnaryOp(op=ast.Not(), lineno=0, col_offset=0)
1607+
e = ast.UnaryOp(op=ast.Not(), lineno=0, col_offset=0, operand=ast.Constant(1))
15741608
e.operand = e
15751609
with self.assertRaises(RecursionError):
15761610
with support.infinite_recursion():
15771611
compile(ast.Expression(e), "<test>", "eval")
15781612

15791613
def test_recursion_indirect(self):
1580-
e = ast.UnaryOp(op=ast.Not(), lineno=0, col_offset=0)
1581-
f = ast.UnaryOp(op=ast.Not(), lineno=0, col_offset=0)
1614+
e = ast.UnaryOp(op=ast.Not(), lineno=0, col_offset=0, operand=ast.Constant(1))
1615+
f = ast.UnaryOp(op=ast.Not(), lineno=0, col_offset=0, operand=ast.Constant(1))
15821616
e.operand = f
15831617
f.operand = e
15841618
with self.assertRaises(RecursionError):
@@ -2866,6 +2900,23 @@ def visit_Call(self, node: ast.Call):
28662900
self.assertASTTransformation(PrintToLog, code, expected)
28672901

28682902

2903+
class ASTConstructorTests(unittest.TestCase):
2904+
"""Test the autogenerated constructors for AST nodes."""
2905+
2906+
def test_FunctionDef(self):
2907+
args = ast.arguments()
2908+
self.assertEqual(args.args, [])
2909+
self.assertEqual(args.posonlyargs, [])
2910+
with self.assertWarnsRegex(DeprecationWarning,
2911+
r"FunctionDef\.__init__ missing 1 required positional argument: 'name'"):
2912+
node = ast.FunctionDef(args=args)
2913+
self.assertFalse(hasattr(node, "name"))
2914+
self.assertEqual(node.decorator_list, [])
2915+
node = ast.FunctionDef(name='foo', args=args)
2916+
self.assertEqual(node.name, 'foo')
2917+
self.assertEqual(node.decorator_list, [])
2918+
2919+
28692920
@support.cpython_only
28702921
class ModuleStateTests(unittest.TestCase):
28712922
# bpo-41194, bpo-41261, bpo-41631: The _ast module uses a global state.

‎Misc/NEWS.d/next/Core and Builtins/2023-06-16-21-29-06.gh-issue-105858.Q7h0EV.rst

Copy file name to clipboard
+8Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
Improve the constructors for :mod:`ast` nodes. Arguments of list types now
2+
default to an empty list if omitted, and optional fields default to ``None``.
3+
AST nodes now have an
4+
``__annotations__`` attribute with the expected types of their attributes.
5+
Passing unrecognized extra arguments to AST nodes is deprecated and will
6+
become an error in Python 3.15. Omitting a required argument to an AST node
7+
is deprecated and will become an error in Python 3.15. Patch by Jelle
8+
Zijlstra.

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.