Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit 24b51b1

Browse filesBrowse files
miss-islingtonelprans
miss-islington
and
elprans
authored
bpo-34022: Stop forcing of hash-based invalidation with SOURCE_DATE_EPOCH (GH-9607)
Unconditional forcing of ``CHECKED_HASH`` invalidation was introduced in 3.7.0 in bpo-29708. The change is bad, as it unconditionally overrides *invalidation_mode*, even if it was passed as an explicit argument to ``py_compile.compile()`` or ``compileall``. An environment variable should *never* override an explicit argument to a library function. That change leads to multiple test failures if the ``SOURCE_DATE_EPOCH`` environment variable is set. This changes ``py_compile.compile()`` to only look at ``SOURCE_DATE_EPOCH`` if no explicit *invalidation_mode* was specified. I also made various relevant tests run with explicit control over the value of ``SOURCE_DATE_EPOCH``. While looking at this, I noticed that ``zipimport`` does not work with hash-based .pycs _at all_, though I left the fixes for subsequent commits. (cherry picked from commit a6b3ec5) Co-authored-by: Elvis Pranskevichus <elvis@magic.io>
1 parent f71a592 commit 24b51b1
Copy full SHA for 24b51b1

File tree

8 files changed

+161
-30
lines changed
Filter options

8 files changed

+161
-30
lines changed

‎Doc/library/compileall.rst

Copy file name to clipboardExpand all lines: Doc/library/compileall.rst
+7-4Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -85,13 +85,16 @@ compile Python sources.
8585

8686
.. cmdoption:: --invalidation-mode [timestamp|checked-hash|unchecked-hash]
8787

88-
Control how the generated pycs will be invalidated at runtime. The default
89-
setting, ``timestamp``, means that ``.pyc`` files with the source timestamp
88+
Control how the generated byte-code files are invalidated at runtime.
89+
The ``timestamp`` value, means that ``.pyc`` files with the source timestamp
9090
and size embedded will be generated. The ``checked-hash`` and
9191
``unchecked-hash`` values cause hash-based pycs to be generated. Hash-based
9292
pycs embed a hash of the source file contents rather than a timestamp. See
93-
:ref:`pyc-invalidation` for more information on how Python validates bytecode
94-
cache files at runtime.
93+
:ref:`pyc-invalidation` for more information on how Python validates
94+
bytecode cache files at runtime.
95+
The default is ``timestamp`` if the :envvar:`SOURCE_DATE_EPOCH` environment
96+
variable is not set, and ``checked-hash`` if the ``SOURCE_DATE_EPOCH``
97+
environment variable is set.
9598

9699
.. versionchanged:: 3.2
97100
Added the ``-i``, ``-b`` and ``-h`` options.

‎Doc/library/py_compile.rst

Copy file name to clipboardExpand all lines: Doc/library/py_compile.rst
+9-4Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -54,10 +54,10 @@ byte-code cache files in the directory containing the source code.
5454
level of the current interpreter.
5555

5656
*invalidation_mode* should be a member of the :class:`PycInvalidationMode`
57-
enum and controls how the generated ``.pyc`` files are invalidated at
58-
runtime. If the :envvar:`SOURCE_DATE_EPOCH` environment variable is set,
59-
*invalidation_mode* will be forced to
60-
:attr:`PycInvalidationMode.CHECKED_HASH`.
57+
enum and controls how the generated bytecode cache is invalidated at
58+
runtime. The default is :attr:`PycInvalidationMode.CHECKED_HASH` if
59+
the :envvar:`SOURCE_DATE_EPOCH` environment variable is set, otherwise
60+
the default is :attr:`PycInvalidationMode.TIMESTAMP`.
6161

6262
.. versionchanged:: 3.2
6363
Changed default value of *cfile* to be :PEP:`3147`-compliant. Previous
@@ -77,6 +77,11 @@ byte-code cache files in the directory containing the source code.
7777
*invalidation_mode* will be forced to
7878
:attr:`PycInvalidationMode.CHECKED_HASH`.
7979

80+
.. versionchanged:: 3.7.2
81+
The :envvar:`SOURCE_DATE_EPOCH` environment variable no longer
82+
overrides the value of the *invalidation_mode* argument, and determines
83+
its default value instead.
84+
8085

8186
.. class:: PycInvalidationMode
8287

‎Lib/compileall.py

Copy file name to clipboardExpand all lines: Lib/compileall.py
+13-7Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ def _walk_dir(dir, ddir=None, maxlevels=10, quiet=0):
4949

5050
def compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None,
5151
quiet=0, legacy=False, optimize=-1, workers=1,
52-
invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP):
52+
invalidation_mode=None):
5353
"""Byte-compile all modules in the given directory tree.
5454
5555
Arguments (only dir is required):
@@ -100,7 +100,7 @@ def compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None,
100100

101101
def compile_file(fullname, ddir=None, force=False, rx=None, quiet=0,
102102
legacy=False, optimize=-1,
103-
invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP):
103+
invalidation_mode=None):
104104
"""Byte-compile one file.
105105
106106
Arguments (only fullname is required):
@@ -186,7 +186,7 @@ def compile_file(fullname, ddir=None, force=False, rx=None, quiet=0,
186186

187187
def compile_path(skip_curdir=1, maxlevels=0, force=False, quiet=0,
188188
legacy=False, optimize=-1,
189-
invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP):
189+
invalidation_mode=None):
190190
"""Byte-compile all module on sys.path.
191191
192192
Arguments (all optional):
@@ -259,9 +259,12 @@ def main():
259259
type=int, help='Run compileall concurrently')
260260
invalidation_modes = [mode.name.lower().replace('_', '-')
261261
for mode in py_compile.PycInvalidationMode]
262-
parser.add_argument('--invalidation-mode', default='timestamp',
262+
parser.add_argument('--invalidation-mode',
263263
choices=sorted(invalidation_modes),
264-
help='How the pycs will be invalidated at runtime')
264+
help=('set .pyc invalidation mode; defaults to '
265+
'"checked-hash" if the SOURCE_DATE_EPOCH '
266+
'environment variable is set, and '
267+
'"timestamp" otherwise.'))
265268

266269
args = parser.parse_args()
267270
compile_dests = args.compile_dest
@@ -290,8 +293,11 @@ def main():
290293
if args.workers is not None:
291294
args.workers = args.workers or None
292295

293-
ivl_mode = args.invalidation_mode.replace('-', '_').upper()
294-
invalidation_mode = py_compile.PycInvalidationMode[ivl_mode]
296+
if args.invalidation_mode:
297+
ivl_mode = args.invalidation_mode.replace('-', '_').upper()
298+
invalidation_mode = py_compile.PycInvalidationMode[ivl_mode]
299+
else:
300+
invalidation_mode = None
295301

296302
success = True
297303
try:

‎Lib/py_compile.py

Copy file name to clipboardExpand all lines: Lib/py_compile.py
+10-3Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -69,8 +69,15 @@ class PycInvalidationMode(enum.Enum):
6969
UNCHECKED_HASH = 3
7070

7171

72+
def _get_default_invalidation_mode():
73+
if os.environ.get('SOURCE_DATE_EPOCH'):
74+
return PycInvalidationMode.CHECKED_HASH
75+
else:
76+
return PycInvalidationMode.TIMESTAMP
77+
78+
7279
def compile(file, cfile=None, dfile=None, doraise=False, optimize=-1,
73-
invalidation_mode=PycInvalidationMode.TIMESTAMP):
80+
invalidation_mode=None):
7481
"""Byte-compile one Python source file to Python bytecode.
7582
7683
:param file: The source file name.
@@ -112,8 +119,8 @@ def compile(file, cfile=None, dfile=None, doraise=False, optimize=-1,
112119
the resulting file would be regular and thus not the same type of file as
113120
it was previously.
114121
"""
115-
if os.environ.get('SOURCE_DATE_EPOCH'):
116-
invalidation_mode = PycInvalidationMode.CHECKED_HASH
122+
if invalidation_mode is None:
123+
invalidation_mode = _get_default_invalidation_mode()
117124
if cfile is None:
118125
if optimize >= 0:
119126
optimization = optimize if optimize >= 1 else ''

‎Lib/test/test_compileall.py

Copy file name to clipboardExpand all lines: Lib/test/test_compileall.py
+44-6Lines changed: 44 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,11 @@
2222
from test import support
2323
from test.support import script_helper
2424

25-
class CompileallTests(unittest.TestCase):
25+
from .test_py_compile import without_source_date_epoch
26+
from .test_py_compile import SourceDateEpochTestMeta
27+
28+
29+
class CompileallTestsBase:
2630

2731
def setUp(self):
2832
self.directory = tempfile.mkdtemp()
@@ -46,7 +50,7 @@ def add_bad_source_file(self):
4650
with open(self.bad_source_path, 'w') as file:
4751
file.write('x (\n')
4852

49-
def data(self):
53+
def timestamp_metadata(self):
5054
with open(self.bc_path, 'rb') as file:
5155
data = file.read(12)
5256
mtime = int(os.stat(self.source_path).st_mtime)
@@ -57,16 +61,18 @@ def data(self):
5761
def recreation_check(self, metadata):
5862
"""Check that compileall recreates bytecode when the new metadata is
5963
used."""
64+
if os.environ.get('SOURCE_DATE_EPOCH'):
65+
raise unittest.SkipTest('SOURCE_DATE_EPOCH is set')
6066
py_compile.compile(self.source_path)
61-
self.assertEqual(*self.data())
67+
self.assertEqual(*self.timestamp_metadata())
6268
with open(self.bc_path, 'rb') as file:
6369
bc = file.read()[len(metadata):]
6470
with open(self.bc_path, 'wb') as file:
6571
file.write(metadata)
6672
file.write(bc)
67-
self.assertNotEqual(*self.data())
73+
self.assertNotEqual(*self.timestamp_metadata())
6874
compileall.compile_dir(self.directory, force=False, quiet=True)
69-
self.assertTrue(*self.data())
75+
self.assertTrue(*self.timestamp_metadata())
7076

7177
def test_mtime(self):
7278
# Test a change in mtime leads to a new .pyc.
@@ -189,6 +195,21 @@ def test_compile_missing_multiprocessing(self, compile_file_mock):
189195
compileall.compile_dir(self.directory, quiet=True, workers=5)
190196
self.assertTrue(compile_file_mock.called)
191197

198+
199+
class CompileallTestsWithSourceEpoch(CompileallTestsBase,
200+
unittest.TestCase,
201+
metaclass=SourceDateEpochTestMeta,
202+
source_date_epoch=True):
203+
pass
204+
205+
206+
class CompileallTestsWithoutSourceEpoch(CompileallTestsBase,
207+
unittest.TestCase,
208+
metaclass=SourceDateEpochTestMeta,
209+
source_date_epoch=False):
210+
pass
211+
212+
192213
class EncodingTest(unittest.TestCase):
193214
"""Issue 6716: compileall should escape source code when printing errors
194215
to stdout."""
@@ -212,7 +233,7 @@ def test_error(self):
212233
sys.stdout = orig_stdout
213234

214235

215-
class CommandLineTests(unittest.TestCase):
236+
class CommandLineTestsBase:
216237
"""Test compileall's CLI."""
217238

218239
@classmethod
@@ -285,6 +306,7 @@ def test_no_args_compiles_path(self):
285306
self.assertNotCompiled(self.initfn)
286307
self.assertNotCompiled(self.barfn)
287308

309+
@without_source_date_epoch # timestamp invalidation test
288310
def test_no_args_respects_force_flag(self):
289311
self._skip_if_sys_path_not_writable()
290312
bazfn = script_helper.make_script(self.directory, 'baz', '')
@@ -353,6 +375,7 @@ def test_multiple_runs(self):
353375
self.assertTrue(os.path.exists(self.pkgdir_cachedir))
354376
self.assertFalse(os.path.exists(cachecachedir))
355377

378+
@without_source_date_epoch # timestamp invalidation test
356379
def test_force(self):
357380
self.assertRunOK('-q', self.pkgdir)
358381
pycpath = importlib.util.cache_from_source(self.barfn)
@@ -556,5 +579,20 @@ def test_workers_available_cores(self, compile_dir):
556579
self.assertEqual(compile_dir.call_args[-1]['workers'], None)
557580

558581

582+
class CommmandLineTestsWithSourceEpoch(CommandLineTestsBase,
583+
unittest.TestCase,
584+
metaclass=SourceDateEpochTestMeta,
585+
source_date_epoch=True):
586+
pass
587+
588+
589+
class CommmandLineTestsNoSourceEpoch(CommandLineTestsBase,
590+
unittest.TestCase,
591+
metaclass=SourceDateEpochTestMeta,
592+
source_date_epoch=False):
593+
pass
594+
595+
596+
559597
if __name__ == "__main__":
560598
unittest.main()

‎Lib/test/test_importlib/source/test_file_loader.py

Copy file name to clipboardExpand all lines: Lib/test/test_importlib/source/test_file_loader.py
+15Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,9 @@
1919

2020
from test.support import make_legacy_pyc, unload
2121

22+
from test.test_py_compile import without_source_date_epoch
23+
from test.test_py_compile import SourceDateEpochTestMeta
24+
2225

2326
class SimpleTest(abc.LoaderTests):
2427

@@ -359,6 +362,17 @@ def test_overiden_unchecked_hash_based_pyc(self):
359362
abc=importlib_abc, util=importlib_util)
360363

361364

365+
class SourceDateEpochTestMeta(SourceDateEpochTestMeta,
366+
type(Source_SimpleTest)):
367+
pass
368+
369+
370+
class SourceDateEpoch_SimpleTest(Source_SimpleTest,
371+
metaclass=SourceDateEpochTestMeta,
372+
source_date_epoch=True):
373+
pass
374+
375+
362376
class BadBytecodeTest:
363377

364378
def import_(self, file, module_name):
@@ -617,6 +631,7 @@ def test_bad_marshal(self):
617631

618632
# [bad timestamp]
619633
@util.writes_bytecode_files
634+
@without_source_date_epoch
620635
def test_old_timestamp(self):
621636
# When the timestamp is older than the source, bytecode should be
622637
# regenerated.

‎Lib/test/test_py_compile.py

Copy file name to clipboardExpand all lines: Lib/test/test_py_compile.py
+60-6Lines changed: 60 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import functools
12
import importlib.util
23
import os
34
import py_compile
@@ -10,7 +11,44 @@
1011
from test import support
1112

1213

13-
class PyCompileTests(unittest.TestCase):
14+
def without_source_date_epoch(fxn):
15+
"""Runs function with SOURCE_DATE_EPOCH unset."""
16+
@functools.wraps(fxn)
17+
def wrapper(*args, **kwargs):
18+
with support.EnvironmentVarGuard() as env:
19+
env.unset('SOURCE_DATE_EPOCH')
20+
return fxn(*args, **kwargs)
21+
return wrapper
22+
23+
24+
def with_source_date_epoch(fxn):
25+
"""Runs function with SOURCE_DATE_EPOCH set."""
26+
@functools.wraps(fxn)
27+
def wrapper(*args, **kwargs):
28+
with support.EnvironmentVarGuard() as env:
29+
env['SOURCE_DATE_EPOCH'] = '123456789'
30+
return fxn(*args, **kwargs)
31+
return wrapper
32+
33+
34+
# Run tests with SOURCE_DATE_EPOCH set or unset explicitly.
35+
class SourceDateEpochTestMeta(type(unittest.TestCase)):
36+
def __new__(mcls, name, bases, dct, *, source_date_epoch):
37+
cls = super().__new__(mcls, name, bases, dct)
38+
39+
for attr in dir(cls):
40+
if attr.startswith('test_'):
41+
meth = getattr(cls, attr)
42+
if source_date_epoch:
43+
wrapper = with_source_date_epoch(meth)
44+
else:
45+
wrapper = without_source_date_epoch(meth)
46+
setattr(cls, attr, wrapper)
47+
48+
return cls
49+
50+
51+
class PyCompileTestsBase:
1452

1553
def setUp(self):
1654
self.directory = tempfile.mkdtemp()
@@ -99,16 +137,18 @@ def test_bad_coding(self):
99137
importlib.util.cache_from_source(bad_coding)))
100138

101139
def test_source_date_epoch(self):
102-
testtime = 123456789
103-
with support.EnvironmentVarGuard() as env:
104-
env["SOURCE_DATE_EPOCH"] = str(testtime)
105-
py_compile.compile(self.source_path, self.pyc_path)
140+
py_compile.compile(self.source_path, self.pyc_path)
106141
self.assertTrue(os.path.exists(self.pyc_path))
107142
self.assertFalse(os.path.exists(self.cache_path))
108143
with open(self.pyc_path, 'rb') as fp:
109144
flags = importlib._bootstrap_external._classify_pyc(
110145
fp.read(), 'test', {})
111-
self.assertEqual(flags, 0b11)
146+
if os.environ.get('SOURCE_DATE_EPOCH'):
147+
expected_flags = 0b11
148+
else:
149+
expected_flags = 0b00
150+
151+
self.assertEqual(flags, expected_flags)
112152

113153
@unittest.skipIf(sys.flags.optimize > 0, 'test does not work with -O')
114154
def test_double_dot_no_clobber(self):
@@ -153,5 +193,19 @@ def test_invalidation_mode(self):
153193
self.assertEqual(flags, 0b1)
154194

155195

196+
class PyCompileTestsWithSourceEpoch(PyCompileTestsBase,
197+
unittest.TestCase,
198+
metaclass=SourceDateEpochTestMeta,
199+
source_date_epoch=True):
200+
pass
201+
202+
203+
class PyCompileTestsWithoutSourceEpoch(PyCompileTestsBase,
204+
unittest.TestCase,
205+
metaclass=SourceDateEpochTestMeta,
206+
source_date_epoch=False):
207+
pass
208+
209+
156210
if __name__ == "__main__":
157211
unittest.main()

‎Misc/NEWS.d/next/Library/2018-09-27-13-14-15.bpo-34022.E2cl0r.rst

Copy file name to clipboard
+3Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
The :envvar:`SOURCE_DATE_EPOCH` environment variable no longer overrides the
2+
value of the *invalidation_mode* argument to :func:`py_compile.compile`, and
3+
determines its default value instead.

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.