Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit d50c37d

Browse filesBrowse files
pganssleZac-HD
pganssle
and
Zac-HD
authored
GH-86275: Implementation of hypothesis stubs for property-based tests, with zoneinfo tests (#22863)
These are stubs to be used for adding hypothesis (https://hypothesis.readthedocs.io/en/latest/) tests to the standard library. When the tests are run in an environment where `hypothesis` and its various dependencies are not installed, the stubs will turn any tests with examples into simple parameterized tests and any tests without examples are skipped. It also adds hypothesis tests for the `zoneinfo` module, and a Github Actions workflow to run the hypothesis tests as a non-required CI job. The full hypothesis interface is not stubbed out — missing stubs can be added as necessary. Co-authored-by: Zac Hatfield-Dodds <zac.hatfield.dodds@gmail.com>
1 parent 45f5aa8 commit d50c37d
Copy full SHA for d50c37d

File tree

9 files changed

+719
-1
lines changed
Filter options

9 files changed

+719
-1
lines changed

‎.github/workflows/build.yml

Copy file name to clipboardExpand all lines: .github/workflows/build.yml
+96Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ jobs:
3636
timeout-minutes: 10
3737
outputs:
3838
run_tests: ${{ steps.check.outputs.run_tests }}
39+
run_hypothesis: ${{ steps.check.outputs.run_hypothesis }}
3940
steps:
4041
- uses: actions/checkout@v3
4142
- name: Check for source changes
@@ -61,6 +62,17 @@ jobs:
6162
git diff --name-only origin/$GITHUB_BASE_REF.. | grep -qvE '(\.rst$|^Doc|^Misc)' && echo "run_tests=true" >> $GITHUB_OUTPUT || true
6263
fi
6364
65+
# Check if we should run hypothesis tests
66+
GIT_BRANCH=${GITHUB_BASE_REF:-${GITHUB_REF#refs/heads/}}
67+
echo $GIT_BRANCH
68+
if $(echo "$GIT_BRANCH" | grep -q -w '3\.\(8\|9\|10\|11\)'); then
69+
echo "Branch too old for hypothesis tests"
70+
echo "run_hypothesis=false" >> $GITHUB_OUTPUT
71+
else
72+
echo "Run hypothesis tests"
73+
echo "run_hypothesis=true" >> $GITHUB_OUTPUT
74+
fi
75+
6476
check_generated_files:
6577
name: 'Check if generated files are up to date'
6678
runs-on: ubuntu-latest
@@ -291,6 +303,90 @@ jobs:
291303
- name: SSL tests
292304
run: ./python Lib/test/ssltests.py
293305

306+
test_hypothesis:
307+
name: "Hypothesis Tests on Ubuntu"
308+
runs-on: ubuntu-20.04
309+
timeout-minutes: 60
310+
needs: check_source
311+
if: needs.check_source.outputs.run_tests == 'true' && needs.check_source.outputs.run_hypothesis == 'true'
312+
env:
313+
OPENSSL_VER: 1.1.1t
314+
PYTHONSTRICTEXTENSIONBUILD: 1
315+
steps:
316+
- uses: actions/checkout@v3
317+
- name: Register gcc problem matcher
318+
run: echo "::add-matcher::.github/problem-matchers/gcc.json"
319+
- name: Install Dependencies
320+
run: sudo ./.github/workflows/posix-deps-apt.sh
321+
- name: Configure OpenSSL env vars
322+
run: |
323+
echo "MULTISSL_DIR=${GITHUB_WORKSPACE}/multissl" >> $GITHUB_ENV
324+
echo "OPENSSL_DIR=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}" >> $GITHUB_ENV
325+
echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> $GITHUB_ENV
326+
- name: 'Restore OpenSSL build'
327+
id: cache-openssl
328+
uses: actions/cache@v3
329+
with:
330+
path: ./multissl/openssl/${{ env.OPENSSL_VER }}
331+
key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }}
332+
- name: Install OpenSSL
333+
if: steps.cache-openssl.outputs.cache-hit != 'true'
334+
run: python3 Tools/ssl/multissltests.py --steps=library --base-directory $MULTISSL_DIR --openssl $OPENSSL_VER --system Linux
335+
- name: Add ccache to PATH
336+
run: |
337+
echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV
338+
- name: Configure ccache action
339+
uses: hendrikmuhs/ccache-action@v1.2
340+
- name: Setup directory envs for out-of-tree builds
341+
run: |
342+
echo "CPYTHON_RO_SRCDIR=$(realpath -m ${GITHUB_WORKSPACE}/../cpython-ro-srcdir)" >> $GITHUB_ENV
343+
echo "CPYTHON_BUILDDIR=$(realpath -m ${GITHUB_WORKSPACE}/../cpython-builddir)" >> $GITHUB_ENV
344+
- name: Create directories for read-only out-of-tree builds
345+
run: mkdir -p $CPYTHON_RO_SRCDIR $CPYTHON_BUILDDIR
346+
- name: Bind mount sources read-only
347+
run: sudo mount --bind -o ro $GITHUB_WORKSPACE $CPYTHON_RO_SRCDIR
348+
- name: Configure CPython out-of-tree
349+
working-directory: ${{ env.CPYTHON_BUILDDIR }}
350+
run: ../cpython-ro-srcdir/configure --with-pydebug --with-openssl=$OPENSSL_DIR
351+
- name: Build CPython out-of-tree
352+
working-directory: ${{ env.CPYTHON_BUILDDIR }}
353+
run: make -j4
354+
- name: Display build info
355+
working-directory: ${{ env.CPYTHON_BUILDDIR }}
356+
run: make pythoninfo
357+
- name: Remount sources writable for tests
358+
# some tests write to srcdir, lack of pyc files slows down testing
359+
run: sudo mount $CPYTHON_RO_SRCDIR -oremount,rw
360+
- name: Setup directory envs for out-of-tree builds
361+
run: |
362+
echo "CPYTHON_BUILDDIR=$(realpath -m ${GITHUB_WORKSPACE}/../cpython-builddir)" >> $GITHUB_ENV
363+
- name: "Create hypothesis venv"
364+
working-directory: ${{ env.CPYTHON_BUILDDIR }}
365+
run: |
366+
VENV_LOC=$(realpath -m .)/hypovenv
367+
VENV_PYTHON=$VENV_LOC/bin/python
368+
echo "HYPOVENV=${VENV_LOC}" >> $GITHUB_ENV
369+
echo "VENV_PYTHON=${VENV_PYTHON}" >> $GITHUB_ENV
370+
./python -m venv $VENV_LOC && $VENV_PYTHON -m pip install -U hypothesis
371+
- name: "Run tests"
372+
working-directory: ${{ env.CPYTHON_BUILDDIR }}
373+
run: |
374+
# Most of the excluded tests are slow test suites with no property tests
375+
#
376+
# (GH-104097) test_sysconfig is skipped because it has tests that are
377+
# failing when executed from inside a virtual environment.
378+
${{ env.VENV_PYTHON }} -m test \
379+
-W \
380+
-x test_asyncio \
381+
-x test_multiprocessing_fork \
382+
-x test_multiprocessing_forkserver \
383+
-x test_multiprocessing_spawn \
384+
-x test_concurrent_futures \
385+
-x test_socket \
386+
-x test_subprocess \
387+
-x test_signal \
388+
-x test_sysconfig
389+
294390
295391
build_asan:
296392
name: 'Address sanitizer'

‎Lib/test/libregrtest/save_env.py

Copy file name to clipboardExpand all lines: Lib/test/libregrtest/save_env.py
+3-1Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -257,8 +257,10 @@ def restore_sysconfig__INSTALL_SCHEMES(self, saved):
257257
sysconfig._INSTALL_SCHEMES.update(saved[2])
258258

259259
def get_files(self):
260+
# XXX: Maybe add an allow-list here?
260261
return sorted(fn + ('/' if os.path.isdir(fn) else '')
261-
for fn in os.listdir())
262+
for fn in os.listdir()
263+
if not fn.startswith(".hypothesis"))
262264
def restore_files(self, saved_value):
263265
fn = os_helper.TESTFN
264266
if fn not in saved_value and (fn + '/') not in saved_value:

‎Lib/test/support/_hypothesis_stubs/__init__.py

Copy file name to clipboard
+111Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
from enum import Enum
2+
import functools
3+
import unittest
4+
5+
__all__ = [
6+
"given",
7+
"example",
8+
"assume",
9+
"reject",
10+
"register_random",
11+
"strategies",
12+
"HealthCheck",
13+
"settings",
14+
"Verbosity",
15+
]
16+
17+
from . import strategies
18+
19+
20+
def given(*_args, **_kwargs):
21+
def decorator(f):
22+
if examples := getattr(f, "_examples", []):
23+
24+
@functools.wraps(f)
25+
def test_function(self):
26+
for example_args, example_kwargs in examples:
27+
with self.subTest(*example_args, **example_kwargs):
28+
f(self, *example_args, **example_kwargs)
29+
30+
else:
31+
# If we have found no examples, we must skip the test. If @example
32+
# is applied after @given, it will re-wrap the test to remove the
33+
# skip decorator.
34+
test_function = unittest.skip(
35+
"Hypothesis required for property test with no " +
36+
"specified examples"
37+
)(f)
38+
39+
test_function._given = True
40+
return test_function
41+
42+
return decorator
43+
44+
45+
def example(*args, **kwargs):
46+
if bool(args) == bool(kwargs):
47+
raise ValueError("Must specify exactly one of *args or **kwargs")
48+
49+
def decorator(f):
50+
base_func = getattr(f, "__wrapped__", f)
51+
if not hasattr(base_func, "_examples"):
52+
base_func._examples = []
53+
54+
base_func._examples.append((args, kwargs))
55+
56+
if getattr(f, "_given", False):
57+
# If the given decorator is below all the example decorators,
58+
# it would be erroneously skipped, so we need to re-wrap the new
59+
# base function.
60+
f = given()(base_func)
61+
62+
return f
63+
64+
return decorator
65+
66+
67+
def assume(condition):
68+
if not condition:
69+
raise unittest.SkipTest("Unsatisfied assumption")
70+
return True
71+
72+
73+
def reject():
74+
assume(False)
75+
76+
77+
def register_random(*args, **kwargs):
78+
pass # pragma: no cover
79+
80+
81+
def settings(*args, **kwargs):
82+
return lambda f: f # pragma: nocover
83+
84+
85+
class HealthCheck(Enum):
86+
data_too_large = 1
87+
filter_too_much = 2
88+
too_slow = 3
89+
return_value = 5
90+
large_base_example = 7
91+
not_a_test_method = 8
92+
93+
@classmethod
94+
def all(cls):
95+
return list(cls)
96+
97+
98+
class Verbosity(Enum):
99+
quiet = 0
100+
normal = 1
101+
verbose = 2
102+
debug = 3
103+
104+
105+
class Phase(Enum):
106+
explicit = 0
107+
reuse = 1
108+
generate = 2
109+
target = 3
110+
shrink = 4
111+
explain = 5

‎Lib/test/support/_hypothesis_stubs/_helpers.py

Copy file name to clipboard
+43Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Stub out only the subset of the interface that we actually use in our tests.
2+
class StubClass:
3+
def __init__(self, *args, **kwargs):
4+
self.__stub_args = args
5+
self.__stub_kwargs = kwargs
6+
self.__repr = None
7+
8+
def _with_repr(self, new_repr):
9+
new_obj = self.__class__(*self.__stub_args, **self.__stub_kwargs)
10+
new_obj.__repr = new_repr
11+
return new_obj
12+
13+
def __repr__(self):
14+
if self.__repr is not None:
15+
return self.__repr
16+
17+
argstr = ", ".join(self.__stub_args)
18+
kwargstr = ", ".join(f"{kw}={val}" for kw, val in self.__stub_kwargs.items())
19+
20+
in_parens = argstr
21+
if kwargstr:
22+
in_parens += ", " + kwargstr
23+
24+
return f"{self.__class__.__qualname__}({in_parens})"
25+
26+
27+
def stub_factory(klass, name, *, with_repr=None, _seen={}):
28+
if (klass, name) not in _seen:
29+
30+
class Stub(klass):
31+
def __init__(self, *args, **kwargs):
32+
super().__init__()
33+
self.__stub_args = args
34+
self.__stub_kwargs = kwargs
35+
36+
Stub.__name__ = name
37+
Stub.__qualname__ = name
38+
if with_repr is not None:
39+
Stub._repr = None
40+
41+
_seen.setdefault((klass, name, with_repr), Stub)
42+
43+
return _seen[(klass, name, with_repr)]

‎Lib/test/support/_hypothesis_stubs/strategies.py

Copy file name to clipboard
+91Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
import functools
2+
3+
from ._helpers import StubClass, stub_factory
4+
5+
6+
class StubStrategy(StubClass):
7+
def __make_trailing_repr(self, transformation_name, func):
8+
func_name = func.__name__ or repr(func)
9+
return f"{self!r}.{transformation_name}({func_name})"
10+
11+
def map(self, pack):
12+
return self._with_repr(self.__make_trailing_repr("map", pack))
13+
14+
def flatmap(self, expand):
15+
return self._with_repr(self.__make_trailing_repr("flatmap", expand))
16+
17+
def filter(self, condition):
18+
return self._with_repr(self.__make_trailing_repr("filter", condition))
19+
20+
def __or__(self, other):
21+
new_repr = f"one_of({self!r}, {other!r})"
22+
return self._with_repr(new_repr)
23+
24+
25+
_STRATEGIES = {
26+
"binary",
27+
"booleans",
28+
"builds",
29+
"characters",
30+
"complex_numbers",
31+
"composite",
32+
"data",
33+
"dates",
34+
"datetimes",
35+
"decimals",
36+
"deferred",
37+
"dictionaries",
38+
"emails",
39+
"fixed_dictionaries",
40+
"floats",
41+
"fractions",
42+
"from_regex",
43+
"from_type",
44+
"frozensets",
45+
"functions",
46+
"integers",
47+
"iterables",
48+
"just",
49+
"lists",
50+
"none",
51+
"nothing",
52+
"one_of",
53+
"permutations",
54+
"random_module",
55+
"randoms",
56+
"recursive",
57+
"register_type_strategy",
58+
"runner",
59+
"sampled_from",
60+
"sets",
61+
"shared",
62+
"slices",
63+
"timedeltas",
64+
"times",
65+
"text",
66+
"tuples",
67+
"uuids",
68+
}
69+
70+
__all__ = sorted(_STRATEGIES)
71+
72+
73+
def composite(f):
74+
strategy = stub_factory(StubStrategy, f.__name__)
75+
76+
@functools.wraps(f)
77+
def inner(*args, **kwargs):
78+
return strategy(*args, **kwargs)
79+
80+
return inner
81+
82+
83+
def __getattr__(name):
84+
if name not in _STRATEGIES:
85+
raise AttributeError(f"Unknown attribute {name}")
86+
87+
return stub_factory(StubStrategy, f"hypothesis.strategies.{name}")
88+
89+
90+
def __dir__():
91+
return __all__

‎Lib/test/support/hypothesis_helper.py

Copy file name to clipboard
+4Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
try:
2+
import hypothesis
3+
except ImportError:
4+
from . import _hypothesis_stubs as hypothesis

‎Lib/test/test_zoneinfo/__init__.py

Copy file name to clipboard
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
11
from .test_zoneinfo import *
2+
from .test_zoneinfo_property import *

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.