Speed up umbrella tox -e cover#2500
Speed up umbrella tox -e cover#2500dhermes wants to merge 5 commits intogoogleapis:mastergoogleapis/google-cloud-python:masterfrom dhermes:move-tests-side-by-sidedhermes/google-cloud-python:move-tests-side-by-sideCopy head branch name to clipboard
Conversation
To avoid collision had to move most tests into a uniquely named directory. For example the tests in bigquery/unit_tests needed to move to bigquery/unit_tests/bigquery to avoid any name collision. Done via: https://gist.github.com/dhermes/397c731966674da1b42612b723e64895
Done via: $ git grep -l 'from unit_tests._fixtures import' -- vision | > xargs sed -i s/'from unit_tests._fixtures import'/'from unit_tests.vision._fixtures import'/g $ $ git grep -l 'from unit_tests._fixtures import' -- speech | > xargs sed -i s/'from unit_tests._fixtures import'/'from unit_tests.speech._fixtures import'/g $ $ git grep -l 'from unit_tests._testing import ' -- bigtable | > xargs sed -i s/'from unit_tests._testing import '/'from unit_tests.bigtable._testing import '/g
This requires manually collecting from the installed site-packages. See: https://testrun.org/tox/latest/example/pytest.html
Issue introduced in googleapis#2486.
|
|
| python_files = [ | ||
| candidate for candidate in candidates if candidate.endswith('.py')] | ||
| candidate for candidate in candidates | ||
| if candidate.endswith('.py') and os.path.exists(candidate)] |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
|
I'm grinding my teeth about the issues caused from moving the unit tests out-of-tree: I still think they belong there, which would remove the need to mess with the |
|
So from a selfish standpoint, I like having the unit_tests for a package in the package. It makes for less mental tracking while working on things. Might just be me though. |
|
@daspecster RE:
That is just |
|
Chatted with @tseaver and @jonparrott about this and will be putting the unit tests back. |
Fixes internal issue 418025765 🦕
This took a lot of wrangling, but eventually I found a solution: make
unit_testsa namespace package and then collect from site-packages. This way we only runpy.testonce instead of 15 times, a huge speedup.In order to do this I needed to do some renames to avoid collisions (e.g.
unit_tests.test_clientwould have been installed for every sub-package).The first 3 (
of 4of 5) commits are for the rename. The first two are automated (i.e. I didn't make changes to any files myself) and the third is just a tiny lint cleanup based on the automated changes.Also relevant: https://testrun.org/tox/latest/example/pytest.html#known-issues-and-limitations