Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

TST Do not use cache in test in test_fetch_openml_iris #19594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Mar 12, 2021
Merged

TST Do not use cache in test in test_fetch_openml_iris #19594

merged 1 commit into from
Mar 12, 2021

Conversation

thomasjpfan
Copy link
Member

Reference Issues/PRs

Closes #19587

What does this implement/fix? Explain your changes.

Test in sklearn/datasets/tests/test_openml.py should not be touching the cache.

ogrisel
ogrisel approved these changes Mar 2, 2021
View reviewed changes
Copy link
Member

@ogrisel ogrisel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds reasonable, but can you share your analysis of the original problem? Is this because the fetch_openml's caching mechanism is not concurrency-safe when running on pytest-xdist?

@thomasjpfan
Copy link
Member Author

My guess is one process creates the directory and begins to populates the cache with 61.gz a second process will try to read it and find that is is corrupted because the first process did not finish writing the cache file. This second process calls

scikit-learn/sklearn/datasets/_openml.py

Line 49 in 28ee486

def _retry_with_clean_cache(

which removes the file. Then the first process tries to read the file that it suppose to be caching and found that it was removed. I can see this randomly happen if test_fetch_openml_iris[True] and test_fetch_openml_iris[False] are run on different processes.

jeremiedbb
jeremiedbb approved these changes Mar 12, 2021
View reviewed changes
Copy link
Member

@jeremiedbb jeremiedbb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks @thomasjpfan

@jeremiedbb jeremiedbb merged commit bfd7b58 into scikit-learn:main Mar 12, 2021
marrodion pushed a commit to marrodion/scikit-learn that referenced this pull request Mar 17, 2021
@thomasjpfan @marrodion
TST Do not use cache in test_fetch_openml_iris (scikit-learn#19594)
64726b1
@glemaitre glemaitre mentioned this pull request Apr 22, 2021
Merged
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ogrisel ogrisel ogrisel approved these changes

@jeremiedbb jeremiedbb jeremiedbb approved these changes

Assignees
No one assigned
Labels
Build / CI module:datasets module:test-suite everything related to our tests No Changelog Needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Random failure in fetch_openml tests
4 participants
@thomasjpfan @ogrisel @jeremiedbb @cmarmo
Morty Proxy This is a proxified and sanitized view of the page, visit original site.