-
-
Notifications
You must be signed in to change notification settings - Fork 32.1k
bpo-37421: test_urllib calls urlcleanup() #14529
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. It seems test_urllibnet
already uses a wrapper to urlretrieve
to delete temporary files automatically. Thanks.
urllib.request tests now call urlcleanup() to remove temporary files created by urlretrieve() tests and to clear the _opener global variable set by urlopen() and functions calling indirectly urlopen(). regrtest now checks if urllib.request._url_tempfiles and urllib.request._opener are changed by tests.
@tirkarthi: Would you mind to review the updated PR?
See also https://bugs.python.org/issue37475 "What is urllib.request.urlcleanup() function?" In this issue, I also discovered that urlopen() sets a "_opener" global variable and so have a side effect on following tests. I modified my PR to test in regrtest that urllib.requests._url_tempfiles and urllib.requests._opener are not modified, and I modified way more tests to call urlcleanup(). urlopen() and urlretrieve() side effects are really surprising, but let's discuss that in https://bugs.python.org/issue37475 |
If I understand regrtest correctly, cpython/Lib/test/libregrtest/refleak.py Line 223 in 61bf97e
urllib.request._url_tempfiles as empty due to clear_caches and by restore_urllib_requests__url_tempfiles it will just be looping over an empty list? Similar thing with _opener being set as None. Is it worth doing the changes using get and restore?
|
clear_caches() is not used by default. It's only used when running tests with -R 3:3.
It allows to detect bugs in tests. Try to modify my PR to remove some calls to urlcleanup() and run modified tests, you will see tests failing with ENV_CHANGED. Pass --fail-env-changed option to make the command fail. |
save_test_environment() helps to detect side effects of unit tests. Unit tests should have no side effect. |
Okay, I just grepped for cpython/Lib/test/libregrtest/runtest.py Line 263 in 039fb49
|
One more note that I found is that Also I just commented out the part in def test_url_cleanup_opener(self):
url = 'http://docs.python.org/library/urllib.html'
self.fakehttp(b"HTTP/1.1 200 OK\r\n\r\nHello!")
try:
fp = urllib.request.urlopen(url)
self.assertIsInstance(urllib.request._opener, urllib.request.OpenerDirector)
urllib.request.urlcleanup()
self.assertIsNone(urllib.request._opener)
finally:
self.unfakehttp() |
Oh ok, I didn't know. Well, my save_env.py change is still relevant.
Oh wow, that's super weird. Why test_urllib tests is test urlopen() function rather than testing urllib.request module?! I propose to address this issue in a separated PR and/or in https://bugs.python.org/issue37475
I tried this change on top of my PR:
The test fails as expected:
|
@berkerpeksag: Thanks for the review. I merged my PR to unblock https://bugs.python.org/issue37421 It seems like there are other issues which can be addressed separately. |
Thanks Victor, I tried without fail-env-changed and hence the confusion over expecting an assertion error. |
urllib.request tests now call urlcleanup() to remove temporary files created by urlretrieve() tests and to clear the _opener global variable set by urlopen() and functions calling indirectly urlopen(). regrtest now checks if urllib.request._url_tempfiles and urllib.request._opener are changed by tests.
urllib.request tests now call urlcleanup() to remove temporary files created by urlretrieve() tests and to clear the _opener global variable set by urlopen() and functions calling indirectly urlopen(). regrtest now checks if urllib.request._url_tempfiles and urllib.request._opener are changed by tests.
urlretrieve_HttpTests tests now explicitly call urlcleanup() to
remove temporary files.
https://bugs.python.org/issue37421