Testsuite driven py3 #17

ydirson · 2022-09-21T09:53:12Z

Last commits for full python3 compatibility

ydirson · 2022-09-26T08:50:16Z

Commit "fix xmlunwrap and its test to align with the use of bytes" raises a question, whether we don't want, rather, to fix the xcp.xmlunwrap API to use str and not bytes.

ydirson · 2022-09-26T11:54:55Z

Commit "switch from ConfigParser to configparser" raises a problem with the Accessor API, which does not allow to request text or binary streams, which allowed different implementations to return one or the other. Commit "test_accessor: check for I/O on binary files" then proceeds to demonstrate the problem, in the way it will impact host-upgrade-plugin.

My opinion is, we should change the Accessor API to add a mode parameter to openAddress(). Making it mandatory will make sure we don't miss any client code. @edwintorok what do you think ?

Too bad by parametrization of the tests make it harder to mark just the single failing case as expectedFailure, but I expect that one is not a problem with you ?

edwintorok · 2022-09-26T12:27:39Z

Adding the mode parameter seems like the right way forward but it will tie merging and pushing this PR together with other PRs fixing up all users, in that case it might be good to leave that for a 2nd PR so we can merge part of the 2-to-3 conversion without requiring other packages to move or upgrade just yet.

ydirson · 2022-09-26T13:03:11Z

Adding the mode parameter seems like the right way forward but it will tie merging and pushing this PR together with other PRs fixing up all users, in that case it might be good to leave that for a 2nd PR so we can merge part of the 2-to-3 conversion without requiring other packages to move or upgrade just yet.

Yes we could have a separate PR, but this is still part of the 2-to-3 conversion, and requires to push "switch from ConfigParser to configparser" as is, including the workaround for the issue - or just acknowledge that the 2-to-3 work is known not to be complete at this stage and not include the hackish commit, waiting for API refactor.

Similarly for the xmlunwrap API issue, which also revolves around str vs. bytes.

Use of `unicode` needed to be immediately handled, but a few checks relying on `str` could become insufficient in python2 with the larger usage of unicode strings. Signed-off-by: Yann Dirson <[email protected]>

…conversion Signed-off-by: Yann Dirson <[email protected]>

…s to open() as ths is considered best practice. (cherry picked from cpython commit 6cef076ba5edbfa42239924951d8acbb087b3b19) Signed-off-by: Yann Dirson <[email protected]>

…fication Signed-off-by: Yann Dirson <[email protected]>

…ated Signed-off-by: Yann Dirson <[email protected]>

Running tests on python3 did reveal some of them. Signed-off-by: Yann Dirson <[email protected]>

Signed-off-by: Yann Dirson <[email protected]>

There is no guaranty about ordering of dict elements, and tests compare results derived from enumerating a dict element. We could have used an OrderedDict to store the formulae and get a predictible output order, but just considering the output as a set seems better. Only applying this to rules expected to hold more than one element. Signed-off-by: Yann Dirson <[email protected]>

Caught by extended test. Signed-off-by: Yann Dirson <[email protected]>

This goes away in python3. Signed-off-by: Yann Dirson <[email protected]>

FIXME: I'm quite unsure why xcp.xmlunwrap would want to use bytes and not unicode strings, but the encode/decode calls make it quite clear it wants to work with bytes. That makes the API painful to use in python3.

hashlib came with python 2.5, and old md5 module disappears in 3.0 Signed-off-by: Yann Dirson <[email protected]>

This is supposed to be just a module renaming to conform to PEP8, see https://docs.python.org/3/whatsnew/3.0.html#library-changes The SafeConfigParser class has been renamed to ConfigParser in Python 3.2, and backported as addon package. The `readfp` method now triggers a deprecation warning to replace it with `read_file`. FIXME: With python3 some Accessor implementations (e.g. FileAccessor) provide a text stream for repository config (and with python2 all implementations), while others (e.g. HTTPAccessor) provide a binary stream. But on python3 ConfigParser will bomb out if given a binary stream, so use a TextIOWrapper to access the config. This is a hack, which cannot be used when it is binary data which has to be read (see later commits), so I don't consider this commit to be correct in that respect.

Testing several accessor classes causes code duplication, which can be avoided with help from the `parametrized` package (unfortunately, `pytest` support cannot be used together with `unittest`). Not a big deal right now, but starts becoming painful when adding new tests or testing other Accessor classes. Signed-off-by: Yann Dirson <[email protected]>

This test uses the same kind of I/O (file copy) that prepare_host_upgrade.py does. FIXME: the copy cannot proceed this way in python3

This works properly for the http case, but FileAccessor provides us with a text fileobj handle, and `read()` gets a UTF-8 decoding error. FIXME: Accessor ctor requires a `mode` argument

Signed-off-by: Yann Dirson <[email protected]>

Reported under python3 for members created on-the-fly in `setUp()` Signed-off-by: Yann Dirson <[email protected]>

With python3, pylint complains about `else: raise()` constructs. This rework avoids them and reduces cyclomatic complexity by using the error-out-first idiom. Signed-off-by: Yann Dirson <[email protected]>

diff-cover defaults to origin/main in new version, it seems. Signed-off-by: Yann Dirson <[email protected]>

bernhardkaindl · 2023-05-09T14:25:51Z

This PR has been used as the basis of #27 which as been merged to master now, closing as done!

…/CP-41819-pip-pyproject-isort CP-41819: Start py3 migration with config for isort and apply isort

ydirson force-pushed the testsuite-driven-py3 branch 2 times, most recently from 25326e7 to ec9d4ea Compare September 26, 2022 08:46

ydirson marked this pull request as draft September 26, 2022 08:47

ydirson force-pushed the testsuite-driven-py3 branch from ec9d4ea to 92b2913 Compare September 26, 2022 11:47

ydirson force-pushed the testsuite-driven-py3 branch from 92b2913 to c41e019 Compare September 26, 2022 12:19

ydirson mentioned this pull request Sep 26, 2022

Python3 compatibility #5

Closed

ydirson force-pushed the testsuite-driven-py3 branch 2 times, most recently from 503296c to c973ed2 Compare September 29, 2022 13:14

This was referenced Nov 3, 2022

Changing xcp.xmlunwrap API to use unicode (py3:str) instead of str (py3:bytes) #18

Closed

Changing xcp.accessor API to explicitly specify text or binary mode #19

Closed

ydirson mentioned this pull request Nov 28, 2022

Testcpio improvements, cpiofile improvements and port to py3 #16

Merged

MarkSymsCtx requested review from rosslagerwall and MarkSymsCtx January 20, 2023 15:50

psafont marked this pull request as ready for review January 20, 2023 16:20

ydirson and others added 12 commits January 20, 2023 17:45

python3: use six.string_types not version-dependant types

05a4e60

Use of `unicode` needed to be immediately handled, but a few checks relying on `str` could become insufficient in python2 with the larger usage of unicode strings. Signed-off-by: Yann Dirson <[email protected]>

python3: use "six.ensure_binary" and "six.ensure_text" for str/bytes …

84d172a

…conversion Signed-off-by: Yann Dirson <[email protected]>

Remove direct call's to file's constructor and replace them with call…

7ac16be

…s to open() as ths is considered best practice. (cherry picked from cpython commit 6cef076ba5edbfa42239924951d8acbb087b3b19) Signed-off-by: Yann Dirson <[email protected]>

python3: xcp.net.mac: use six.python_2_unicode_compatible for stringi…

520d419

…fication Signed-off-by: Yann Dirson <[email protected]>

xcp.net.ifrename.logic: use "logger.warning", "logger.warn" is deprec…

a59c3ba

…ated Signed-off-by: Yann Dirson <[email protected]>

python3: use raw strings for regexps, fixes insufficient quoting

0832486

Running tests on python3 did reveal some of them. Signed-off-by: Yann Dirson <[email protected]>

test_dom0: mock "open()" in a python3-compatible way

7e14499

Signed-off-by: Yann Dirson <[email protected]>

test_cpio: ensure paths are handled as text

ae79078

Caught by extended test. Signed-off-by: Yann Dirson <[email protected]>

cpiofile: migrate last "list.sort()" call still using a "cmp" argument

346ebc0

This goes away in python3. Signed-off-by: Yann Dirson <[email protected]>

WIP python3: fix xmlunwrap and its test to align with the use of bytes

5fd6cdf

FIXME: I'm quite unsure why xcp.xmlunwrap would want to use bytes and not unicode strings, but the encode/decode calls make it quite clear it wants to work with bytes. That makes the API painful to use in python3.

xcp.repository: switch from md5 to hashlib.md5

67225f7

hashlib came with python 2.5, and old md5 module disappears in 3.0 Signed-off-by: Yann Dirson <[email protected]>

ydirson added 9 commits January 20, 2023 17:45

WIP test_accessor: check for I/O on binary files

cdd5ee8

This test uses the same kind of I/O (file copy) that prepare_host_upgrade.py does. FIXME: the copy cannot proceed this way in python3

WIP test_accessor: write into copy file as binary

71183e4

This works properly for the http case, but FileAccessor provides us with a text fileobj handle, and `read()` gets a UTF-8 decoding error. FIXME: Accessor ctor requires a `mode` argument

Pylint complements: honor len-as-condition convention

d6566dd

Signed-off-by: Yann Dirson <[email protected]>

Pylint complements: whitespace in expressions

ac1b1b8

Signed-off-by: Yann Dirson <[email protected]>

Pylint complements: test_ifrename_logic: disable "no-member" warning

d58e988

Reported under python3 for members created on-the-fly in `setUp()` Signed-off-by: Yann Dirson <[email protected]>

Pylint complements: avoid no-else-raise "refactor" issues

efeecfa

With python3, pylint complains about `else: raise()` constructs. This rework avoids them and reduces cyclomatic complexity by using the error-out-first idiom. Signed-off-by: Yann Dirson <[email protected]>

CI: also run tests with python3

09ba68a

diff-cover defaults to origin/main in new version, it seems. Signed-off-by: Yann Dirson <[email protected]>

ydirson force-pushed the testsuite-driven-py3 branch from c973ed2 to 09ba68a Compare January 20, 2023 16:45

psafont mentioned this pull request Apr 18, 2023

CP-41972: update xcp to support ack py3 #21

Closed

This was referenced Apr 21, 2023

UTF-8 encoding fixes + tests for xcp.pci, .cmd and .net.biosdevname #22

Closed

xcp.accessor, xcp.repository: Use binary mode for file I/O #24

Closed

TDD Py3 merge: Commits up to test_accessor.py: Test reading binary data from http and file #25

Closed

bernhardkaindl mentioned this pull request May 9, 2023

Initial Python3 support with Test-Driven-Development enforced by CI (with git diff coverage and diff lint checking) + mypy, pyre and pytype #27

Merged

bernhardkaindl closed this May 9, 2023

bernhardkaindl mentioned this pull request Jun 9, 2023

Add Accessor.openText() as context manager for openAddress() #76

Merged

bernhardkaindl added a commit to rosslagerwall/python-libs that referenced this pull request May 8, 2024

Merge pull request xenserver#17 from xenserver-next/private/bernhardk…

ee16423

…/CP-41819-pip-pyproject-isort CP-41819: Start py3 migration with config for isort and apply isort

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Testsuite driven py3 #17

Testsuite driven py3 #17

Uh oh!

ydirson commented Sep 21, 2022 •

edited

Loading

Uh oh!

ydirson commented Sep 26, 2022

Uh oh!

ydirson commented Sep 26, 2022 •

edited

Loading

Uh oh!

edwintorok commented Sep 26, 2022

Uh oh!

ydirson commented Sep 26, 2022

Uh oh!

bernhardkaindl commented May 9, 2023

Uh oh!

Uh oh!

Testsuite driven py3 #17

Testsuite driven py3 #17

Uh oh!

Conversation

ydirson commented Sep 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ydirson commented Sep 26, 2022

Uh oh!

ydirson commented Sep 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edwintorok commented Sep 26, 2022

Uh oh!

ydirson commented Sep 26, 2022

Uh oh!

bernhardkaindl commented May 9, 2023

Uh oh!

Uh oh!

ydirson commented Sep 21, 2022 •

edited

Loading

ydirson commented Sep 26, 2022 •

edited

Loading