xcp.accessor, xcp.repository: Use binary mode for file I/O #24
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Use of `unicode` needed to be immediately handled, but a few checks relying on `str` could become insufficient in python2 with the larger usage of unicode strings. Signed-off-by: Yann Dirson <[email protected]>
…conversion Signed-off-by: Yann Dirson <[email protected]>
…s to open() as ths is considered best practice. (cherry picked from cpython commit 6cef076ba5edbfa42239924951d8acbb087b3b19) Signed-off-by: Yann Dirson <[email protected]>
…fication Signed-off-by: Yann Dirson <[email protected]>
…ated Signed-off-by: Yann Dirson <[email protected]>
Running tests on python3 did reveal some of them. Signed-off-by: Yann Dirson <[email protected]>
Signed-off-by: Yann Dirson <[email protected]>
There is no guaranty about ordering of dict elements, and tests compare results derived from enumerating a dict element. We could have used an OrderedDict to store the formulae and get a predictible output order, but just considering the output as a set seems better. Only applying this to rules expected to hold more than one element. Signed-off-by: Yann Dirson <[email protected]>
Caught by extended test. Signed-off-by: Yann Dirson <[email protected]>
This goes away in python3. Signed-off-by: Yann Dirson <[email protected]>
FIXME: I'm quite unsure why xcp.xmlunwrap would want to use bytes and not unicode strings, but the encode/decode calls make it quite clear it wants to work with bytes. That makes the API painful to use in python3.
hashlib came with python 2.5, and old md5 module disappears in 3.0 Signed-off-by: Yann Dirson <[email protected]>
This is supposed to be just a module renaming to conform to PEP8, see https://docs.python.org/3/whatsnew/3.0.html#library-changes The SafeConfigParser class has been renamed to ConfigParser in Python 3.2, and backported as addon package. The `readfp` method now triggers a deprecation warning to replace it with `read_file`. FIXME: With python3 some Accessor implementations (e.g. FileAccessor) provide a text stream for repository config (and with python2 all implementations), while others (e.g. HTTPAccessor) provide a binary stream. But on python3 ConfigParser will bomb out if given a binary stream, so use a TextIOWrapper to access the config. This is a hack, which cannot be used when it is binary data which has to be read (see later commits), so I don't consider this commit to be correct in that respect.
Testing several accessor classes causes code duplication, which can be avoided with help from the `parametrized` package (unfortunately, `pytest` support cannot be used together with `unittest`). Not a big deal right now, but starts becoming painful when adding new tests or testing other Accessor classes. Signed-off-by: Yann Dirson <[email protected]>
This test uses the same kind of I/O (file copy) that prepare_host_upgrade.py does. FIXME: the copy cannot proceed this way in python3
This works properly for the http case, but FileAccessor provides us with a text fileobj handle, and `read()` gets a UTF-8 decoding error. FIXME: Accessor ctor requires a `mode` argument
Signed-off-by: Yann Dirson <[email protected]>
Signed-off-by: Yann Dirson <[email protected]>
Reported under python3 for members created on-the-fly in `setUp()` Signed-off-by: Yann Dirson <[email protected]>
With python3, pylint complains about `else: raise()` constructs. This rework avoids them and reduces cyclomatic complexity by using the error-out-first idiom. Signed-off-by: Yann Dirson <[email protected]>
diff-cover defaults to origin/main in new version, it seems. Signed-off-by: Yann Dirson <[email protected]>
Even though .github/workflows/main.yml does a curl of branding.py GitHub CI still failed with ImportError for branding. Signed-off-by: Bernhard Kaindl <[email protected]>
Signed-off-by: Bernhard Kaindl <[email protected]>
…for de/encoding Signed-off-by: Bernhard Kaindl <[email protected]>
bernhardkaindl
force-pushed
the
testsuite-driven-py3-xcp.accessor-use-binary-mode
branch
from
28be2cd to
b5bd2e2
Compare
bernhardkaindl
added a commit
to xenserver-next/python-libs
that referenced
this pull request
Apr 25, 2023
Fix issue xenserver#19 based on the description and progress from PR xenserver#24. Allows for opening text and binary files in text and binary modes. Mode, encoding and error handling can be set by passing the parameters "encoding" and "errors" using the kwargs parameters from openAddress() and writeFile() to open(mode, **kwargs) and ftp.makefile(mode, **kwargs). Signed-off-by: Bernhard Kaindl <[email protected]>
bernhardkaindl
added a commit
to xenserver-next/python-libs
that referenced
this pull request
Apr 25, 2023
Fix issue xenserver#19 based on the description and progress from PR xenserver#24. Allows for opening text and binary files in text and binary modes. Mode, encoding and error handling can be set by passing the parameters "encoding" and "errors" using the kwargs parameters from openAddress() and writeFile() to open(mode, **kwargs) and ftp.makefile(mode, **kwargs). Signed-off-by: Bernhard Kaindl <[email protected]>
bernhardkaindl
added a commit
to xenserver-next/python-libs
that referenced
this pull request
Apr 26, 2023
Fix issue xenserver#19 based on the description and progress from PR xenserver#24. Allows for opening text and binary files in text and binary modes. Mode, encoding and error handling can be set by passing the parameters "encoding" and "errors" using the kwargs parameters from openAddress() and writeFile() to open(mode, **kwargs) and ftp.makefile(mode, **kwargs). Signed-off-by: Bernhard Kaindl <[email protected]>
bernhardkaindl
added a commit
to xenserver-next/python-libs
that referenced
this pull request
Apr 26, 2023
Fix issue xenserver#19 based on the description and progress from PR xenserver#24. Allows for opening text and binary files in text and binary modes. Mode, encoding and error handling can be set by passing the parameters "encoding" and "errors" using the kwargs parameters from openAddress() and writeFile() to open(mode, **kwargs) and ftp.makefile(mode, **kwargs). Signed-off-by: Bernhard Kaindl <[email protected]>
bernhardkaindl
added a commit
to xenserver-next/python-libs
that referenced
this pull request
Apr 26, 2023
Fix issue xenserver#19 based on the description and progress from PR xenserver#24. Allows for opening text and binary files in text and binary modes. Mode, encoding and error handling can be set by passing the parameters "encoding" and "errors" using the kwargs parameters from openAddress() and writeFile() to open(mode, **kwargs) and ftp.makefile(mode, **kwargs). Signed-off-by: Bernhard Kaindl <[email protected]>
bernhardkaindl
added a commit
to xenserver-next/python-libs
that referenced
this pull request
Apr 28, 2023
Fix issue xenserver#19 based on the description and progress from PR xenserver#24. Allows for opening text and binary files in text and binary modes. Mode, encoding and error handling can be set by passing the parameters "encoding" and "errors" using the kwargs parameters from openAddress() and writeFile() to open(mode, **kwargs) and ftp.makefile(mode, **kwargs). Signed-off-by: Bernhard Kaindl <[email protected]>
bernhardkaindl
added
Will be fixed by other PRs
|
Closing as obsoleted by other PRs being worked on now and being prepared. |
bernhardkaindl
added a commit
to rosslagerwall/python-libs
that referenced
this pull request
May 8, 2024
…/py2-py3-six.moves-urllib Update urlopen() and getoutput() to support Python3 as well
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
xcp.accessor must be able to access binary file content.
One example are the bootloader files. The
https://github.com/ydirson/xenserver-python-libs/blob/testsuite-driven-py3/tests/test_accessor.py#L21 updated by #17 reads
boot/isolinux/mboot.c32to show this.To support accessing binary files, open(mode="b") must be used, Otherwise (at least when the interpreter's effective
LC_CTYPElocale/charset is UTF-8) encoding arbitrary binary data into Unicode will fail on reading and writing.encoding=anderrors=toopen()andmakefile()(for FTP accesses) to enable inserting the givencodecwith the given handling of codingerrors.Explanation:
Conversion between the Python3
strandbytesduring I/O requires anencoding. While it can be assumed to byutf-8, at least in theory, for historical reasons, files could use other encodings.One example is https://raw.githubusercontent.com/ydirson/xenserver-python-libs/testsuite-driven-py3/xcp/cpiofile.py which is still encoded using iso-8859-1 as you can see by the broken display of the name of Lars Gustäbel in
Copyright (C) 2002 Lars Gust�bel <[email protected]>when retrieving the raw, unconverted file.When decoding bytes using the UTF-8 codec to Unicode for the Python3
strtype, errors can occur when the input is not 100% well-formed, and there are many valid options to handle such errors.This shows that a simplification just decode
bytesusing the UTF-8 decoder tostris risky.Most Python3 program dealing with strings from outside sources will have to deal with them and need manual attention and at least testing when converting them to Python3. The second commit provides the flexibility to pass
encoding=anderrors=when a conversion to/fromstris desired.