Support pickling slots in subclasses of common classes

[serhiy-storchaka]

BPO	26579
Nosy	@rhettinger, @pitrou, @vstinner, @avassalotti, @bitdancer, @berkerpeksag, @serhiy-storchaka
PRs	bpo-26579: Add object.__getstate__(). #2821
Files	copyreg_getstate.patch copyreg_getstate2.patch object_getstate.patch: Add object.getstate

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-04-06.17:02:02.310>
created_at = <Date 2016-03-17.08:43:06.421>
labels = ['extension-modules', 'type-feature', 'library', '3.11']
title = 'Support pickling slots in subclasses of common classes'
updated_at = <Date 2022-04-07.14:57:28.953>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2022-04-07.14:57:28.953>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-04-06.17:02:02.310>
closer = 'serhiy.storchaka'
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2016-03-17.08:43:06.421>
creator = 'serhiy.storchaka'
dependencies = []
files = ['42183', '43462', '43716']
hgrepos = []
issue_num = 26579
keywords = ['patch']
message_count = 5.0
messages = ['261903', '268830', '270403', '416888', '416931']
nosy_count = 7.0
nosy_names = ['rhettinger', 'pitrou', 'vstinner', 'alexandre.vassalotti', 'r.david.murray', 'berker.peksag', 'serhiy.storchaka']
pr_nums = ['2821']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26579'
versions = ['Python 3.11']

Linked PRs

• gh-70766: Mention the object getstate caveat in 3.11 What's new. #108379
• [3.12] gh-70766: Mention the object getstate caveat in 3.11 What's new. (GH-108379) #108384
• [3.11] gh-70766: Mention the object getstate caveat in 3.11 What's new. (GH-108379) #108385

[serhiy-storchaka]

Pickling and copying instances of subclasses of some basic classes pickles and copies instance attributes. Example:

>>> class BA(bytearray):
...     pass
... 
>>> b = BA(b'abc')
>>> b.x = 10
>>> c = copy.copy(b)
>>> c.x
10
>>> c = pickle.loads(pickle.dumps(b))
>>> c.x
10

But this doesn't work if attributes are saved not in instance dictionary, but in slots.

>>> class BA(bytearray):
...     __slots__ = ('x',)
... 
>>> b = BA(b'abc')
>>> b.x = 10
>>> c = copy.copy(b)
>>> c.x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: x
>>> c = pickle.loads(pickle.dumps(b))
>>> c.x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: x

Since using __slots__ is implementation detail, this failure can be considered as a bug.

Proposed patch adds support of pickling and copying slots in subclasses of all classes that already support pickling and copying non-slot attributes. It is backward compatible, classes with slots can be unpickled on older Python versions without slots. Affected classes: bytearray, set, frozenset, weakref.WeakSet, collections.OrderedDict, collections.deque, datetime.tzinfo.

The patch adds the copyreg._getstate() function for Python classes and exposes the _PyObject_GetState() function for extension classes. An alternative (and simpler for end user) solution would be to add default implementation as object.__getstate__(). But this is not easy to reject non-pickleable classes (bpo-22995) in this case, since __getstate__ is looked up as instance attribute, not as other special methods.

[serhiy-storchaka]

Synchronized with current sources.

[serhiy-storchaka]

An alternative way is to expose a default state as object.__getstate__(). It is more efficient since it is implemented in C. Following patch implements this approach.

[serhiy-storchaka]

New changeset 884eba3 by Serhiy Storchaka in branch 'main':
bpo-26579: Add object.__getstate__(). (GH-2821)
884eba3

[vstinner]

bpo-26579: Add object.__getstate__(). (GH-2821)
884eba3

This change introduced reference leaks: see bpo-47250.