Regression in 3.14: setting __dict__ on custom type broken (mypyc, pybind11)

[henryiii]

Bug report

Bug description:

The work in #115776 by @markshannon broke mypyc and pybind11's tests related to pickling and setting __dict__ on custom types. Specially, #117750 introduced the regression for pybind11, I tested all the released for 3.14 then bisected between the commit after the fork point and 3.14.0a1 to find the commit in the PR above introduced the issue. More types are now inlined, but this breaks dict setting.

• [mypyc] Pickle test failure with Python 3.14.0a1 mypy#17973
• Pickle test failure with Python 3.14.0a5+ mypyc/mypyc#1091
• fix: support Python 3.14 pybind/pybind11#5646

I've prepared a MWE without pybind11 or mypyc:

src/main.c

#define PY_SSIZE_T_CLEAN
#include <Python.h>

typedef struct {
    PyObject_VAR_HEAD
} ManagedDictObject;

int ManagedDict_traverse(PyObject *self, visitproc visit, void *arg) {
    PyObject_VisitManagedDict(self, visit, arg);
    Py_VISIT(Py_TYPE(self));
    return 0;
}

int ManagedDict_clear(PyObject *self) {
    PyObject_ClearManagedDict(self);
    return 0;
}

static PyGetSetDef ManagedDict_getset[] = {
    {"__dict__", PyObject_GenericGetDict, PyObject_GenericSetDict, NULL, NULL},
    {NULL, NULL, NULL, NULL, NULL},
};

static PyType_Slot ManagedDict_slots[] = {
    {Py_tp_new, (void *)PyType_GenericNew},
    {Py_tp_getset, (void *)ManagedDict_getset},
    {Py_tp_traverse, (void *)ManagedDict_traverse},
    {Py_tp_clear, (void *)ManagedDict_clear},
    {0}
};

static PyType_Spec ManagedDict_spec = {
    "manageddictbug.ManagedDict",
    sizeof(ManagedDictObject),
    0, // itemsize
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_MANAGED_DICT | Py_TPFLAGS_HEAPTYPE | Py_TPFLAGS_HAVE_GC,
    ManagedDict_slots
};

static PyModuleDef manageddictbugmodule = {
    PyModuleDef_HEAD_INIT,
    "manageddictbug",
    NULL,
    -1,
    NULL,
};

PyMODINIT_FUNC
PyInit_manageddictbug(void) {
    PyObject *m = PyModule_Create(&manageddictbugmodule);
    if (m == NULL)
        return NULL;

    PyObject *ManagedDictType = PyType_FromSpec(&ManagedDict_spec);
    if (ManagedDictType == NULL) {
        Py_DECREF(m);
        return NULL;
    }

    if (PyModule_AddObject(m, "ManagedDict", ManagedDictType) < 0) {
        Py_DECREF(ManagedDictType);
        Py_DECREF(m);
        return NULL;
    }

    return m;
}

pyproject.toml

[build-system]
requires = ["scikit-build-core"]
build-backend = "scikit_build_core.build"

[project]
name = "example-broken"
version = "0.1.0"

CMakeLists.txt

cmake_minimum_required(VERSION 3.15...4.0)
project(${SKBUILD_PROJECT_NAME} LANGUAGES C)

find_package(Python REQUIRED COMPONENTS Development.Module)

python_add_library(manageddictbug MODULE WITH_SOABI src/main.c)
install(TARGETS manageddictbug DESTINATION .)

example.py

import manageddictbug

obj = manageddictbug.ManagedDict()
obj.foo = 42
print(obj.foo)
print(obj.__dict__)

obj.__dict__ = {"bar": 3}
print(obj.__dict__)
print(obj.bar)

$ uv venv -p 3.13 -q && uv pip install . -q && .venv/bin/python example.py
42
{'foo': 42}
{'bar': 3}
3
$ uv venv -p ~/git/software/pybind11/.venv/bin/python3.14 -q && uv pip install . -q && .venv/bin/python example.py
42
{'foo': 42}
{'bar': 3}
Traceback (most recent call last):
  File "/Users/henryschreiner/git/scikit-build-proj/example_broken/example.py", line 13, in <module>
    print(obj.bar)
          ^^^^^^^
AttributeError: 'manageddictbug.ManagedDict' object has no attribute 'bar'

There is custom code in PyObject_GenericSetDict that is supposed to be handling the inline case, but maybe it wasn't hit before and is faulty?

CPython versions tested on:

3.14.0b1 (and back to 3.14.0a1)

Operating systems tested on:

macOS (and Linux in CI too; Windows CI broken due to https://gitlab.kitware.com/cmake/cmake/-/issues/26926, which I haven't opened a CPython issue for yet).

Linked PRs

• GH-133912: Fix PyObject_GenericSetDict to handle inline values #134725
• [3.14] GH-133912: Fix PyObject_GenericSetDict to handle inline values (GH-134725) #134859

[encukou]

Marking as a potential release blocker.

[encukou]

@markshannon, do you want to investigate, or should I?

[hugovk]

I'm not planning on blocking beta 2 for this, but let's try and resolve it for beta 3 -- 2025-06-17, three weeks from now.

@markshannon Please can you look into it?

[markshannon]

Yes. Thanks for the reminder

[cdce8p]

Can confirm #134725 fixes the mypyc test case as well. Thanks @henryiii for providing the independent reproducer and @markshannon for the fix! For mypy / mypyc this was the last test failure for 3.14. Looking forward to 3.14.0b3 🚀

[itamaro]

Looks like this is done, thanks everyone!
Closing, but please reopen if there's anything left to do here.