Improve mixed-version handling

[llchan]

I upgraded pybind11 and rebuilt an extension, but there was another extension in an unrelated package/repo that was not rebuilt. When I imported both of them, the process segfaulted.

Can we add some runtime version checking to ensure that an newly imported module is compatible with any existing pybind11 version(s) that have already been loaded? It would be a better user experience to get an informative exception about mismatched versions rather than a segfault + gdb on a core file to figure out it's pybind11-related.

I'm not sure how strict we want to make the version checking, but this particular instance was 2.1.x vs 2.0.x, so minor versions arent compatible. Should we add an internal ABI version that evolves independent of the user-facing API version?

[wjakob]

Hi @llchan,

pybind11 versions with different minor versions live in different namespaces, i.e. they don't see each other and can't even exchange objeccts. For this reason, I would be quite surprised if the crash is really a pybind11 issue. Do you have a reproducible way of triggering the issue?

Wenzel

[jagerman]

I'm not seeing where we do that. It seems to me that if Python loads mx.so compiled with pybind11 2.x and my.so compiled with 2.y (x ≠ y) we're going to run into issues because there is only one pybind11_object_new (and likewise for various other functions).

Am I missing something?

[wjakob]

The PYBIND11_INTERNALS_ID macro generates a version-specific ID that isolates the internals data structure from other ones. This used to be enough, though it's possible that later commits broke this.

[wjakob]

(see cast.h:51 and common.h:188)

[jagerman]

Ah, right. I don't see how it could feasibly be enough now, though: we can't expect that different versions of the extern "C" functions, in particular, are going to be compatible across versions.

@nbecker also ran into this today, with the same cause—modules compiled under different versions. (Reported today in the gitter channel).

[jagerman]

Cc @dean0x7d for thoughts.

[dean0x7d]

I don't see why pybind11_object_new and the other extern "C" functions would be an issue. Even within the same pybind11 version, each .so file will get its own set of those functions with unique addresses. Types wrapped in that .so will point only to their own functions (e.g. type->tp_new). None of those functions are visible outside their own .so files. Only the PyInit_<module_name> function is exported.

The only thing that I can think of is some inheritance situation where a type from one .so inherits type->tp_new from a type from another .so. Could that occur somehow?

[llchan]

I just looked back at the project that hit this, and one important detail I forgot to mention was that the .sos are loaded with RTLD_GLOBAL. This is necessary because they link to a shared library that has global state that must be shared across multiple extension modules (not my library, so I can't change that).

Is there something I should be doing with symbol visibility that I'm not doing? Something to make the library's symbols shared, but the pybind11 symbols module-local?

[llchan]

To give a more concrete example:

There's one pybind11 module foo which has hooks into libA.
foo.cpp -> libA.so

There's anther pybind11 module bar which has hooks into libB, which also links against libA.
bar.cpp -> libB.so -> libA.so

The two libAs must be shared, because they share global state. How can I do this safely? I don't need it to work with mismatched versions, I just want it to emit an error message rather than a mysterious segfault at an unpredictable point in the program [edit: now that I think about it I think it was deterministic, on the second import, but the point still stands that a segfault is not python-friendly].

[jagerman]

I've tracked this down to the static internals *internals_ptr in get_internals(): if I load two different modules, one built where I changed the MINOR version macro to 1 instead of 2, I get this when loading the first one:

get_internals called; internal id: __pybind11_2_2__
internals_ptr = 0
get_internals called; internal id: __pybind11_2_2__
internals_ptr = 0x56459c69a6d0
...

then loading the second:

Import a done. Importing b
get_internals called; internal id: __pybind11_2_1__
internals_ptr = 0x56459c69a6d0

The segfault then happens because, as @dean0x7d guessed, there's mixed version inheritance, but it's happening through the base pybind11_object_48 object.

[jagerman]

Also informative: this only happens with GCC. Under clang:

get_internals called; internal id: __pybind11_2_2__
internals_ptr = 0
get_internals called; internal id: __pybind11_2_2__
internals_ptr = 0x56228e03c3a0

...

Import a done. Importing b
get_internals called; internal id: __pybind11_2_1__
internals_ptr = 0
get_internals called; internal id: __pybind11_2_1__
internals_ptr = 0x56228e09caa0

[nbecker]

As far as I know, none of my code causing segfault had any inheritance.

[jagerman]

Not explicitly, but all pybind registered objects inherit from an internal pybind11_object_xx (where xx is a number), and it was in that implicit inheritance where the problem came in (under GCC specifically).

The PR I filed won't fix problems with incompatible dev versions (e.g. compiled with different, incompatible master commits), but should fix mixed release modules.