Skip to content

Commit c5bb69e

Browse files
rwgkwangxf123456
andauthored
Introduce return_value_policy_pack (will need follow-on adjustments) (#30011)
* [smart_holder] Implement `try_as_void_ptr_capsule` as a free function (pybind#4539) * Move try_as_void_ptr_capsule out from modified_type_caster_generic_load_impl. * Try fixing clangtidy * Introduce return_value_policy_pack Currently only for string_caster, tuple_caster, map_caster * Add return_value_policy_pack::override_policy() helpers. * PYBIND11_TYPE_CASTER_RVPP return_value_policy_pack * Systematically use return_value_policy_pack in stl.h * clang-tidy auto fix * Fix MSVC warning C4458: declaration of 'policy' hides class member * Fix oversight (when renaming return_value_policy_opts to return_value_policy_pack). * Cover all changed casters in stl.h * WIP callbacks * Explicitly specify return_value_policy::automatic_reference in functional.h * WIP proof of concept manipulating the return value policy for callbacks. * Introduce from_python_policies as generalization of load bool convert * Change `std::vector<bool> args_convert;` to `std::vector<from_python_policies> args_policies;` * clang-tidy auto fix * WIP: tests pass * Replace `argument_record.convert`,`none` with `from_python_policies` Core diff: ``` diff --git a/include/pybind11/attr.h b/include/pybind11/attr.h - bool Zonvert : 1; ///< True if the argument is allowed to convert when loading - bool Mone : 1; ///< True if None is allowed when loading + from_python_policies policies; ``` * functional.h using fpp.rvpp (instead of fpp.convert) but arg is non-constexpr - value = func_wrapper(func_handle(std::move(func)), fpp.convert); + value = func_wrapper(func_handle(std::move(func)), fpp.rvpp); Passes: scons -j 24 selected_test_cpp=test_exceptions.cpp,test_return_value_policy_pack.cpp && /usr/bin/python3 $HOME/clone/pybind11_scons/run_tests.py ../pybind11 test_return_value_policy_pack -s -vv But many other tests broken because arg is non-constexpr and therefore this had to be commented out in cast.h: constexpr arg operator"" _a(const char *name, size_t) { return arg(name); } * Split out `detail::arg_literal` to make ALL tests work again. * clang-format auto fixes * Quick workaround for MSVC private/friend issue. * Add test_nested_callbacks_rtn_string * WIP: Add collect_arguments_rvpp() functions (unused). * Use collect_arguments_rvpp() from object_api<Derived>::operator() * clang-format auto fixes * Make test_return_value_policy_pack.cpp compatible with C++11 (skip optional, variant). Also fix oversight: missing Python-side test for variant. * Add test_return_value_policy_pack to tests/CMakeLists.txt and clang-tidy auto fix. * Resolve `-Wmissing-braces` emitted by old clang versions (3.6, 3.7, 3.9, 5): ``` test_return_value_policy_pack.cpp:51:70: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces] ``` * Resolve "pointless comparison of unsigned integer with zero" warnings emitted by CUDA and ICC: CUDA 11.7 Ubuntu 22.04: ``` cast.h(1318): error pybind#186-D: pointless comparison of unsigned integer with zero ``` ICC latest x64 (Intel 2021.8.0.20221119): ``` cast.h(1318): error pybind#186: pointless comparison of unsigned integer with zero ``` * object_api<Derived>::call_with_policies() * Use object_api<Derived>::call_with_policies() in functional.h * test_call_callback_pass_pair_string * Systematically exercise nested callbacks to level 4. * make_tuple -> make_tuple_rvpp * simple_collector -> simple_collector_rvpp * unpacking_collector -> unpacking_collector_rvpp * PYBIND11_TYPE_CASTER_IMPL * Universally pass return_value_policy_pack via const & * Rename arg_literal to arg_base * Fix git rebase accident. * Suppress MINGW `-Wmismatched-new-delete` (seen in mingw32 & mingw64 ci.yml logs). ``` 2023-02-17T09:47:45.7114041Z D:\a\pybind11\pybind11\tests\test_factory_constructors.cpp:381:30: error: 'void operator delete(void*)' called on pointer returned from a mismatched allocation function [-Werror=mismatched-new-delete] 2023-02-17T09:47:45.7114692Z 381 | ::operator delete(p); 2023-02-17T09:47:45.7115055Z | ~~~~~~~~~~~~~~~~~^~~ 2023-02-17T09:47:45.7115444Z In lambda function, 2023-02-17T09:47:45.7116873Z inlined from 'pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>' at D:/a/pybind11/pybind11/include/pybind11/detail/init.h:376:33, 2023-02-17T09:47:45.7119232Z inlined from 'Return pybind11::detail::argument_loader<Args>::call_impl(Func&&, pybind11::detail::index_sequence<Is ...>, Guard&&) && [with Return = void; Func = pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>&; long long unsigned int ...Is = {0, 1, 2}; Guard = pybind11::detail::void_type; Args = {pybind11::detail::value_and_holder&, int, int}]' at D:/a/pybind11/pybind11/include/pybind11/cast.h:1563:37, 2023-02-17T09:47:45.7122014Z inlined from 'pybind11::detail::enable_if_t<std::is_void<_Dummy>::value, pybind11::detail::void_type> pybind11::detail::argument_loader<Args>::call(Func&&) && [with Return = void; Guard = pybind11::detail::void_type; Func = pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>&; Args = {pybind11::detail::value_and_holder&, int, int}]' at D:/a/pybind11/pybind11/include/pybind11/cast.h:1537:65, 2023-02-17T09:47:45.7125679Z inlined from 'pybind11::cpp_function::initialize<pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>, void, pybind11::detail::value_and_holder&, int, int, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>&&, void (*)(pybind11::detail::value_and_holder&, int, int), const pybind11::name&, const pybind11::is_method&, const pybind11::sibling&, const pybind11::detail::is_new_style_constructor&)::<lambda(pybind11::detail::function_call&)>' at D:/a/pybind11/pybind11/include/pybind11/pybind11.h:249:71, 2023-02-17T09:47:45.7130204Z inlined from 'static pybind11::handle pybind11::cpp_function::initialize<pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>, void, pybind11::detail::value_and_holder&, int, int, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>&&, void (*)(pybind11::detail::value_and_holder&, int, int), const pybind11::name&, const pybind11::is_method&, const pybind11::sibling&, const pybind11::detail::is_new_style_constructor&)::<lambda(pybind11::detail::function_call&)>::_FUN(pybind11::detail::function_call&)' at D:/a/pybind11/pybind11/include/pybind11/pybind11.h:224:21: 2023-02-17T09:47:45.7132694Z D:\a\pybind11\pybind11\tests\test_factory_constructors.cpp:399:71: note: returned from 'static void* test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc::operator new(size_t)' 2023-02-17T09:47:45.7133385Z 399 | pyNoisyAlloc.def(py::init([](int i, int) { return new NoisyAlloc(i); })); 2023-02-17T09:47:45.7133874Z | ^ 2023-02-17T09:47:45.7134610Z In static member function 'static void test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc::operator delete(void*, size_t)', 2023-02-17T09:47:45.7135502Z inlined from 'test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>' at D:\a\pybind11\pybind11\tests\test_factory_constructors.cpp:408:74, 2023-02-17T09:47:45.7139172Z inlined from 'pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>' at D:/a/pybind11/pybind11/include/pybind11/detail/init.h:376:33, 2023-02-17T09:47:45.7141646Z inlined from 'Return pybind11::detail::argument_loader<Args>::call_impl(Func&&, pybind11::detail::index_sequence<Is ...>, Guard&&) && [with Return = void; Func = pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>&; long long unsigned int ...Is = {0, 1, 2}; Guard = pybind11::detail::void_type; Args = {pybind11::detail::value_and_holder&, int, double}]' at D:/a/pybind11/pybind11/include/pybind11/cast.h:1563:37, 2023-02-17T09:47:45.7144252Z inlined from 'pybind11::detail::enable_if_t<std::is_void<_Dummy>::value, pybind11::detail::void_type> pybind11::detail::argument_loader<Args>::call(Func&&) && [with Return = void; Guard = pybind11::detail::void_type; Func = pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>&; Args = {pybind11::detail::value_and_holder&, int, double}]' at D:/a/pybind11/pybind11/include/pybind11/cast.h:1537:65, 2023-02-17T09:47:45.7147880Z inlined from 'pybind11::cpp_function::initialize<pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>, void, pybind11::detail::value_and_holder&, int, double, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>&&, void (*)(pybind11::detail::value_and_holder&, int, double), const pybind11::name&, const pybind11::is_method&, const pybind11::sibling&, const pybind11::detail::is_new_style_constructor&)::<lambda(pybind11::detail::function_call&)>' at D:/a/pybind11/pybind11/include/pybind11/pybind11.h:249:71, 2023-02-17T09:47:45.7358442Z inlined from 'static pybind11::handle pybind11::cpp_function::initialize<pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>, void, pybind11::detail::value_and_holder&, int, double, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>&&, void (*)(pybind11::detail::value_and_holder&, int, double), const pybind11::name&, const pybind11::is_method&, const pybind11::sibling&, const pybind11::detail::is_new_style_constructor&)::<lambda(pybind11::detail::function_call&)>::_FUN(pybind11::detail::function_call&)' at D:/a/pybind11/pybind11/include/pybind11/pybind11.h:224:21: ``` * Fix copy-paste mishap (thanks @lalaland for catching this). * ruff auto-fixes * Undo temporary change to test_gil_scoped.py --------- Co-authored-by: Xiaofei Wang <[email protected]>
1 parent f8128f0 commit c5bb69e

12 files changed

+863
-140
lines changed

include/pybind11/attr.h

Lines changed: 34 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -176,11 +176,13 @@ struct argument_record {
176176
const char *name; ///< Argument name
177177
const char *descr; ///< Human-readable version of the argument value
178178
handle value; ///< Associated Python object
179-
bool convert : 1; ///< True if the argument is allowed to convert when loading
180-
bool none : 1; ///< True if None is allowed when loading
179+
from_python_policies policies;
181180

182-
argument_record(const char *name, const char *descr, handle value, bool convert, bool none)
183-
: name(name), descr(descr), value(value), convert(convert), none(none) {}
181+
argument_record(const char *name,
182+
const char *descr,
183+
handle value,
184+
const from_python_policies &policies)
185+
: name(name), descr(descr), value(value), policies(policies) {}
184186
};
185187

186188
/// Internal data structure which holds metadata about a bound function (signature, overloads,
@@ -212,8 +214,8 @@ struct function_record {
212214
/// Pointer to custom destructor for 'data' (if needed)
213215
void (*free_data)(function_record *ptr) = nullptr;
214216

215-
/// Return value policy associated with this function
216-
return_value_policy policy = return_value_policy::automatic;
217+
/// Return value policies associated with this function
218+
return_value_policy_pack rvpp;
217219

218220
/// True if name == '__init__'
219221
bool is_constructor : 1;
@@ -359,7 +361,7 @@ struct type_record {
359361

360362
inline function_call::function_call(const function_record &f, handle p) : func(f), parent(p) {
361363
args.reserve(f.nargs);
362-
args_convert.reserve(f.nargs);
364+
args_policies.reserve(f.nargs);
363365
}
364366

365367
/// Tag for a new-style `__init__` defined in `detail/init.h`
@@ -407,7 +409,12 @@ struct process_attribute<char *> : process_attribute<const char *> {};
407409
/// Process an attribute indicating the function's return value policy
408410
template <>
409411
struct process_attribute<return_value_policy> : process_attribute_default<return_value_policy> {
410-
static void init(const return_value_policy &p, function_record *r) { r->policy = p; }
412+
static void init(const return_value_policy &p, function_record *r) { r->rvpp.policy = p; }
413+
};
414+
template <>
415+
struct process_attribute<return_value_policy_pack>
416+
: process_attribute_default<return_value_policy_pack> {
417+
static void init(const return_value_policy_pack &rvpp, function_record *r) { r->rvpp = rvpp; }
411418
};
412419

413420
/// Process an attribute which indicates that this is an overloaded function associated with a
@@ -455,7 +462,8 @@ inline void check_kw_only_arg(const arg &a, function_record *r) {
455462

456463
inline void append_self_arg_if_needed(function_record *r) {
457464
if (r->is_method && r->args.empty()) {
458-
r->args.emplace_back("self", nullptr, handle(), /*convert=*/true, /*none=*/false);
465+
r->args.emplace_back(
466+
"self", nullptr, handle(), from_python_policies(/*convert=*/true, /*none=*/false));
459467
}
460468
}
461469

@@ -464,19 +472,27 @@ template <>
464472
struct process_attribute<arg> : process_attribute_default<arg> {
465473
static void init(const arg &a, function_record *r) {
466474
append_self_arg_if_needed(r);
467-
r->args.emplace_back(a.name, nullptr, handle(), !a.flag_noconvert, a.flag_none);
475+
r->args.emplace_back(
476+
a.name,
477+
nullptr,
478+
handle(),
479+
from_python_policies(a.m_policies.rvpp, !a.flag_noconvert, a.flag_none));
468480

469481
check_kw_only_arg(a, r);
470482
}
471483
};
484+
template <>
485+
struct process_attribute<detail::arg_base> : process_attribute<arg> {};
472486

473487
/// Process a keyword argument attribute (*with* a default value)
474488
template <>
475489
struct process_attribute<arg_v> : process_attribute_default<arg_v> {
476490
static void init(const arg_v &a, function_record *r) {
477491
if (r->is_method && r->args.empty()) {
478-
r->args.emplace_back(
479-
"self", /*descr=*/nullptr, /*parent=*/handle(), /*convert=*/true, /*none=*/false);
492+
r->args.emplace_back("self",
493+
/*descr=*/nullptr,
494+
/*parent=*/handle(),
495+
from_python_policies(/*convert=*/true, /*none=*/false));
480496
}
481497

482498
if (!a.value) {
@@ -505,7 +521,11 @@ struct process_attribute<arg_v> : process_attribute_default<arg_v> {
505521
"more information.");
506522
#endif
507523
}
508-
r->args.emplace_back(a.name, a.descr, a.value.inc_ref(), !a.flag_noconvert, a.flag_none);
524+
r->args.emplace_back(
525+
a.name,
526+
a.descr,
527+
a.value.inc_ref(),
528+
from_python_policies(a.m_policies.rvpp, !a.flag_noconvert, a.flag_none));
509529

510530
check_kw_only_arg(a, r);
511531
}
@@ -667,7 +687,7 @@ using extract_guard_t = typename exactly_one_t<is_call_guard, call_guard<>, Extr
667687

668688
/// Check the number of named arguments at compile time
669689
template <typename... Extra,
670-
size_t named = constexpr_sum(std::is_base_of<arg, Extra>::value...),
690+
size_t named = constexpr_sum(std::is_base_of<arg_base, Extra>::value...),
671691
size_t self = constexpr_sum(std::is_same<is_method, Extra>::value...)>
672692
constexpr bool expected_num_args(size_t nargs, bool has_args, bool has_kwargs) {
673693
PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(nargs, has_args, has_kwargs);

0 commit comments

Comments
 (0)