bpo-33056 FIX leaking fd in concurrent.futures.ProcessPoolExecutor

[tomMoral]

The recent changes introduced by #3895 leaks some file descriptors (the Pipe open in _ThreadWakeup, see here).
They should be properly closed at shutdown.

https://bugs.python.org/issue33056

[asvetlov]

Please add a NEWS entry

[bedevere-bot]

@asvetlov: Please replace # with GH- in the commit message next time. Thanks!

[tomMoral]

Thank you for the fast review! I will try to use GH- next time.

[asvetlov]

@tomMoral sure.
It's my bad, committer should edit commit message always.
Unfortunately it cannot be done by robot, thus we have too many messages like this from bot.
Very common problem for core devs

[miss-islington]

Thanks @tomMoral for the PR, and @asvetlov for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

[bedevere-bot]

GH-6092 is a backport of this pull request to the 3.7 branch.

[rad-pat]

@tomMoral, I believe that either this change and/or #3895 causes OSError: handle is closed when shutting down ProcessPoolExecutor with wait=False due to self._queue_management_thread_wakeup being closed, but still referenced in _threads_wakeups on exit of Python, e.g. with script:

from concurrent.futures import ProcessPoolExecutor


class ObjectWithPickleError():
    """Triggers a RuntimeError when sending job to the workers"""

    def __reduce__(self):
        raise RuntimeError()


if __name__ == "__main__":
    e = ProcessPoolExecutor()
    f = e.submit(id, ObjectWithPickleError())
    e.shutdown(wait=False)
    # f.result()  # Deadlock on get

I'm not sure what the right solution is here, obviously this change is to prevent leaking the file descriptors which are taken care of in the close(), but perhaps the _python_exit method needs a review. Hopefully you'll have some insight please?

[tomMoral]

@rad-pat Thanks a lot for the reproducing script!

there is 2 issues here:

• The first one is the one I described and tried to fix in bpo-39104: Fix hanging ProcessPoolExecutor on shutdown nowait with pickling failure #17670. The queue_management_thread should handle all the closing and the wakeup method should be robust to closed fd as there might be concurent events that make the executor shutdown. This explains the hang when using the f.result
• The second one is due to the fact that on_queue_feeder_error is ignored when the interpreter is closing. This means that the queue_management_thread will never be notified of the fact that the pending task it is waiting already failed. For that, I don't really know what is the right solution.