bpo-31540 Add context management for concurrent.futures.ProcessPoolExecutor #3682

tomMoral · 2017-09-21T09:00:13Z

The ProcessPoolExecutor processes start method can only be change by changing the global default context with set_start_method at the beginning of a script. We propose to allow passing a context argument in the constructor to allow more flexible control of the executor. Doing so, we also add some tests for all the available context, to make sure the executor is working correctly.

In addition, we made the following changes, which can be put in another PR if necessary:

Rename the global variable _shutdown to _global_shutdown to make its function in the code clearer.
Liberate the ressource earlier in the _worker_process. Indeed, with the actual behavior, the ressources are not freed before the worker receives a new task or shutdown.

This work was done as part of the loky project in collaboration with
@ogrisel. See #1013 for the details.

https://bugs.python.org/issue31540

tomMoral/loky#48 * Add context argument to allow non forking ProcessPoolExecutor * Do some cleaning (pep8+nonused code+naming) * Liberate the ressource earlier in the `_worker_process`

pitrou · 2017-10-02T15:30:49Z

Lib/concurrent/futures/process.py

@@ -180,6 +177,11 @@ def _process_worker(call_queue, result_queue):
            result_queue.put(_ResultItem(call_item.work_id,
                                         result=r))

+        # Liberate the resource as soon as possible, to avoid holding onto


Would it be easy to add a test for this?

I added a test for this behavior.

pitrou · 2017-10-02T15:31:45Z

Lib/concurrent/futures/process.py

        """Initializes a new ProcessPoolExecutor instance.

        Args:
            max_workers: The maximum number of processes that can be used to
                execute the given calls. If None or not given then as many
                worker processes will be created as the machine has processors.
+            context: A multiprocessing context to launch the workers. This


As a nit, I think calling this parameter mp_context would be a bit more explicit.

pitrou · 2017-10-02T15:34:37Z

Lib/test/test_concurrent_futures.py

-            t = {executor_type}(5)
-            t.submit(sleep_and_print, 1.0, "apple")
+            if __name__ == "__main__":
+                t = {executor_type}(5)


Perhaps we want to pass the right context argument here?

pitrou · 2017-10-02T15:35:20Z

.travis.yml

@@ -56,7 +56,7 @@ matrix:
            ./venv/bin/python -m test.pythoninfo
      script:
        # Skip tests that re-run the entire test suite.
-        - ./venv/bin/python -m coverage run --pylib -m test --fail-env-changed -uall,-cpu -x test_multiprocessing_fork -x test_multiprocessing_forkserver -x test_multiprocessing_spawn
+        - ./venv/bin/python -m coverage run --pylib -m test --fail-env-changed -uall,-cpu -x test_multiprocessing_fork -x test_multiprocessing_forkserver -x test_multiprocessing_spawn -x test_concurrent_futures


I forgot the rationale for this?

When coverage is used with multiprocessing, the spawning of new interpreter launch a new test session, resulting in a mess. I think it was previously working with the fork context but when I launch the test with the three backends, it also launches new test sessions.

Thus, I disabled coverage tests with test_concurrent_futures. I am not sure of what I should do if this is not the case. The duplicated test sessions could results from some command line arguments parsing either in the semaphore tracker or the forkserver but I do not think it is linked to this PR.

Let me know if this make sense.

It does make sense, thank you.

pitrou · 2017-10-02T15:36:12Z

Thank you! This looks fine on the principle (apart from a couple small things mentioned in the review). You'll need to update the docs in Doc/library/concurrent.futures.rst.

- Rename context to mp_context in ProcessPoolExecutor constructor - Fix the context used in test_interpreter_shutdown

- Ensure that the job argument passed are freed asap

pitrou · 2017-10-03T07:00:43Z

Doc/library/concurrent.futures.rst


   An :class:`Executor` subclass that executes calls asynchronously using a pool
   of at most *max_workers* processes.  If *max_workers* is ``None`` or not
   given, it will default to the number of processors on the machine.
   If *max_workers* is lower or equal to ``0``, then a :exc:`ValueError`
   will be raised.
+   *mp_context* can be a multiprocessing context or any object providing a


I'd rather restrict it to a multiprocessing context. Perhaps we'll use other multiprocessing APIs in the future.

pitrou · 2017-10-03T07:02:29Z

Lib/test/test_concurrent_futures.py

@@ -55,6 +56,15 @@ def my_method(self):
        pass


+class EventfulGCObj():
+    def __init__(self, ctx):
+        mgr = get_context(ctx).Manager()


What is the rationale for using a manager here? Since the object is instantiated in the parent, it should be inheritable by the child anyway.

The executor is launched in the setup method of the TestCase. Thus, there is no possibility to pass the Event object thru inheritence and the job (id, obj) is passed via pickle, which requires the Manager.

Oh, right. I've never used managers and I was surprised to see this...

pitrou · 2017-10-03T07:03:15Z

Lib/test/test_concurrent_futures.py

+        future = self.executor.submit(id, obj)
+        future.result()
+
+        assert obj.event.wait(timeout=1)


By convention, we'd use self.assertTrue (also so that assertions are still checked if running with -O).

pitrou · 2017-10-03T09:53:35Z

Thank you @tomMoral !

tomMoral · 2017-10-03T10:24:48Z

Thank @pitrou
I will follow up with the other PR from #1013 asap.

bedevere-bot added the awaiting review label Sep 21, 2017

the-knights-who-say-ni added the CLA signed label Sep 21, 2017

tomMoral changed the title ~~Add context management for concurrent.futures.ProcessPoolExecutor~~ bpo-31540 Add context management for concurrent.futures.ProcessPoolExecutor Sep 21, 2017

tomMoral mentioned this pull request Sep 21, 2017

bpo-30006 More robust concurrent.futures.ProcessPoolExecutor #1013

Closed

Add context management for ProcessPoolExecutor+CLN

6376291

tomMoral/loky#48 * Add context argument to allow non forking ProcessPoolExecutor * Do some cleaning (pep8+nonused code+naming) * Liberate the ressource earlier in the `_worker_process`

tomMoral force-pushed the PR_contextual_executor branch from f2f41e0 to 6376291 Compare September 22, 2017 12:25

tomMoral added 2 commits September 22, 2017 14:55

FIX skip tests that re-run the entire test suite

e787b6f

NEW add whatsnew entry

926ad27

pitrou reviewed Oct 2, 2017

View reviewed changes

tomMoral added 3 commits October 2, 2017 18:22

CLN contex->mp_context+FIX context in test

a945891

- Rename context to mp_context in ProcessPoolExecutor constructor - Fix the context used in test_interpreter_shutdown

TST add test_ressources_gced_in_workers

7a093f0

- Ensure that the job argument passed are freed asap

DOC update ProcessPoolExecutor doc

3953ee3

bedevere-bot added awaiting core review and removed awaiting review labels Oct 2, 2017

pitrou reviewed Oct 3, 2017

View reviewed changes

tomMoral and others added 3 commits October 3, 2017 09:48

FIX doc context + use assertTrue

9ac6305

Doc nit: simplify sentence

6e4104b

NEWS nits

8ec5ab4

pitrou merged commit e8c368d into python:master Oct 3, 2017

tomMoral deleted the PR_contextual_executor branch October 3, 2017 10:23

Mariatta removed the awaiting core review label Oct 4, 2017

tomMoral mentioned this pull request Apr 9, 2018

Port robust ProcessPoolExecutor to concurrent.futures joblib/loky#48

Closed

11 tasks

Uh oh!

bpo-31540 Add context management for concurrent.futures.ProcessPoolExecutor #3682

bpo-31540 Add context management for concurrent.futures.ProcessPoolExecutor #3682

Uh oh!

Conversation

tomMoral commented Sep 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou commented Oct 2, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomMoral Oct 3, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou commented Oct 3, 2017

Uh oh!

tomMoral commented Oct 3, 2017

Uh oh!

Uh oh!

tomMoral commented Sep 21, 2017 •

edited

Loading

tomMoral Oct 3, 2017 •

edited

Loading