"RuntimeError: This event loop is already running" When attempting REPL example

[JacobCallahan]

Each time I attempt the REPl example, I get the traceback below. Sync context-manager example worked though.
playwright==0.8.0
REPL Example: https://github.com/microsoft/playwright-python#repl-support-without-context-managers

IPython session

 jake  (e) venvplaywright  ~  ipython
/home/jake/.local/lib/python3.8/site-packages/IPython/core/interactiveshell.py:935: UserWarning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
  warn("Attempting to work in a virtualenv. If you encounter problems, please "
Python 3.8.3 (default, May 29 2020, 00:00:00) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from playwright import sync_playwright                                                                                                                                                

In [2]: playwright = sync_playwright().start()                                                                                                                                                

Traceback (most recent call last):
  File "/usr/local/bin/ipython", line 11, in <module>
    sys.exit(start_ipython())
  File "/home/jake/.local/lib/python3.8/site-packages/IPython/__init__.py", line 126, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "/home/jake/.local/lib/python3.8/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/home/jake/.local/lib/python3.8/site-packages/IPython/terminal/ipapp.py", line 356, in start
    self.shell.mainloop()
  File "/home/jake/.local/lib/python3.8/site-packages/IPython/terminal/interactiveshell.py", line 558, in mainloop
    self.interact()
  File "/home/jake/.local/lib/python3.8/site-packages/IPython/terminal/interactiveshell.py", line 541, in interact
    code = self.prompt_for_code()
  File "/home/jake/.local/lib/python3.8/site-packages/IPython/terminal/interactiveshell.py", line 467, in prompt_for_code
    text = self.pt_app.prompt(
  File "/home/jake/.local/lib/python3.8/site-packages/prompt_toolkit/shortcuts/prompt.py", line 994, in prompt
    return self.app.run(set_exception_handler=set_exception_handler)
  File "/home/jake/.local/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 811, in run
    return loop.run_until_complete(
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 592, in run_until_complete
    self._check_running()
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 552, in _check_running
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running

If you suspect this is an IPython 7.13.0 bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@python.org

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

sys:1: RuntimeWarning: coroutine 'Application.run_async' was never awaited

[mxschmitt]

for iPython / Jupiter notebooks it seems like the current workaround would be to use that before you run your shell:

import nest_asyncio
nest_asyncio.apply()

Out of curiosity to understand more the user case. You are trying out Playwright in a REPL or are you using it only with with the REPL in the end?

[JacobCallahan]

@mxschmitt thanks for the quick response!
nest_asyncio isn't a standard package or one installed as a requirement for playwright. As such, I'm guessing most users wouldn't have it installed. However, that did solve the problem:

In [1]: import nest_asyncio                                                                                                                                   

In [2]: nest_asyncio.apply()                                                                                                                                  

In [3]: from playwright import sync_playwright                                                                                                                

In [4]: playwright = sync_playwright().start()                                                                                                                

In [5]:     

My use-case within IPython is largely API exploration and interactive process development before writing a script or integration with an application.

Additional note, your example works perfectly in the base python REPL.

[bubthegreat]

@mxschmitt - FYI, the nested async doesn't work for jupyter notebooks for at least the sync REPL example:

import nest_asyncio
nest_asyncio.apply()

from playwright import sync_playwright
playwright = sync_playwright().start()
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-3-85d403adca24> in <module>
----> 1 playwright = sync_playwright().start()

c:\users\bubth\development\playwright\venv\lib\site-packages\playwright\__init__.py in sync_playwright()
     32 
     33 def sync_playwright() -> SyncPlaywrightContextManager:
---> 34     return SyncPlaywrightContextManager()
     35 
     36 

c:\users\bubth\development\playwright\venv\lib\site-packages\playwright\main.py in __init__(self)
     81 class SyncPlaywrightContextManager:
     82     def __init__(self) -> None:
---> 83         self._connection = run_driver()
     84         self._playwright: SyncPlaywright
     85 

c:\users\bubth\development\playwright\venv\lib\site-packages\playwright\main.py in run_driver()
     76     # if loop.is_running():
     77     #     raise Error("Can only run one Playwright at a time.")
---> 78     return loop.run_until_complete(run_driver_async())
     79 
     80 

c:\users\bubth\development\playwright\venv\lib\site-packages\nest_asyncio.py in run_until_complete(self, future)
     93                 raise RuntimeError(
     94                     'Event loop stopped before Future completed.')
---> 95             return f.result()
     96         finally:
     97             events._set_running_loop(old_running_loop)

C:\Program Files\Python37\lib\asyncio\futures.py in result(self)
    176         self.__log_traceback = False
    177         if self._exception is not None:
--> 178             raise self._exception
    179         return self._result
    180 

C:\Program Files\Python37\lib\asyncio\tasks.py in __step(***failed resolving arguments***)
    221                 # We use the `send` method directly, because coroutines
    222                 # don't have `__iter__` and `__next__` methods.
--> 223                 result = coro.send(None)
    224             else:
    225                 result = coro.throw(exc)

c:\users\bubth\development\playwright\venv\lib\site-packages\playwright\main.py in run_driver_async()
     62         stdout=asyncio.subprocess.PIPE,
     63         stderr=_get_stderr_fileno(),
---> 64         limit=32768,
     65     )
     66     assert proc.stdout

C:\Program Files\Python37\lib\asyncio\subprocess.py in create_subprocess_exec(program, stdin, stdout, stderr, loop, limit, *args, **kwds)
    215         program, *args,
    216         stdin=stdin, stdout=stdout,
--> 217         stderr=stderr, **kwds)
    218     return Process(transport, protocol, loop)

C:\Program Files\Python37\lib\asyncio\base_events.py in subprocess_exec(self, protocol_factory, program, stdin, stdout, stderr, universal_newlines, shell, bufsize, *args, **kwargs)
   1531         transport = await self._make_subprocess_transport(
   1532             protocol, popen_args, False, stdin, stdout, stderr,
-> 1533             bufsize, **kwargs)
   1534         if self._debug and debug_log is not None:
   1535             logger.info('%s: %r', debug_log, transport)

C:\Program Files\Python37\lib\asyncio\base_events.py in _make_subprocess_transport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, extra, **kwargs)
    461                                          extra=None, **kwargs):
    462         """Create subprocess transport."""
--> 463         raise NotImplementedError
    464 
    465     def _write_to_self(self):

NotImplementedError: 

[pavelfeldman]

@bubthegreat: This looks like a SelectorEventLoop's limitation on Python 3.7 on Windows. Starting with 3.8, a more complete implementation of the loop is used on Windows and it should support running subprocesses. See the platform support section for more details.

We do install this more capable ProactorEventLoop manually on 3.7 see here, but it might be that nest_asyncio does not respect our selection. So it is a combination of Windows + Python 3.7 + nest_asyncio that triggers it.

[bubthegreat]

I assume upgrading to 3.9 would improve this, still saw issues, and downgraded to 3.8.3 - still having issues with sync_playwright:

import nest_asyncio
nest_asyncio.apply()
from playwright import sync_playwright                                                                                                                
playwright = sync_playwright().start()    
---------------------------------------------------------------------------
Error                                     Traceback (most recent call last)
<ipython-input-2-dfa883f2869c> in <module>
      1 from playwright import sync_playwright
----> 2 playwright = sync_playwright().start()

c:\users\bubth\development\playwright-test\venv\lib\site-packages\playwright\__init__.py in sync_playwright()
     32 
     33 def sync_playwright() -> SyncPlaywrightContextManager:
---> 34     return SyncPlaywrightContextManager()
     35 
     36 

c:\users\bubth\development\playwright-test\venv\lib\site-packages\playwright\main.py in __init__(self)
     86 class SyncPlaywrightContextManager:
     87     def __init__(self) -> None:
---> 88         self._connection = run_driver()
     89         self._playwright: SyncPlaywright
     90 

c:\users\bubth\development\playwright-test\venv\lib\site-packages\playwright\main.py in run_driver()
     80     loop = asyncio.get_event_loop()
     81     if loop.is_running():
---> 82         raise Error("Can only run one Playwright at a time.")
     83     return loop.run_until_complete(run_driver_async())
     84 

Error: Can only run one Playwright at a time.

[ddelange]

Currently, playwright disallows attaching to a running loop:

playwright-python/playwright/main.py

Lines 79 to 83 in d98d38b

	def run_driver() -> Connection:
	loop = asyncio.get_event_loop()
	if loop.is_running():
	raise Error("Can only run one Playwright at a time.")
	return loop.run_until_complete(run_driver_async())

This should be fine in plain IPython (does not launch an asyncio event loop in the main thread), but will fail in Jupyter that uses tornado>=5 (launches asyncio event loop on startup under the hood).

As calling run_until_complete is disallowed on running loops, this will fail in Jupyter.

Minimal reproducible example that will run in latest IPython, but will break on Jupyter environments that use tornado>=5 under the hood:

import asyncio
print(asyncio.run(asyncio.sleep(0)))

So for Jupyter, additionally a pip install 'tornado<5' 'notebook<5.7.9' should fix loop issues introduced in tornado>=5 (and if you need jupyterhub, use jupyterhub<0.9.0). refs jupyter/notebook#3397 (comment) ipython/ipykernel#548

[pavelfeldman]

My understanding is that resolving this requires cooperation between the embedder that controls the loop and sync Playwright that also wants to dispatch its messages forever. And I don't think such cooperation is the right way to go. Sync API is designed as self-contained, assumes it controls execution. Is there a way to use async in Jupyter?

[bubthegreat]

If I get time tonight ill play around with it- I dont see any reason for it not to run in the existing loop - or are there architectural challenges with it sharing the loop?

…

On Tue, Oct 6, 2020, 8:35 AM Pavel Feldman ***@***.***> wrote: My understanding is that resolving this requires cooperation between the embedder that controls the loop and sync Playwright that also wants to dispatch its messages forever. And I don't think such cooperation is the right way to go. Sync API is designed as self-contained, assumes it controls execution. Is there a way to use async in Jupyter? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#178 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC5CHIMV52GR76C7R7XFYKTSJMTMLANCNFSM4QLBAFHQ> .

[ddelange]

Executing a cell in Jupyter will actually require a lock on the main loop from ipykernel, so you could schedule execution of a Future in one cell (which will start execution once that cell has finished), and get the result in the next cell. Doing it all in one cell is certainly not trivial without nested loops.