Finalization of non-exhausted asynchronous generators is deferred

[serhiy-storchaka]

BPO	44518
Nosy	@gvanrossum, @asvetlov, @markshannon, @serhiy-storchaka, @1st1, @gvanrossum

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-06-27.09:11:32.621>
labels = ['interpreter-core', '3.11', '3.9', '3.10', 'performance']
title = 'Finalization of non-exhausted asynchronous generators is deferred'
updated_at = <Date 2021-06-28.18:33:50.454>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-06-28.18:33:50.454>
actor = 'gvanrossum'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2021-06-27.09:11:32.621>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 44518
keywords = []
message_count = 6.0
messages = ['396569', '396610', '396611', '396644', '396650', '396659']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'asvetlov', 'Mark.Shannon', 'serhiy.storchaka', 'yselivanov', 'Guido.van.Rossum']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue44518'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[serhiy-storchaka]

In the following example:

def gen():
    try:
        yield 1
    finally:
        print('finalize inner')

def func():
    try:
        for x in gen():
            break
    finally:
        print('finalize outer')

func()
print('END')

the output in CPython is:

finalize inner
finalize outer
END

But in similar example for asynchronous generator:

async def gen():
    try:
        yield 1
    finally:
        print('finalize inner')

async def func():
    try:
        async for x in gen():
            break
    finally:
        print('finalize outer')

import asyncio
asyncio.run(func())
print('END')

the output in CPython is:

finalize outer
finalize inner
END

There is a strong link somewhere which prevents finalization of the asynchronous generator object until leaving the outer function.

Tested on CPython 3.7-3.11. In PyPy "finalize inner" is not printed at all. Using closing() and aclosing() is the right way to get deterministic finalization, but in any case it would be better to get rid of strong link which prevents finalization of the asynchronous generator object.

[gvanrossum]

Can you repro this without asyncio?

[serhiy-storchaka]

Sorry, I do not understand what do you mean. The problem is that non-exhausted asynchronous generators are not finalized until asyncio.run() is finished. This differs from synchronous generators which are finalized as early as possible (in CPython).

[gvanrossum]

I am wondering whether the issue is in asyncio.run, or in the basic async generator implementation. If the latter, you should be able to demonstrate this with a manual "driver" (trampoline?) instead of asyncio.run.

[serhiy-storchaka]

Inlined asyncio.run(func()) is roughly equivalent to the following code:

loop = asyncio.new_event_loop()
loop.run_until_complete(func())
loop.run_until_complete(loop.shutdown_asyncgens())
loop.close()

If comment out loop.run_until_complete(loop.shutdown_asyncgens()) I get the following output:

finalize outer
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<<async_generator_athrow without __name__>()>>
END

No "finalize inner" but a warning about pending task instead.

On other hand, the following code

for _ in func().__await__(): pass

produces the output in expected order:

finalize inner
finalize outer
END

[gvanrossum]

Okay, that suggests that the extra reference is being held by asyncio, right? I suppose it's in the hands of the asyncio devs then.

[kumaraditya303]

The extra reference is required so that the event loop can finalize the generator in the correct context. The issue is that once the generator is about to be finalized, the event loop creates a new task to finalize it in the context but since there are no awaits after iterating over the generator the task is pending.

The following script finalizes it properly as it add a sleep to switch the tasks and run the finalizer.

async def gen():
    try:
        yield 1
    finally:
        print('finalize inner')

async def func():
    try:
        async for x in gen():
            break
        await asyncio.sleep(0.1) # Switch task and run the finalizer 
    finally:
        print('finalize outer')

import asyncio
asyncio.run(func())
print('END')

Output:

finalize inner
finalize outer
END

[kumaraditya303]

The following example demonstrates the number of context switches required to run the finalizer:

async def gen():
    try:
        yield 1
    finally:
        print('finalize inner')

async def func():
    try:
        async for x in gen():
            break
        # Schedules callback to create a task on next loop iteration 
    finally:
        await asyncio.sleep(0) # Creates a task to finalize gen
        await asyncio.sleep(0) # Run the task to finalize gen
        print('finalize outer')

import asyncio
asyncio.run(func())
print('END')

Output:

finalize inner
finalize outer
END

[gvanrossum]

Hopefully @serhiy-storchaka can do something with your research!

[kumaraditya303]

This would probably require something like https://peps.python.org/pep-0533/

[gvanrossum]

Could you implement a prototype? That might help the PEP move forward. @njs @1st1

[graingert]

Can you repro this without asyncio?

async def gen():
    try:
        yield 1
    finally:
        print('finalize inner')

async def func():
    try:
        async for x in gen():
            break
    finally:
        print('finalize outer')

import anyio
anyio.run(func, backend="trio")
print('END')

this happens with trio too:

python -W error demo.py
Exception ignored in: <async_generator object gen at 0x7fd98b9db040>
Traceback (most recent call last):
  File "/home/graingert/.virtualenvs/testing39/lib/python3.9/site-packages/trio/_core/_asyncgens.py", line 79, in finalizer
    warnings.warn(
ResourceWarning: Async generator '__main__.gen' was garbage collected before it had been exhausted. Surround its use in 'async with aclosing(...):' to ensure that it gets cleaned up as soon as you're done using it.
finalize outer
finalize inner
END

[graingert]

running it with plain coroutines results in:

import threading

async def gen():
    try:
        yield 1
    finally:
        print('finalize inner')

async def func():
    try:
        async for x in gen():
            break
    finally:
        print('finalize outer')

import threading

coro = func()
coro.send(None)

finalize inner
finalize outer
Traceback (most recent call last):
  File "/home/graingert/projects/cpython/demo.py", line 19, in <module>
    coro.send(None)
StopIteration

[gvanrossum]

That's a rather literalist interpretation of my question. :-) I meant whether you can come up with a repro without an event loop framework, since that would presumably make reasoning about the root cause of the problem easier.

[graingert]

trio documents a one-tick finalization window here: https://github.com/python-trio/trio/blob/35319b3d4b881032416bfd733e0ac5239ce4c46d/trio/_core/_asyncgens.py#L26-L30 and here https://trio.readthedocs.io/en/stable/reference-core.html#finalization

[gvanrossum]

Ah, sorry. The whole point of the bug seems to be that the event loop hangs on to an extra reference, and there is a reason.

I should really recuse myself from this issue since I don't understand async generators.

[graingert]

In PyPy "finalize inner" is not printed at all.

This looks like it could be an asyncio bug - asyncio should use the firstiter hook to keep a reference to generators that need shutting down

[graingert]

In PyPy "finalize inner" is not printed at all.

seems that got fixed:

import threading
import trio
import asyncio

async def gen(runner):
    try:
        yield 1
    finally:
        print(f'{runner=} finalize inner')

async def func(runner):
    try:
        async for x in gen(runner):
            break
    finally:
        print(f'{runner=} finalize outer')

print(f"{trio.run(func, 'trio')=}")
print(f"{asyncio.run(func('asyncio'))=}")
print(f"{func('manual').send(None)=}")  # on pypy3 this doesn't print finalize inner

prints:

$ pypy3.8 demo.py
runner='trio' finalize outer
runner='trio' finalize inner
trio.run(func, 'trio')=None
runner='asyncio' finalize outer
runner='asyncio' finalize inner
asyncio.run(func('asyncio'))=None
runner='manual' finalize outer
Traceback (most recent call last):
  File "demo.py", line 20, in <module>
    print(f"{func('manual').send(None)=}")
StopIteration

[kumaraditya303]

The behaviour cannot be changed without significant changes to the interpreter. The documentation will be handled for this as part of #100108 so closing.

[kumaraditya303]

For anyone who comes here for a possible fix which always works:

import contextlib

async def gen():
    try:
        yield 1
    finally:
        print('finalize inner')

async def func():
    try:
        async with contextlib.aclosing(gen()) as g:
            async for x in g:
                break
    finally:
        print('finalize outer')

import asyncio
asyncio.run(func())
print('END')