Not all generators containing `await` are compiled to async generators

[mniip]

Documentation

A generator of the form f(x) for x in await a first awaits a and then creates a regular sync generator, as can be seen in the following minimal reproducer:

import asyncio
async def foo():
    print("Entered foo")
    return [1, 2, 3]
async def main():
    gen = (x for x in await foo())
    print(gen)
    print(list(gen))
asyncio.run(main())

Entered foo
<generator object main.<locals>.<genexpr> at 0x7f8fce7f8110>
[1, 2, 3]

However the python language reference 6.2.8. Generator expressions states:

If a generator expression contains either async for clauses or await expressions it is called an asynchronous generator expression. An asynchronous generator expression returns a new asynchronous generator object, which is an asynchronous iterator (see Asynchronous Iterators).

This seems to suggest that the generator f(x) for x in await a does contain an await expression, so it should become an async_generator? However there is also:

However, the iterable expression in the leftmost for clause is immediately evaluated, so that an error produced by it will be emitted at the point where the generator expression is defined, rather than at the point where the first value is retrieved.

This seems to hint at an implementation detail that the leftmost for clause has special treatment and is always evaluated before the generator is constructed, but the interactions with await are unclear from this, especially whether this disqualifies the generator from becoming async.

I don't know whether this is an undocumented feature or a bug.

Linked PRs

• gh-114104: clarify asynchronous comprehension docs to match runtime behavior #121175
• [3.13] gh-114104: clarify asynchronous comprehension docs to match runtime behavior (GH-121175) #121234
• [3.12] gh-114104: clarify asynchronous comprehension docs to match runtime behavior (GH-121175) #121235

[serhiy-storchaka]

cc @gvanrossum

I think that it is easier to make the documentation more explicit. The iterable expression in the leftmost for clause is not included in a generator expression. It is evaluated (and implicit iter() is called) before entering a generator function, so it is not a part of the generator function and does not make it asynchronous.

We can, of cause, change the implementation, but what is the use case for this? If you want to make an asynchronous generator and await foo() in it, you can use the following workaround:

gen = (x for _ in [1] for x in await foo())

But it is more clear to write an inner function:

async def gen():
    for x in await foo():
        yield x

In this way you can even make an asynchronous generator that does not contain any await.

async def gen():
    for x in sync_foo():
        yield x

[gvanrossum]

The behavior is as intended. If that's not clear from the docs, those doc should be updated.