-
-
Notifications
You must be signed in to change notification settings - Fork 32k
GH-91048: Add utils for capturing async call stack for asyncio programs and enable profiling #124640
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+2,915
−233
Merged
GH-91048: Add utils for capturing async call stack for asyncio programs and enable profiling #124640
Changes from all commits
Commits
Show all changes
88 commits
Select commit
Hold shift + click to select a range
1b01a91
Add `f_generator` property to Python frame objects
1st1 0fc5511
Working implementation of `asyncio.capture_call_stack()`
1st1 1d20a51
Address Guido's comments
1st1 c8be18e
Add a comment for capture_call_stack()
1st1 abf2cb9
Add a couple more tests
1st1 20ceab7
Remove setter for C impl of Task._awaited_by
1st1 72d9321
Intoduce cr_task
1st1 c9475f6
Unbreak shield() and gather()
1st1 e1099e9
Add convinience fields to C Task/Future for profilers
1st1 817f88b
Fix ups
1st1 54386ac
Add basic docs
1st1 98434f0
Implement, test, and document asyncio.print_call_stack()
1st1 485c166
Reorder a few things
1st1 8802be7
Add a test to exercise asyncio stack traces in out-of-process profilers
pablogsal 1ddc9cf
Refactor the section-generated code into an evil macro
pablogsal bc9beb8
Rename "capture_call_stack()" et al to "capture_call_graph()"
1st1 fd141d4
Add NEWS
1st1 2d72f24
Per mpage's suggestion: use traceback style formatring for frames
1st1 391defa
mpage feedback: fix typo
1st1 d6357fd
Per mpage suggestion: get rid of CoroutineCallGraphEntry
1st1 bb3b6df
Strip sloppy white space!
1st1 54c99ec
Fix sloppy set iteration!
1st1 c1a4f09
Fix a sloppy test!
1st1 027d522
Run 'make regen-all'
1st1 c2d5ec6
Add a what's new entry
1st1 08d09eb
Fix (hopefully) a compiler warning
1st1 fe3113b
Fix sloppy what's new!
1st1 18ec26d
Fix CI complaining about suspicious global in C
1st1 e4cc462
Add a test for depth=2
1st1 d5cdc36
Add critical sections for free threaded builds
pablogsal 83606f2
Add more slopy tests
pablogsal 5edac41
Apply suggestions from code review
1st1 8dc6d34
Apply suggestions from code review
1st1 30884ea
Update Modules/_asynciomodule.c
1st1 1317658
Update Modules/_asynciomodule.c
1st1 81b0a31
Use dataclasses instead of tuples for asyncio.stack
1st1 258ce3d
Fix sloppy whitespace!
1st1 b9ecefb
Remove critical sections for now
pablogsal b77dcb0
Eplain the weird macro
1st1 8867946
Fix test
pablogsal 87d2524
Merge remote-tracking branch 'upstream/main' into stack
pablogsal b47bef1
Add a docs clarification
1st1 230b7ec
Fix typing in asyncio.stack
ambv b1d6158
Fix memory leak
pablogsal ac51364
Address comments from Kumar's review
ambv c7e59eb
Fix rare Mach-O linker bug when .bss is uninitialized data
ambv 9eba5e1
Merge branch 'main' into pr-124640
ambv 59121f6
you saw nothing, okay
ambv f8f48f0
Update Lib/asyncio/futures.py
1st1 74c5ad1
Remove unnecessary imports from tests
ambv 067c043
Remove cr_task/gi_task
ambv 9f04911
Fix refleaks
ambv 0774805
Rename "asyncio.stack" to "asyncio.graph"
ambv 3048493
Allow returning a string with `format_call_graph`
ambv 1f42873
Add test for eager task factory support
ambv 7799391
Merge branch 'main' into pr-124640
ambv 03ed5c1
Make _asyncio_awaited_by a frozenset in the Python version as well pe…
ambv 21f9ea9
Address picnixz' feedback
1st1 8a43dfa
Blacken the C/Py test code by hand, by request from picnixz
1st1 b3fae68
More style fixes per picnixz' suggestions
1st1 d0aedf0
Address Kumar's latest comment
1st1 df0032a
diff cleanup
kumaraditya303 0ce241b
change dataclasses to use tuples
kumaraditya303 8f126f6
doc fixes
kumaraditya303 966d84e
remove reduntant headers include and add my name to whatsnew
kumaraditya303 f56468a
improve comment
kumaraditya303 404b88a
fix leave task
kumaraditya303 911fed8
fix external inspection on linux
kumaraditya303 ab511a4
minor format
kumaraditya303 c3c685a
try to fix docs build
kumaraditya303 785adeb
Update Doc/whatsnew/3.14.rst
kumaraditya303 a577328
Match indentation with _asynciomodule.c after ab511a4f67ffd88f7709c1b…
ambv 064129a
Improve comments after f56468af8b3789d0e20be30715c20ac1bb367641
ambv ce332d9
Restore the Oxford comma
ambv d6d943f
Address remaining suggestions of Andrew Svetlov
ambv 703ff46
Fix gather()
1st1 e867863
Replace lambda by closure
pablogsal 9cb5b29
Let’s not emphasize *asyncio*
1st1 61b2b7b
Apply suggestions from Irit's review
ambv 9533ab9
Address remaining review by Irit
ambv 596191d
Test pure-Python and C-accelerated versions of future_add_to/future_d…
ambv ad9152e
fix blooper
ambv 066bf21
Fix another blooper
ambv 4caeec4
Fix crash
pablogsal 38f061d
Merge remote-tracking branch 'upstream/main' into stack
pablogsal a8dd667
use private method for the policy
pablogsal cf8f5e5
Avoid importing typing
ambv eda9c7c
Remove debug printing from test_asyncio.test_graph
ambv File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,145 @@ | ||
.. currentmodule:: asyncio | ||
|
||
|
||
.. _asyncio-graph: | ||
|
||
======================== | ||
Call Graph Introspection | ||
======================== | ||
|
||
**Source code:** :source:`Lib/asyncio/graph.py` | ||
|
||
------------------------------------- | ||
|
||
asyncio has powerful runtime call graph introspection utilities | ||
to trace the entire call graph of a running *coroutine* or *task*, or | ||
a suspended *future*. These utilities and the underlying machinery | ||
can be used from within a Python program or by external profilers | ||
and debuggers. | ||
|
||
.. versionadded:: next | ||
|
||
|
||
.. function:: print_call_graph(future=None, /, *, file=None, depth=1, limit=None) | ||
|
||
Print the async call graph for the current task or the provided | ||
:class:`Task` or :class:`Future`. | ||
|
||
This function prints entries starting from the top frame and going | ||
ambv marked this conversation as resolved.
Show resolved
Hide resolved
|
||
down towards the invocation point. | ||
|
||
The function receives an optional *future* argument. | ||
If not passed, the current running task will be used. | ||
ambv marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
If the function is called on *the current task*, the optional | ||
keyword-only *depth* argument can be used to skip the specified | ||
number of frames from top of the stack. | ||
|
||
If the optional keyword-only *limit* argument is provided, each call stack | ||
in the resulting graph is truncated to include at most ``abs(limit)`` | ||
entries. If *limit* is positive, the entries left are the closest to | ||
the invocation point. If *limit* is negative, the topmost entries are | ||
left. If *limit* is omitted or ``None``, all entries are present. | ||
If *limit* is ``0``, the call stack is not printed at all, only | ||
"awaited by" information is printed. | ||
|
||
If *file* is omitted or ``None``, the function will print | ||
to :data:`sys.stdout`. | ||
|
||
**Example:** | ||
|
||
The following Python code: | ||
|
||
.. code-block:: python | ||
|
||
import asyncio | ||
|
||
async def test(): | ||
asyncio.print_call_graph() | ||
|
||
async def main(): | ||
async with asyncio.TaskGroup() as g: | ||
g.create_task(test()) | ||
|
||
asyncio.run(main()) | ||
|
||
will print:: | ||
|
||
* Task(name='Task-2', id=0x1039f0fe0) | ||
+ Call stack: | ||
| File 't2.py', line 4, in async test() | ||
+ Awaited by: | ||
* Task(name='Task-1', id=0x103a5e060) | ||
+ Call stack: | ||
| File 'taskgroups.py', line 107, in async TaskGroup.__aexit__() | ||
| File 't2.py', line 7, in async main() | ||
|
||
.. function:: format_call_graph(future=None, /, *, depth=1, limit=None) | ||
|
||
Like :func:`print_call_graph`, but returns a string. | ||
If *future* is ``None`` and there's no current task, | ||
the function returns an empty string. | ||
|
||
|
||
.. function:: capture_call_graph(future=None, /, *, depth=1, limit=None) | ||
|
||
Capture the async call graph for the current task or the provided | ||
:class:`Task` or :class:`Future`. | ||
|
||
The function receives an optional *future* argument. | ||
If not passed, the current running task will be used. If there's no | ||
current task, the function returns ``None``. | ||
|
||
If the function is called on *the current task*, the optional | ||
keyword-only *depth* argument can be used to skip the specified | ||
number of frames from top of the stack. | ||
|
||
Returns a ``FutureCallGraph`` data class object: | ||
|
||
* ``FutureCallGraph(future, call_stack, awaited_by)`` | ||
|
||
Where *future* is a reference to a :class:`Future` or | ||
a :class:`Task` (or their subclasses.) | ||
|
||
``call_stack`` is a tuple of ``FrameCallGraphEntry`` objects. | ||
|
||
``awaited_by`` is a tuple of ``FutureCallGraph`` objects. | ||
|
||
* ``FrameCallGraphEntry(frame)`` | ||
|
||
Where *frame* is a frame object of a regular Python function | ||
in the call stack. | ||
|
||
|
||
Low level utility functions | ||
=========================== | ||
|
||
To introspect an async call graph asyncio requires cooperation from | ||
control flow structures, such as :func:`shield` or :class:`TaskGroup`. | ||
Any time an intermediate :class:`Future` object with low-level APIs like | ||
:meth:`Future.add_done_callback() <asyncio.Future.add_done_callback>` is | ||
involved, the following two functions should be used to inform asyncio | ||
about how exactly such intermediate future objects are connected with | ||
the tasks they wrap or control. | ||
|
||
|
||
.. function:: future_add_to_awaited_by(future, waiter, /) | ||
|
||
Record that *future* is awaited on by *waiter*. | ||
|
||
Both *future* and *waiter* must be instances of | ||
:class:`Future` or :class:`Task` or their subclasses, | ||
otherwise the call would have no effect. | ||
|
||
A call to ``future_add_to_awaited_by()`` must be followed by an | ||
eventual call to the :func:`future_discard_from_awaited_by` function | ||
with the same arguments. | ||
|
||
|
||
.. function:: future_discard_from_awaited_by(future, waiter, /) | ||
|
||
Record that *future* is no longer awaited on by *waiter*. | ||
|
||
Both *future* and *waiter* must be instances of | ||
:class:`Future` or :class:`Task` or their subclasses, otherwise | ||
the call would have no effect. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.