Skip to content

gh-96143: Improve perf profiler docs #96445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Oct 27, 2022
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 31 additions & 25 deletions Doc/howto/perf_profiling.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,11 @@ Python support for the Linux ``perf`` profiler

:author: Pablo Galindo

The Linux ``perf`` profiler is a very powerful tool that allows you to profile and
obtain information about the performance of your application. ``perf`` also has
a very vibrant ecosystem of tools that aid with the analysis of the data that it
produces.
`The Linux perf profiler <http://perf.wiki.kernel.org>`_
is a very powerful tool that allows you to profile and obtain
information about the performance of your application.
``perf`` also has a very vibrant ecosystem of tools
that aid with the analysis of the data that it produces.

The main problem with using the ``perf`` profiler with Python applications is that
``perf`` only allows to get information about native symbols, this is, the names of
Expand All @@ -25,7 +26,7 @@ fly before the execution of every Python function and it will teach ``perf`` the
relationship between this piece of code and the associated Python function using
`perf map files`_.

.. warning::
.. note::

Support for the ``perf`` profiler is only currently available for Linux on
selected architectures. Check the output of the configure build step or
Expand All @@ -51,11 +52,11 @@ For example, consider the following script:
if __name__ == "__main__":
baz(1000000)

We can run perf to sample CPU stack traces at 9999 Hertz:
We can run ``perf`` to sample CPU stack traces at 9999 Hertz::

$ perf record -F 9999 -g -o perf.data python my_script.py

Then we can use perf report to analyze the data:
Then we can use ``perf`` report to analyze the data:

.. code-block:: shell-session

Expand Down Expand Up @@ -101,7 +102,7 @@ As you can see here, the Python functions are not shown in the output, only ``_P
functions use the same C function to evaluate bytecode so we cannot know which Python function corresponds to which
bytecode-evaluating function.

Instead, if we run the same experiment with perf support activated we get:
Instead, if we run the same experiment with ``perf`` support activated we get:

.. code-block:: shell-session

Expand Down Expand Up @@ -147,19 +148,29 @@ Instead, if we run the same experiment with perf support activated we get:



Enabling perf profiling mode
----------------------------
How to enable the ``perf`` profiling mode
-----------------------------------------

There are two main ways to activate the perf profiling mode. If you want it to be
active since the start of the Python interpreter, you can use the `-Xperf` option:
There are three ways to activate the ``perf`` profiling mode:

* using the :option:`-Xperf <-X>` command-line option
* using the :envvar:`PYTHONPERFSUPPORT` environment variable
* using the :func:`sys.activate_stack_trampoline` and
:func:`sys.deactivate_stack_trampoline` APIs

If you want profiling to be active when you start the Python interpreter,
use the :option:`-Xperf <-X>` option::

$ python -Xperf my_script.py

You can also set the :envvar:`PYTHONPERFSUPPORT` to a nonzero value to actiavate perf
profiling mode globally.
If you need to activate the ``perf`` profiling mode globally,
set the environment variable :envvar:`PYTHONPERFSUPPORT`
to a nonzero value.

There is also support for dynamically activating and deactivating the perf
profiling mode by using the APIs in the :mod:`sys` module:
If you need to dynamically activate and deactivate the ``perf`` profiling mode
in response to a signal or other communication mechanisms with your process,
use the :func:`sys.activate_stack_trampoline` and
:func:`sys.deactivate_stack_trampoline` APIs:

.. code-block:: python

Expand All @@ -172,27 +183,22 @@ profiling mode by using the APIs in the :mod:`sys` module:

# Perf profiling is not active anymore

These APIs can be handy if you want to activate/deactivate profiling mode in
response to a signal or other communication mechanism with your process.



Now we can analyze the data with ``perf report``:
Now we can analyze the data with ``perf report``::

$ perf report -g -i perf.data


How to obtain the best results
-------------------------------
------------------------------

For the best results, Python should be compiled with
``CFLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"`` as this allows
profilers to unwind using only the frame pointer and not on DWARF debug
information. This is because as the code that is interposed to allow perf
information. This is because as the code that is interposed to allow ``perf``
support is dynamically generated it doesn't have any DWARF debugging information
available.

You can check if you system has been compiled with this flag by running:
You can check if your system has been compiled with this flag by running::

$ python -m sysconfig | grep 'no-omit-frame-pointer'

Expand Down
30 changes: 30 additions & 0 deletions Doc/library/sys.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1524,6 +1524,36 @@ always available.
This function has been added on a provisional basis (see :pep:`411`
for details.) Use it only for debugging purposes.

.. function:: activate_stack_trampoline(backend, /)

Activate the stack profiler trampoline *backend*.
The only supported backend is `"perf"`.

.. availability:: Linux.

.. versionadded:: 3.12

.. seealso::

* :ref:`perf_profiling`
* https://perf.wiki.kernel.org

.. function:: deactivate_stack_trampoline()

Deactivate the stack profiler trampoline.

.. availability:: Linux.

.. versionadded:: 3.12

.. function:: is_stack_trampoline_active()

Return ``True`` if the stack profiler trampoline is active.

.. availability:: Linux.

.. versionadded:: 3.12

.. function:: _enablelegacywindowsfsencoding()

Changes the :term:`filesystem encoding and error handler` to 'mbcs' and
Expand Down
16 changes: 11 additions & 5 deletions Doc/using/cmdline.rst
Original file line number Diff line number Diff line change
Expand Up @@ -535,12 +535,13 @@ Miscellaneous options
development (running from the source tree) then the default is "off".
Note that the "importlib_bootstrap" and "importlib_bootstrap_external"
frozen modules are always used, even if this flag is set to "off".
* ``-X perf`` to activate compatibility mode with the ``perf`` profiler.
* ``-X perf`` activates compatibility mode with the ``perf`` profiler.
When this option is activated, the Linux ``perf`` profiler will be able to
report Python calls. This option is only available on some platforms and
will do nothing if is not supported on the current system. The default value
is "off". See also :envvar:`PYTHONPERFSUPPORT` and :ref:`perf_profiling`
for more information.
is "off". See :ref:`perf_profiling` for more details.
This option is equivalent to setting the environment variable
:envvar:`PYTHONPERFSUPPORT` to ``1``.

It also allows passing arbitrary values and retrieving them through the
:data:`sys._xoptions` dictionary.
Expand Down Expand Up @@ -1036,8 +1037,13 @@ conflict.
.. envvar:: PYTHONPERFSUPPORT

If this variable is set to a nonzero value, it activates compatibility mode
with the ``perf`` profiler so Python calls can be detected by it. See the
:ref:`perf_profiling` section for more information.
with the ``perf`` profiler so Python calls can be detected by it.
This is exactly equivalent to setting :option:`-X perf <-X>`
on the command line.

If set to ``0``, disable ``perf`` profiler support.

See :ref:`perf_profiling` for more details.

.. versionadded:: 3.12

Expand Down
21 changes: 21 additions & 0 deletions Doc/whatsnew/3.12.rst
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,14 @@ Important deprecations, removals or restrictions:
New Features
============

* Add :ref:`perf_profiling` through the new command-line option
:option:`-X perf <-X>`, as well as the new
:func:`sys.activate_stack_trampoline`,
:func:`sys.deactivate_stack_trampoline`,
and :func:`sys.is_stack_trampoline_active` APIs.
(Design by Pablo Galindo. Contributed by Pablo Galindo and Christian Heimes
with contributions from Gregory P. Smith [Google] and Mark Shannon
in :gh:`96123`.)


Other Language Changes
Expand Down Expand Up @@ -126,6 +134,19 @@ sqlite3
(Contributed by Erlend E. Aasland in :gh:`77617`.)


sys
---

* Add :func:`sys.activate_stack_trampoline` and
:func:`sys.deactivate_stack_trampoline` for activating and deactivating
stack profiler trampolines,
and :func:`sys.is_stack_trampoline_active` for querying if stack profiler
trampolines are active.
(Contributed by Pablo Galindo and Christian Heimes
with contributions from Gregory P. Smith [Google] and Mark Shannon
in :gh:`96123`.)


Optimizations
=============

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
Add a new ``-X perf`` Python command line option as well as
:func:`sys.activate_stack_trampoline` and :func:`sys.deactivate_stack_trampoline`
function in the :mod:`sys` module that allows to set/unset the interpreter in a
way that the Linux ``perf`` profiler can detect Python calls. The new
:func:`sys.is_stack_trampoline_active` function allows to query the state of the
perf trampoline. Design by Pablo Galindo. Patch by Pablo Galindo and Christian Heimes
Add :ref:`perf_profiling` through the new command-line option
:option:`-X perf <-X>`, as well as the new
:func:`sys.activate_stack_trampoline`, :func:`sys.deactivate_stack_trampoline`,
and :func:`sys.is_stack_trampoline_active` APIs.
Design by Pablo Galindo. Patch by Pablo Galindo and Christian Heimes
with contributions from Gregory P. Smith [Google] and Mark Shannon.
8 changes: 4 additions & 4 deletions Python/clinic/sysmodule.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 6 additions & 6 deletions Python/sysmodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -2059,12 +2059,12 @@ sys.activate_stack_trampoline
backend: str
/

Activate the perf profiler trampoline.
Activate stack profiler trampoline *backend*.
[clinic start generated code]*/

static PyObject *
sys_activate_stack_trampoline_impl(PyObject *module, const char *backend)
/*[clinic end generated code: output=5783cdeb51874b43 input=b09020e3a17c78c5]*/
/*[clinic end generated code: output=5783cdeb51874b43 input=a12df928758a82b4]*/
{
#ifdef PY_HAVE_PERF_TRAMPOLINE
if (strcmp(backend, "perf") == 0) {
Expand Down Expand Up @@ -2095,12 +2095,12 @@ sys_activate_stack_trampoline_impl(PyObject *module, const char *backend)
/*[clinic input]
sys.deactivate_stack_trampoline

Dectivate the perf profiler trampoline.
Dectivate stack profiler trampoline backend.
[clinic start generated code]*/

static PyObject *
sys_deactivate_stack_trampoline_impl(PyObject *module)
/*[clinic end generated code: output=b50da25465df0ef1 input=491f4fc1ed615736]*/
/*[clinic end generated code: output=b50da25465df0ef1 input=be0c9a8737fe7e8f]*/
{
if (_PyPerfTrampoline_Init(0) < 0) {
return NULL;
Expand All @@ -2111,12 +2111,12 @@ sys_deactivate_stack_trampoline_impl(PyObject *module)
/*[clinic input]
sys.is_stack_trampoline_active

Returns *True* if the perf profiler trampoline is active.
Return *True* if the stack profiler trampoline is active.
[clinic start generated code]*/

static PyObject *
sys_is_stack_trampoline_active_impl(PyObject *module)
/*[clinic end generated code: output=ab2746de0ad9d293 input=061fa5776ac9dd59]*/
/*[clinic end generated code: output=ab2746de0ad9d293 input=d802fd4a1afa2de8]*/
{
#ifdef PY_HAVE_PERF_TRAMPOLINE
if (_PyIsPerfTrampolineActive()) {
Expand Down