gh-131591: Implement PEP 768 (#131937)

Co-authored-by: Ivona Stojanovic <[email protected]>
Co-authored-by: Matt Wozniski <[email protected]>

Author: 3 people

Doc/library/sys.rst

@@ -1835,6 +1835,28 @@ always available. Unless explicitly noted otherwise, all variables are read-only
 
    .. versionadded:: 3.12
 
+
+.. function:: remote_exec(pid, script)
+
+   Executes *script*, a file containing Python code in the remote
+   process with the given *pid*.
+
+   This function returns immediately, and the code will be executed by the
+   target process's main thread at the next available opportunity, similarly
+   to how signals are handled. There is no interface to determine when the
+   code has been executed. The caller is responsible for making sure that
+   the file still exists whenever the remote process tries to read it and that
+   it hasn't been overwritten.
+
+   The remote process must be running a CPython interpreter of the same major
+   and minor version as the local process. If either the local or remote
+   interpreter is pre-release (alpha, beta, or release candidate) then the
+   local and remote interpreters must be the same exact version.
+
+   .. availability:: Unix, Windows.
+   .. versionadded:: next
+
+
 .. function:: _enablelegacywindowsfsencoding()
 
    Changes the :term:`filesystem encoding and error handler` to 'mbcs' and

Doc/using/cmdline.rst

@@ -603,6 +603,17 @@ Miscellaneous options
 
      .. versionadded:: 3.13
 
+   * ``-X disable_remote_debug`` disables the remote debugging support as described
+     in :pep:`768`.  This includes both the functionality to schedule code for
+     execution in another process and the functionality to receive code for
+     execution in the current process.
+
+     This option is only available on some platforms and will do nothing
+     if is not supported on the current system. See also
+     :envvar:`PYTHON_DISABLE_REMOTE_DEBUG` and :pep:`768`.
+
+     .. versionadded:: next
+
    * :samp:`-X cpu_count={n}` overrides :func:`os.cpu_count`,
      :func:`os.process_cpu_count`, and :func:`multiprocessing.cpu_count`.
      *n* must be greater than or equal to 1.
@@ -1160,7 +1171,16 @@ conflict.
 
    .. versionadded:: 3.13
 
+.. envvar:: PYTHON_DISABLE_REMOTE_DEBUG
+
+   If this variable is set to a non-empty string, it disables the remote
+   debugging feature described in :pep:`768`. This includes both the functionality
+   to schedule code for execution in another process and the functionality to
+   receive code for execution in the current process.
+
+   See also the :option:`-X disable_remote_debug` command-line option.
 
+   .. versionadded:: next
 
 .. envvar:: PYTHON_CPU_COUNT
 

Doc/using/configure.rst

@@ -660,6 +660,17 @@ also be used to improve performance.
    Add ``-fstrict-overflow`` to the C compiler flags (by default we add
    ``-fno-strict-overflow`` instead).
 
+.. option:: --without-remote-debug
+
+   Deactivate remote debugging support described in :pep:`768` (enabled by default).
+   When this flag is provided the code that allows the interpreter to schedule the
+   execution of a Python file in a separate process as described in :pep:`768` is
+   not compiled. This includes both the functionality to schedule code to be executed
+   and the functionality to receive code to be executed.
+
+
+   .. versionadded:: next
+
 
 .. _debug-build:
 

Doc/whatsnew/3.14.rst

@@ -90,6 +90,63 @@ If you encounter :exc:`NameError`\s or pickling errors coming out of
 New features
 ============
 
+.. _whatsnew314-pep678:
+
+PEP 768: Safe external debugger interface for CPython
+-----------------------------------------------------
+
+:pep:`768` introduces a zero-overhead debugging interface that allows debuggers and profilers
+to safely attach to running Python processes. This is a significant enhancement to Python's
+debugging capabilities allowing debuggers to forego unsafe alternatives.
+
+The new interface provides safe execution points for attaching debugger code without modifying
+the interpreter's normal execution path or adding runtime overhead. This enables tools to
+inspect and interact with Python applications in real-time without stopping or restarting
+them — a crucial capability for high-availability systems and production environments.
+
+For convenience, CPython implements this interface through the :mod:`sys` module with a
+:func:`sys.remote_exec` function::
+
+    sys.remote_exec(pid, script_path)
+
+This function allows sending Python code to be executed in a target process at the next safe
+execution point. However, tool authors can also implement the protocol directly as described
+in the PEP, which details the underlying mechanisms used to safely attach to running processes.
+
+Here's a simple example that inspects object types in a running Python process:
+
+  .. code-block:: python
+
+     import os
+     import sys
+     import tempfile
+
+     # Create a temporary script
+     with tempfile.NamedTemporaryFile(mode='w', suffix='.py', delete=False) as f:
+         script_path = f.name
+         f.write(f"import my_debugger; my_debugger.connect({os.getpid()})")
+     try:
+         # Execute in process with PID 1234
+         print("Behold! An offering:")
+         sys.remote_exec(1234, script_path)
+     finally:
+         os.unlink(script_path)
+
+The debugging interface has been carefully designed with security in mind and includes several
+mechanisms to control access:
+
+* A :envvar:`PYTHON_DISABLE_REMOTE_DEBUG` environment variable.
+* A :option:`-X disable-remote-debug` command-line option.
+* A :option:`--without-remote-debug` configure flag to completely disable the feature at build time.
+
+A key implementation detail is that the interface piggybacks on the interpreter's existing evaluation
+loop and safe points, ensuring zero overhead during normal execution while providing a reliable way
+for external processes to coordinate debugging operations.
+
+See :pep:`768` for more details.
+
+(Contributed by Pablo Galindo Salgado, Matt Wozniski, and Ivona Stojanovic in :gh:`131591`.)
+
 .. _whatsnew314-pep758:
 
 PEP 758 – Allow except and except* expressions without parentheses

Include/cpython/initconfig.h

@@ -143,6 +143,7 @@ typedef struct PyConfig {
     int faulthandler;
     int tracemalloc;
     int perf_profiling;
+    int remote_debug;
     int import_time;
     int code_debug_ranges;
     int show_ref_count;

Include/cpython/pystate.h

@@ -29,6 +29,13 @@ typedef int (*Py_tracefunc)(PyObject *, PyFrameObject *, int, PyObject *);
 #define PyTrace_C_RETURN 6
 #define PyTrace_OPCODE 7
 
+/* Remote debugger support */
+#define MAX_SCRIPT_PATH_SIZE 512
+typedef struct _remote_debugger_support {
+    int32_t debugger_pending_call;
+    char debugger_script_path[MAX_SCRIPT_PATH_SIZE];
+} _PyRemoteDebuggerSupport;
+
 typedef struct _err_stackitem {
     /* This struct represents a single execution context where we might
      * be currently handling an exception.  It is a per-coroutine state
@@ -202,6 +209,7 @@ struct _ts {
        The PyThreadObject must hold the only reference to this value.
     */
     PyObject *threading_local_sentinel;
+    _PyRemoteDebuggerSupport remote_debugger_support;
 };
 
 # define Py_C_RECURSION_LIMIT 5000

Include/internal/pycore_ceval.h

@@ -347,6 +347,18 @@ void _Py_unset_eval_breaker_bit_all(PyInterpreterState *interp, uintptr_t bit);
 
 PyAPI_FUNC(_PyStackRef) _PyFloat_FromDouble_ConsumeInputs(_PyStackRef left, _PyStackRef right, double value);
 
+#ifndef Py_SUPPORTS_REMOTE_DEBUG
+    #if defined(__APPLE__)
+    #  if !defined(TARGET_OS_OSX)
+// Older macOS SDKs do not define TARGET_OS_OSX
+    #     define TARGET_OS_OSX 1
+    #  endif
+    #endif
+    #if ((defined(__APPLE__) && TARGET_OS_OSX) || defined(MS_WINDOWS) || (defined(__linux__) && HAVE_PROCESS_VM_READV))
+    #    define Py_SUPPORTS_REMOTE_DEBUG 1
+    #endif
+#endif
+
 #ifdef __cplusplus
 }
 #endif

Include/internal/pycore_debug_offsets.h

@@ -73,6 +73,7 @@ typedef struct _Py_DebugOffsets {
         uint64_t id;
         uint64_t next;
         uint64_t threads_head;
+        uint64_t threads_main;
         uint64_t gc;
         uint64_t imports_modules;
         uint64_t sysdict;
@@ -206,6 +207,15 @@ typedef struct _Py_DebugOffsets {
         uint64_t gi_iframe;
         uint64_t gi_frame_state;
     } gen_object;
+
+    struct _debugger_support {
+        uint64_t eval_breaker;
+        uint64_t remote_debugger_support;
+        uint64_t remote_debugging_enabled;
+        uint64_t debugger_pending_call;
+        uint64_t debugger_script_path;
+        uint64_t debugger_script_path_size;
+    } debugger_support;
 } _Py_DebugOffsets;
 
 
@@ -223,6 +233,7 @@ typedef struct _Py_DebugOffsets {
         .id = offsetof(PyInterpreterState, id), \
         .next = offsetof(PyInterpreterState, next), \
         .threads_head = offsetof(PyInterpreterState, threads.head), \
+        .threads_main = offsetof(PyInterpreterState, threads.main), \
         .gc = offsetof(PyInterpreterState, gc), \
         .imports_modules = offsetof(PyInterpreterState, imports.modules), \
         .sysdict = offsetof(PyInterpreterState, sysdict), \
@@ -326,6 +337,14 @@ typedef struct _Py_DebugOffsets {
         .gi_iframe = offsetof(PyGenObject, gi_iframe), \
         .gi_frame_state = offsetof(PyGenObject, gi_frame_state), \
     }, \
+    .debugger_support = { \
+        .eval_breaker = offsetof(PyThreadState, eval_breaker), \
+        .remote_debugger_support = offsetof(PyThreadState, remote_debugger_support),  \
+        .remote_debugging_enabled = offsetof(PyInterpreterState, config.remote_debug),  \
+        .debugger_pending_call = offsetof(_PyRemoteDebuggerSupport, debugger_pending_call),  \
+        .debugger_script_path = offsetof(_PyRemoteDebuggerSupport, debugger_script_path),  \
+        .debugger_script_path_size = MAX_SCRIPT_PATH_SIZE, \
+    }, \
 }
 
 

Include/internal/pycore_global_objects_fini_generated.h

@@ -686,6 +686,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(salt)
         STRUCT_FOR_ID(sched_priority)
         STRUCT_FOR_ID(scheduler)
+        STRUCT_FOR_ID(script)
         STRUCT_FOR_ID(second)
         STRUCT_FOR_ID(security_attributes)
         STRUCT_FOR_ID(seek)

Include/internal/pycore_global_strings.h

@@ -24,6 +24,8 @@ extern int _PySys_ClearAttrString(PyInterpreterState *interp,
 extern int _PySys_SetFlagObj(Py_ssize_t pos, PyObject *new_value);
 extern int _PySys_SetIntMaxStrDigits(int maxdigits);
 
+extern int _PySysRemoteDebug_SendExec(int pid, int tid, const char *debugger_script_path);
+
 #ifdef __cplusplus
 }
 #endif

Include/internal/pycore_runtime_init_generated.h

@@ -73,6 +73,7 @@ def test_config_get(self):
             ("program_name", str, None),
             ("pycache_prefix", str | None, "pycache_prefix"),
             ("quiet", bool, None),
+            ("remote_debug", int, None),
             ("run_command", str | None, None),
             ("run_filename", str | None, None),
             ("run_module", str | None, None),

Include/internal/pycore_sysmodule.h

@@ -626,6 +626,7 @@ class InitConfigTests(EmbeddingTestsMixin, unittest.TestCase):
         'write_bytecode': True,
         'verbose': 0,
         'quiet': False,
+        'remote_debug': True,
         'user_site_directory': True,
         'configure_c_stdio': False,
         'buffered_stdio': True,
@@ -975,7 +976,7 @@ def test_init_global_config(self):
             'verbose': True,
             'quiet': True,
             'buffered_stdio': False,
-
+            'remote_debug': True,
             'user_site_directory': False,
             'pathconfig_warnings': False,
         }
@@ -1031,6 +1032,7 @@ def test_init_from_config(self):
             'write_bytecode': False,
             'verbose': 1,
             'quiet': True,
+            'remote_debug': True,
             'configure_c_stdio': True,
             'buffered_stdio': False,
             'user_site_directory': False,