Skip to content

Commit 3179cc3

Browse files
committed
Update quirks doc to match current reality
We configure ncurses with defaults for TERMINFO_DIRS that work on I can find (it works on Debian family, Red Hat family, NixOS, and Alpine), so we shouldn't phrase either the title or the body as if this is a problem that affects most people or setting the environment variable is a necessary thing in general. (Also I think even in cases where we don't have terminfo for your terminal, backspace usually still works, as evidenced by `TERM=potato python`). musl distributions are now dynamic binaries by default and dynamic loading works. The libcrypt transition is behind us. The python3-config file (since its first version, I think) knows how to correct the prefix in output. While it does contain hard-coded paths internally, it uses that for the purpose of fixing its output, so I don't think that fact is helpful to call out. python-build-standalone users who get it through uv are (hopefully) not affected by absolute paths in sysconfig; call this out explicitly for the benefit of people evaluting whether uv will work for them. Also mention sysconfigpatcher for everyone else. In general, tighten up the content.
1 parent 482a9bc commit 3179cc3

File tree

1 file changed

+82
-136
lines changed

1 file changed

+82
-136
lines changed

docs/quirks.rst

Lines changed: 82 additions & 136 deletions
Original file line numberDiff line numberDiff line change
@@ -4,17 +4,21 @@
44
Behavior Quirks
55
===============
66

7+
While these Python distributions are intended to be broadly compatible
8+
with the Python ecosystem, there are a few known behavior quirks that
9+
affect specific environments, packages, or use cases.
10+
711
.. _quirk_backspace_key:
812

9-
Backspace Key Doesn't work in Python REPL
10-
=========================================
13+
If special keys do not work in the Python REPL
14+
==============================================
1115

1216
If you attempt to run ``python`` and the backspace key doesn't
1317
erase characters or the arrow keys don't work as expected, this
1418
is because the executable can't find the *terminfo database*.
1519

16-
A telltale sign of this is the Python REPL printing the following
17-
on startup::
20+
If this happens, the Python REPL will print the following warning
21+
message on startup::
1822

1923
Cannot read termcap database;
2024
using dumb terminal settings.
@@ -35,42 +39,27 @@ you build a program (like Python) locally, you link against
3539
``readline`` or ``libedit`` and get these default locations
3640
*for free*.
3741

38-
Because python-build-standalone Python distributions compile
39-
and use their own version of ``libedit`` and because the build
40-
environment is different from your machine, the default search
41-
locations for the *terminfo database* built into binaries
42-
distributed with this project may point to a path that doesn't
43-
exist. The *terminfo database* cannot be located and ``libedit``
44-
does not know how to convert special key presses to special behavior.
45-
46-
The solution to this is to set an environment variable
47-
with the location of the *terminfo database*.
48-
49-
If running a Debian based Linux distribution (including Ubuntu)::
50-
51-
$ TERMINFO_DIRS=/etc/terminfo:/lib/terminfo:/usr/share/terminfo
52-
53-
If running a RedHat based Linux distribution::
54-
55-
$ TERMINFO_DIRS=/etc/terminfo:/usr/share/terminfo
56-
57-
If running macOS::
58-
59-
$ TERMINFO_DIRS=/usr/share/terminfo
60-
61-
e.g.::
42+
These Python distributions compile and use their own version of
43+
``libedit`` to avoid a dependency on what is (or isn't) installed on
44+
your system. This means that they do not use your system-provided
45+
libraries for reading the *terminfo database*. This version of
46+
``libedit`` is configured to look for in locations that should work for
47+
most OSes (specifically, ``/usr/share/terminfo`` on macOS, and
48+
``/etc/terminfo``, ``/lib/terminfo``, and ``/usr/share/terminfo`` on
49+
Linux, which should cover all major Linux distributions), but it is
50+
possible that your environment has it somewhere else. If your OS stores
51+
the *terminfo database* in an uncommon location, you can set the
52+
``TERMINFO_DIRS`` environment variable so that ``libedit`` can find it.
6253

63-
$ TERMINFO_DIRS=/etc/terminfo:/lib/terminfo:/usr/share/terminfo install/bin/python3.9
54+
For instance, you may need to do something like:
6455

65-
The macOS distributions built with this project should automatically
66-
use the terminfo database in ``/usr/share/terminfo``. Please file
67-
a bug report if the macOS distributions do not behave as expected.
56+
$ TERMINFO_DIRS=/uncommon/place/terminfo install/bin/python3.9
6857

69-
Starting in the first release after 20240107, the Linux distributions are
70-
configured to automatically use the terminfo database in ``/etc/terminfo``,
71-
``/lib/terminfo``, and ``/usr/share/terminfo``.
58+
If you are running on a relatively standard OS and this does not work
59+
out of the box, please file a bug report so we can add the location of
60+
the *terminfo database* to the build.
7261

73-
Also starting in the first release after 20240107, the terminfo database
62+
For convenience, a relatively recent copy of the terminfo database
7463
is distributed in the ``share/terminfo`` directory (``../../share/terminfo``
7564
relative to the ``bin/python3`` executable) in Linux distributions. Note
7665
that ncurses and derived libraries don't know how to find this directory
@@ -150,29 +139,6 @@ Some functionality may behave subtly differently as a result of our choice
150139
to link ``libedit`` by default. (We choose ``libedit`` by default to
151140
avoid GPL licensing requirements of ``readline``.)
152141

153-
Static Linking of musl libc Prevents Extension Module Library Loading
154-
=====================================================================
155-
156-
Our musl libc linked Linux builds link musl libc statically and the resulting
157-
binaries are completely static and don't have any external dependencies.
158-
159-
Due to how Linux/ELF works, a static/non-dynamic binary cannot call
160-
``dlopen()`` and therefore it cannot load shared library based Python
161-
extension modules (``.so`` based extension modules). This significantly
162-
limits the utility of these Python distributions. (If you want to use
163-
additional extension modules you can use the build artifacts in the
164-
distributions to construct a new ``libpython`` with the additional
165-
extension modules configured as builtin extension modules.)
166-
167-
Another consequence of statically linking musl libc is that our musl
168-
distributions aren't compatible with
169-
`PEP 656 <https://www.python.org/dev/peps/pep-0656/>`_. PEP 656
170-
stipulates that Python and extension modules are linked against a
171-
dynamic musl. This is what you'll find in Alpine Linux, for example.
172-
173-
See https://github.com/astral-sh/python-build-standalone/issues/86 for
174-
a tracking issue to improve the state of musl distributions.
175-
176142
.. _quirk_linux_libx11:
177143

178144
Static Linking of ``libX11`` / Incompatibility with PyQt on Linux
@@ -232,40 +198,6 @@ And you can't easily remove ``_tkinter`` and its symbols from the pre-built
232198
and ready-to-use Python install included in this project's distribution
233199
artifacts.
234200

235-
.. _quirk_missing_libcrypt:
236-
237-
Missing ``libcrypt.so.1``
238-
=========================
239-
240-
Linux distributions in the 20230507 release and earlier had a hard dependency
241-
on ``libcrypt.so.1`` due to static linking of the ``_crypt`` extension module,
242-
which imports it.
243-
244-
Presence of ``libcrypt.so.1`` is mandated as part of the Linux Standard Base
245-
Core Specification and therefore should be present in Linux environments
246-
conforming to this specification. Most Linux distributions historically
247-
attempted to conform to this specification.
248-
249-
In 2022, various Linux distributions stopped shipping ``libcrypt.so.1``
250-
(it appears glibc is ceasing to provide this functionality and Linux
251-
distributions aren't backfilling ``libcrypt.so.1`` in the base install
252-
to remain compatible with the Linux Standard Base Core Specification).
253-
254-
In reaction to Linux distributions no longer providing ``libcrypt.so.1`` by
255-
default, we changed the configuration of the ``_crypt`` extension module so
256-
it is compiled/distributed as a standalone shared library and not compiled
257-
into libpython. This means a missing ``libcrypt.so.1`` is only relevant if
258-
the Python interpreter imports the ``crypt`` / ``_crypt`` modules.
259-
260-
If you are using an older release of this project with a hard dependency
261-
on ``libcrypt.so.1`` and don't want to upgrade, you can instruct end-users
262-
to install a ``libxcrypt-compat`` (or comparable) package to provide the
263-
missing ``libcrypt.so.1``.
264-
265-
See https://github.com/astral-sh/python-build-standalone/issues/113 and
266-
https://github.com/astral-sh/python-build-standalone/issues/173 for additional
267-
context on this matter.
268-
269201
.. _quirk_references_to_build_paths:
270202

271203
References to Build-Time Paths
@@ -278,8 +210,8 @@ build-time configuration in a handful of files:
278210
``lib/python3.10/_sysconfigdata__linux_x86_64-linux-gnu.py``.
279211
* In a ``Makefile`` under a ``config-*`` directory in the standard library.
280212
e.g. ``lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile``.
281-
* In ``python*-config`` files. e.g. ``bin/python3.10-config``.
282-
* In ``PYTHON.json`` (mostly reflected values from ``_sysconfigdata_*.py``.
213+
* In python-build-standalone's metadata file ``PYTHON.json`` (mostly
214+
reflected values from ``_sysconfigdata_*.py``).
283215

284216
Each of these serves a different use case. But the general theme is various
285217
aspects of the Python distribution attempt to capture how Python was built.
@@ -290,47 +222,61 @@ module. ``sysconfig`` in turn is used by packaging tools like ``setuptools``
290222
and ``pip`` to figure out how to invoke a compiler for e.g. compiling C
291223
extensions from source.
292224

293-
On Linux, our distributions are built in containers. The container has a
294-
custom build of Clang in a custom filesystem location. And Python is
295-
installed to the prefix ``/install``. So you may see references to
296-
``/install`` in Linux distributions.
297-
298-
On macOS, most distributions are built from GitHub Actions runners. They
299-
use a specific macOS SDK. So you may see references to SDK paths that don't
300-
exist on your machine. e.g.
301-
``/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk``.
302-
303-
On Windows, builds are performed from a temporary directory. So you may
304-
see references to temporary directories in Windows distributions.
305-
306-
**The existence of hard-coded paths in our produced distributions can confuse
307-
consumers of these values and break common workflows, like compiling C
308-
extensions.**
309-
310-
We don't currently have a great idea for how to solve this problem. We
311-
can't hardcode values that will work on every machine because every machine
312-
has different filesystem layouts. For example, if we hardcode ``gcc`` as
313-
the compiler, someone with only ``clang`` installed will complain. And
314-
we certainly don't know where end-users will extract their Python
315-
distribution to!
316-
317-
To solve this problem requires executing dynamic code after extracting
318-
our custom distributions in order to patch these hardcoded values into
319-
conformance with the new machine. We're unsure how to actually do this
320-
because figuring out what values to set is essentially equivalent to
321-
reinventing autoconf / configure! Perhaps we could implement something
322-
that works in common system layouts (e.g. hardcoded defaults for common
323-
distros like Debian/Ubuntu and RedHat).
324-
325-
Until we have a better solution here, just understand that anything looking
326-
at ``sysconfig`` could resolve non-existent paths or names of binaries that
327-
don't exist on the current machine.
328-
329-
Starting with the Linux and macOS distributions released in 2024, we do
330-
normalize some values in these files at build time. Normalizations include:
331-
332-
* Removing compiler flags that are non-portable.
333-
* Removing references to build paths (e.g. ``/tools`` on Linux).
225+
**When installed by `uv <https://docs.astral.sh/uv/>`_ , these absolute
226+
paths are fixed up to point to the actual location on your system where
227+
the distribution was installed, so most of this quirk does not apply.**
228+
The third-party tool `sysconfigpatcher <https://github.com/bluss/sysconfigpatcher>`_
229+
also does this and might be helpful to use or reference if you are
230+
installing these distributions on your own.
231+
232+
In particular, you may see references to our install-time paths on the
233+
build infrastructure, e.g., ``/build`` and ``/install`` on Linux, a
234+
particular SDK in ``/Applications/Xcode.app`` on macOS, and temporary
235+
directories on Windows.
236+
237+
Also, Python reports the compiler and flags in use, just in case it is
238+
needed to make binary-compatible extensions. On Linux, for instance, we
239+
use our own builds of Clang and potentially some flags (warnings,
240+
optimizations, locations of the build environment) that do not work or
241+
apply in other environments. We try to configure Python to remove
242+
unneeded flags and absolute paths to files in the build environment.
243+
references to build-time paths. Python's ``sysconfig`` system requires
244+
listing a compiler, so we leave it set to ``clang`` without the absolute
245+
path, but you should be able to use another compiler like ``gcc`` to
246+
compile extensions, too.
334247

335248
If there is a build time normalization that you think should be performed to
336249
make distributions more portable, please file a GitHub issue.
250+
251+
.. _quirk_former:
252+
.. _quirk_missing_libcrypt:
253+
254+
Former quirks
255+
=============
256+
257+
The following quirks were previously listed on this page but have since
258+
been resolved.
259+
260+
* "Static Linking of musl libc Prevents Extension Module Library
261+
Loading": Starting with the 20250311 release, the default musl
262+
distributions are dynamically linked by default, so extension modules
263+
should work properly. Note that these now require a system-wide
264+
installation of the musl C library. (This is present by default on
265+
musl-based OSes like Alpine, and many glibc-based distros have a
266+
``musl`` package you can safely co-install with glibc, too.) If you
267+
specifically need a statically-linked binary, variants with the
268+
``+static`` build option are available, but these retain the quirk
269+
that compiled extension modules (e.g., ``musllinux`` wheels) cannot be
270+
loaded.
271+
272+
* "Missing ``libcrypt.so.1``": The 20230507 release and earlier required
273+
the system library ``libcrypt.so.1``, which stopped being shipped by
274+
default in several Linux distributions around 2022. Starting with the
275+
20230726 release, this dependency is now only needed by the deprecated
276+
``crypt`` module, which only exists on Python 3.12 and lower. If you
277+
still need this module, your OS may offer a ``libxcrypt`` package to
278+
provide this library. Alternatively, there are suggestions in `What's
279+
New in Python 3.13`_ about third-party replacements for the ``crypt``
280+
module.
281+
282+
.. _What's New in Python 3.13: https://docs.python.org/3/whatsnew/3.13.html#whatsnew313-pep594

0 commit comments

Comments
 (0)