The New REPL Does Not Load My Command History

[ericsnowcurrently]

Bug report

Bug description:

In my ~/.pystartup I have the following code:

import atexit
import os
import readline
import rlcompleter

historyPath = os.path.expanduser("~/.pyhistory")

def save_history(historyPath=historyPath):
    import readline
    readline.write_history_file(historyPath)

if os.path.exists(historyPath):
    readline.read_history_file(historyPath)

atexit.register(save_history)
del os, atexit, readline, rlcompleter, save_history, historyPath

This is something I've been using for years. With this startup file, I can enter the REPL and immediately up-arrow to scroll through the commands I used the last time I used the REPL.

With the new REPL there are two related things going wrong:

• my command history is not loaded (when I immediately up-arrow, it does nothing)
• my command history is not saved in my .pyhistory file (the file is touched but not otherwise modified)

CC @ambv, @pablogsal

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

[pablogsal]

Hummm, it looks like this code it's using readline to handle the story but the new REPL doesn't use readline at all so this will never work. Notice that this will also not work the same if you compile Python against libedit instead of readline because it has different history handling (to the point I know).

@ambv what do you think? Maybe we can monkey patch readline.read_history_file somehow in the new REPL but I am concerned of the side effects.

[whitequark]

I have made a workaround for this issue: #121160 (comment).

[encukou]

@whitequark, AFAIK that's a different issue.
The new REPL noes not use the readline Python module at all. readline.read_history_file won't affect it, regardless of the backend.

[domdfcoding]

I'm seeing a possibly related issue but without a startup file. The first time Python 3.13 runs the history is there, but after exiting and running again the history has gone. Running an older Python populates the history with new entries but they're lost the next time Python 3.13 exits.

Git bisect tracked the issue to 38cfa92. The actual issue is in site.py, where it tries to import CAN_USE_PYREPL from __main__, which it's no longer located in as of that commit.

cpython/Lib/site.py

Lines 527 to 532 in c7991cc

	def write_history():
	try:
	# _pyrepl.__main__ is executed as the __main__ module
	from __main__ import CAN_USE_PYREPL
	except ImportError:
	CAN_USE_PYREPL = False

Changing this to from _pyrepl.main import CAN_USE_PYREPL or adding from .main import CAN_USE_PYREPL to the top of _pyrepl.__main__.py seems to fix it. (I couldn't make it work reliably at first but I think that was me forgetting that site.py is frozen)

[hroncok]

@domdfcoding As this does not require a startup file, I belive it's a different (and more severe issue). I opened #121245

[skirpichev]

@ambv, perhaps #130938 could be closed in favor of this issue? Underlying reason is same: the stdlib's readline module shadows history-related functions from the _pyrepl.readline.

[smontanaro]

I don't use the REPL as much as I used to, and these days don't really use versions before 3.13 all that often, so I'm only stumbling upon this issue now. I have a completions.py file in my interactive startup sequence which does effectively what @ericsnowcurrently's .pystartup file does. I managed to work around it by attempting a _pyrepl import and falling back to my startup if that fails:

try:
    import _pyrepl
    print("using _pyrepl")
except ImportError:
    # ... do the old stuff here ...
    print("using historical readline")

That seems to work. Is there some expectation that the switch when going from <= 3.12 to >= 3.13 should be completely transparent?

[ambv]

Built-in support for the $PYTHON_HISTORY env var was only added in 3.13. The setup people had before in their $PYTHONSTARTUP wasn't standardized and the on-disk format differs between GNU readline and libedit (aka editline). Additionally, the current history format of PyREPL is yet different. We do attempt to import old history files when encountering them, but once saved back, they might not be readable back in older Python versions (or $PYTHON_BASIC_REPL for that matter, which is admittedly a bit of a bummer).