Skip to content

Commit f22036b

Browse files
markshannonmarkshannon
committed
Merge branch 'main' into thread-local-interpreter-state
2 parents 73c410b + 4776b07 commit f22036b

34 files changed

+521
-263
lines changed

‎Doc/library/dis.rst

Lines changed: 20 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -755,15 +755,6 @@ iterations of the loop.
755755
.. versionadded:: 3.11
756756

757757

758-
.. opcode:: COPY_DICT_WITHOUT_KEYS
759-
760-
TOS is a tuple of mapping keys, and TOS1 is the match subject. Replace TOS
761-
with a :class:`dict` formed from the items of TOS1, but without any of the
762-
keys in TOS.
763-
764-
.. versionadded:: 3.10
765-
766-
767758
.. opcode:: GET_LEN
768759

769760
Push ``len(TOS)`` onto the stack.
@@ -795,11 +786,14 @@ iterations of the loop.
795786

796787
TOS is a tuple of mapping keys, and TOS1 is the match subject. If TOS1
797788
contains all of the keys in TOS, push a :class:`tuple` containing the
798-
corresponding values, followed by ``True``. Otherwise, push ``None``,
799-
followed by ``False``.
789+
corresponding values. Otherwise, push ``None``.
800790

801791
.. versionadded:: 3.10
802792

793+
.. versionchanged:: 3.11
794+
Previously, this instruction also pushed a boolean value indicating
795+
success (``True``) or failure (``False``).
796+
803797

804798
All of the following opcodes use their arguments.
805799

@@ -1277,12 +1271,16 @@ All of the following opcodes use their arguments.
12771271
against, and TOS2 is the match subject. *count* is the number of positional
12781272
sub-patterns.
12791273

1280-
Pop TOS. If TOS2 is an instance of TOS1 and has the positional and keyword
1281-
attributes required by *count* and TOS, set TOS to ``True`` and TOS1 to a
1282-
tuple of extracted attributes. Otherwise, set TOS to ``False``.
1274+
Pop TOS, TOS1, and TOS2. If TOS2 is an instance of TOS1 and has the
1275+
positional and keyword attributes required by *count* and TOS, push a tuple
1276+
of extracted attributes. Otherwise, push ``None``.
12831277

12841278
.. versionadded:: 3.10
12851279

1280+
.. versionchanged:: 3.11
1281+
Previously, this instruction also pushed a boolean value indicating
1282+
success (``True``) or failure (``False``).
1283+
12861284
.. opcode:: GEN_START (kind)
12871285

12881286
Pops TOS. If TOS was not ``None``, raises an exception. The ``kind``
@@ -1301,6 +1299,14 @@ All of the following opcodes use their arguments.
13011299
.. versionadded:: 3.10
13021300

13031301

1302+
.. opcode:: COPY (i)
1303+
1304+
Push the *i*-th item to the top of the stack. The item is not removed from its
1305+
original location.
1306+
1307+
.. versionadded:: 3.11
1308+
1309+
13041310
.. opcode:: HAVE_ARGUMENT
13051311

13061312
This is not really an opcode. It identifies the dividing line between

‎Doc/library/mmap.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -256,6 +256,14 @@ To map anonymous memory, -1 should be passed as the fileno along with the length
256256
with :const:`ACCESS_READ` or :const:`ACCESS_COPY`, resizing the map will
257257
raise a :exc:`TypeError` exception.
258258

259+
**On Windows**: Resizing the map will raise an :exc:`OSError` if there are other
260+
maps against the same named file. Resizing an anonymous map (ie against the
261+
pagefile) will silently create a new map with the original data copied over
262+
up to the length of the new size.
263+
264+
.. versionchanged:: 3.11
265+
Correctly fails if attempting to resize when another map is held
266+
Allows resize against an anonymous map on Windows
259267

260268
.. method:: rfind(sub[, start[, end]])
261269

‎Doc/requirements.txt

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@
44
# won't suddenly cause build failures. Updating the version is fine as long
55
# as no warnings are raised by doing so.
66
sphinx==3.2.1
7+
# Docutils version is pinned to a version compatible with Sphinx
8+
# version 3.2.1. It can be removed after bumping Sphinx version to at
9+
# least 3.5.4.
10+
docutils==0.17.1
711

812
blurb
913

‎Doc/whatsnew/3.11.rst

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -308,6 +308,16 @@ CPython bytecode changes
308308
fashion as :opcode:`CALL_METHOD`, but also supports keyword arguments. Works
309309
in tandem with :opcode:`LOAD_METHOD`.
310310

311+
* Removed ``COPY_DICT_WITHOUT_KEYS``.
312+
313+
* :opcode:`MATCH_CLASS` and :opcode:`MATCH_KEYS` no longer push an additional
314+
boolean value indicating whether the match succeeded or failed. Instead, they
315+
indicate failure with :const:`None` (where a tuple of extracted values would
316+
otherwise be).
317+
318+
* Added :opcode:`COPY`, which pushes the *i*-th item to the top of the stack.
319+
The item is not removed from its original location.
320+
311321

312322
Deprecated
313323
==========

‎Include/opcode.h

Lines changed: 19 additions & 19 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Lib/importlib/_bootstrap_external.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -364,6 +364,8 @@ def _write_atomic(path, data, mode=0o666):
364364
# Python 3.11a1 3459 (PEP 657: add end line numbers and column offsets for instructions)
365365
# Python 3.11a1 3460 (Add co_qualname field to PyCodeObject bpo-44530)
366366
# Python 3.11a1 3461 (JUMP_ABSOLUTE must jump backwards)
367+
# Python 3.11a2 3462 (bpo-44511: remove COPY_DICT_WITHOUT_KEYS, change
368+
# MATCH_CLASS and MATCH_KEYS, and add COPY)
367369

368370
#
369371
# MAGIC must change whenever the bytecode emitted by the compiler may no
@@ -373,7 +375,7 @@ def _write_atomic(path, data, mode=0o666):
373375
# Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array
374376
# in PC/launcher.c must also be updated.
375377

376-
MAGIC_NUMBER = (3461).to_bytes(2, 'little') + b'\r\n'
378+
MAGIC_NUMBER = (3462).to_bytes(2, 'little') + b'\r\n'
377379
_RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c
378380

379381
_PYCACHE = '__pycache__'

‎Lib/opcode.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ def jabs_op(name, op):
8585
def_op('MATCH_MAPPING', 31)
8686
def_op('MATCH_SEQUENCE', 32)
8787
def_op('MATCH_KEYS', 33)
88-
def_op('COPY_DICT_WITHOUT_KEYS', 34)
88+
8989
def_op('PUSH_EXC_INFO', 35)
9090

9191
def_op('POP_EXCEPT_AND_RERAISE', 37)
@@ -165,7 +165,7 @@ def jabs_op(name, op):
165165
def_op('IS_OP', 117)
166166
def_op('CONTAINS_OP', 118)
167167
def_op('RERAISE', 119)
168-
168+
def_op('COPY', 120)
169169
jabs_op('JUMP_IF_NOT_EXC_MATCH', 121)
170170

171171
def_op('LOAD_FAST', 124) # Local variable number

‎Lib/test/test_cmd_line.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,21 @@ def run_python(*args):
123123
else:
124124
self.assertEqual(err, b'')
125125

126+
def test_xoption_frozen_modules(self):
127+
tests = {
128+
('=on', 'FrozenImporter'),
129+
('=off', 'SourceFileLoader'),
130+
('=', 'FrozenImporter'),
131+
('', 'FrozenImporter'),
132+
}
133+
for raw, expected in tests:
134+
cmd = ['-X', f'frozen_modules{raw}',
135+
#'-c', 'import os; print(os.__spec__.loader.__name__, end="")']
136+
'-c', 'import os; print(os.__spec__.loader, end="")']
137+
with self.subTest(raw):
138+
res = assert_python_ok(*cmd)
139+
self.assertRegex(res.out.decode('utf-8'), expected)
140+
126141
def test_run_module(self):
127142
# Test expected operation of the '-m' switch
128143
# Switch needs an argument

‎Lib/test/test_embed.py

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1466,6 +1466,30 @@ def test_get_argc_argv(self):
14661466
self.run_embedded_interpreter("test_get_argc_argv")
14671467
# ignore output
14681468

1469+
def test_init_use_frozen_modules(self):
1470+
tests = {
1471+
('=on', 1),
1472+
('=off', 0),
1473+
('=', 1),
1474+
('', 1),
1475+
}
1476+
for raw, expected in tests:
1477+
optval = f'frozen_modules{raw}'
1478+
config = {
1479+
'parse_argv': 2,
1480+
'argv': ['-c'],
1481+
'orig_argv': ['./argv0', '-X', optval, '-c', 'pass'],
1482+
'program_name': './argv0',
1483+
'run_command': 'pass\n',
1484+
'use_environment': 1,
1485+
'xoptions': [optval],
1486+
'use_frozen_modules': expected,
1487+
}
1488+
env = {'TESTFROZEN': raw[1:]} if raw else None
1489+
with self.subTest(repr(raw)):
1490+
self.check_all_configs("test_init_use_frozen_modules", config,
1491+
api=API_PYTHON, env=env)
1492+
14691493

14701494
class SetConfigTests(unittest.TestCase):
14711495
def test_set_config(self):

‎Lib/test/test_mmap.py

Lines changed: 75 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
import os
66
import re
77
import itertools
8+
import random
89
import socket
910
import sys
1011
import weakref
@@ -707,7 +708,6 @@ def test_write_returning_the_number_of_bytes_written(self):
707708
self.assertEqual(mm.write(b"yz"), 2)
708709
self.assertEqual(mm.write(b"python"), 6)
709710

710-
@unittest.skipIf(os.name == 'nt', 'cannot resize anonymous mmaps on Windows')
711711
def test_resize_past_pos(self):
712712
m = mmap.mmap(-1, 8192)
713713
self.addCleanup(m.close)
@@ -796,6 +796,80 @@ def test_madvise(self):
796796
self.assertEqual(m.madvise(mmap.MADV_NORMAL, 0, 2), None)
797797
self.assertEqual(m.madvise(mmap.MADV_NORMAL, 0, size), None)
798798

799+
@unittest.skipUnless(os.name == 'nt', 'requires Windows')
800+
def test_resize_up_when_mapped_to_pagefile(self):
801+
"""If the mmap is backed by the pagefile ensure a resize up can happen
802+
and that the original data is still in place
803+
"""
804+
start_size = PAGESIZE
805+
new_size = 2 * start_size
806+
data = bytes(random.getrandbits(8) for _ in range(start_size))
807+
808+
m = mmap.mmap(-1, start_size)
809+
m[:] = data
810+
m.resize(new_size)
811+
self.assertEqual(len(m), new_size)
812+
self.assertEqual(m[:start_size], data[:start_size])
813+
814+
@unittest.skipUnless(os.name == 'nt', 'requires Windows')
815+
def test_resize_down_when_mapped_to_pagefile(self):
816+
"""If the mmap is backed by the pagefile ensure a resize down up can happen
817+
and that a truncated form of the original data is still in place
818+
"""
819+
start_size = PAGESIZE
820+
new_size = start_size // 2
821+
data = bytes(random.getrandbits(8) for _ in range(start_size))
822+
823+
m = mmap.mmap(-1, start_size)
824+
m[:] = data
825+
m.resize(new_size)
826+
self.assertEqual(len(m), new_size)
827+
self.assertEqual(m[:new_size], data[:new_size])
828+
829+
@unittest.skipUnless(os.name == 'nt', 'requires Windows')
830+
def test_resize_fails_if_mapping_held_elsewhere(self):
831+
"""If more than one mapping is held against a named file on Windows, neither
832+
mapping can be resized
833+
"""
834+
start_size = 2 * PAGESIZE
835+
reduced_size = PAGESIZE
836+
837+
f = open(TESTFN, 'wb+')
838+
f.truncate(start_size)
839+
try:
840+
m1 = mmap.mmap(f.fileno(), start_size)
841+
m2 = mmap.mmap(f.fileno(), start_size)
842+
with self.assertRaises(OSError):
843+
m1.resize(reduced_size)
844+
with self.assertRaises(OSError):
845+
m2.resize(reduced_size)
846+
m2.close()
847+
m1.resize(reduced_size)
848+
self.assertEqual(m1.size(), reduced_size)
849+
self.assertEqual(os.stat(f.fileno()).st_size, reduced_size)
850+
finally:
851+
f.close()
852+
853+
@unittest.skipUnless(os.name == 'nt', 'requires Windows')
854+
def test_resize_succeeds_with_error_for_second_named_mapping(self):
855+
"""If a more than one mapping exists of the same name, none of them can
856+
be resized: they'll raise an Exception and leave the original mapping intact
857+
"""
858+
start_size = 2 * PAGESIZE
859+
reduced_size = PAGESIZE
860+
tagname = "TEST"
861+
data_length = 8
862+
data = bytes(random.getrandbits(8) for _ in range(data_length))
863+
864+
m1 = mmap.mmap(-1, start_size, tagname=tagname)
865+
m2 = mmap.mmap(-1, start_size, tagname=tagname)
866+
m1[:data_length] = data
867+
self.assertEqual(m2[:data_length], data)
868+
with self.assertRaises(OSError):
869+
m1.resize(reduced_size)
870+
self.assertEqual(m1.size(), start_size)
871+
self.assertEqual(m1[:data_length], data)
872+
self.assertEqual(m2[:data_length], data)
799873

800874
class LargeMmapTests(unittest.TestCase):
801875

‎Makefile.pre.in

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
# === Variables set by makesetup ===
2222

2323
MODBUILT_NAMES= _MODBUILT_NAMES_
24+
MODSHARED_NAMES= _MODSHARED_NAMES_
2425
MODDISABLED_NAMES= _MODDISABLED_NAMES_
2526
MODOBJS= _MODOBJS_
2627
MODLIBS= _MODLIBS_
@@ -2118,6 +2119,28 @@ Python/thread.o: @THREADHEADERS@ $(srcdir)/Python/condvar.h
21182119
.PHONY: build_all_generate_profile build_all_merge_profile
21192120
.PHONY: gdbhooks
21202121

2122+
##########################################################################
2123+
# Module dependencies
2124+
2125+
MODULE_CMATH_DEPS=$(srcdir)/Modules/_math.h
2126+
MODULE_MATH_DEPS=$(srcdir)/Modules/_math.h
2127+
MODULE_PYEXPAT_DEPS=$(srcdir)/Modules/expat/ascii.h $(srcdir)/Modules/expat/asciitab.h $(srcdir)/Modules/expat/expat.h $(srcdir)/Modules/expat/expat_config.h $(srcdir)/Modules/expat/expat_external.h $(srcdir)/Modules/expat/internal.h $(srcdir)/Modules/expat/latin1tab.h $(srcdir)/Modules/expat/utf8tab.h $(srcdir)/Modules/expat/xmlrole.h $(srcdir)/Modules/expat/xmltok.h $(srcdir)/Modules/expat/xmltok_impl.h
2128+
MODULE_UNICODEDATA_DEPS=$(srcdir)/Modules/unicodedata_db.h $(srcdir)/Modules/unicodename_db.h
2129+
MODULE__BLAKE2_DEPS=$(srcdir)/Modules/_blake2/impl/blake2-config.h $(srcdir)/Modules/_blake2/impl/blake2-dispatch.c $(srcdir)/Modules/_blake2/impl/blake2-impl.h $(srcdir)/Modules/_blake2/impl/blake2-kat.h $(srcdir)/Modules/_blake2/impl/blake2.h $(srcdir)/Modules/_blake2/impl/blake2b-load-sse2.h $(srcdir)/Modules/_blake2/impl/blake2b-load-sse41.h $(srcdir)/Modules/_blake2/impl/blake2b-ref.c $(srcdir)/Modules/_blake2/impl/blake2b-round.h $(srcdir)/Modules/_blake2/impl/blake2b-test.c $(srcdir)/Modules/_blake2/impl/blake2b.c $(srcdir)/Modules/_blake2/impl/blake2bp-test.c $(srcdir)/Modules/_blake2/impl/blake2bp.c $(srcdir)/Modules/_blake2/impl/blake2s-load-sse2.h $(srcdir)/Modules/_blake2/impl/blake2s-load-sse41.h $(srcdir)/Modules/_blake2/impl/blake2s-load-xop.h $(srcdir)/Modules/_blake2/impl/blake2s-ref.c $(srcdir)/Modules/_blake2/impl/blake2s-round.h $(srcdir)/Modules/_blake2/impl/blake2s-test.c $(srcdir)/Modules/_blake2/impl/blake2s.c $(srcdir)/Modules/_blake2/impl/blake2sp-test.c $(srcdir)/Modules/_blake2/impl/blake2sp.c $(srcdir)/Modules/hashlib.h
2130+
MODULE__CTYPES_DEPS=$(srcdir)/Modules/_ctypes/ctypes.h
2131+
MODULE__DECIMAL_DEPS=$(srcdir)/Modules/_decimal/docstrings.h $(srcdir)/Modules/_decimal/libmpdec/basearith.h $(srcdir)/Modules/_decimal/libmpdec/bits.h $(srcdir)/Modules/_decimal/libmpdec/constants.h $(srcdir)/Modules/_decimal/libmpdec/convolute.h $(srcdir)/Modules/_decimal/libmpdec/crt.h $(srcdir)/Modules/_decimal/libmpdec/difradix2.h $(srcdir)/Modules/_decimal/libmpdec/fnt.h $(srcdir)/Modules/_decimal/libmpdec/fourstep.h $(srcdir)/Modules/_decimal/libmpdec/io.h $(srcdir)/Modules/_decimal/libmpdec/mpalloc.h $(srcdir)/Modules/_decimal/libmpdec/mpdecimal.h $(srcdir)/Modules/_decimal/libmpdec/numbertheory.h $(srcdir)/Modules/_decimal/libmpdec/sixstep.h $(srcdir)/Modules/_decimal/libmpdec/transpose.h $(srcdir)/Modules/_decimal/libmpdec/typearith.h $(srcdir)/Modules/_decimal/libmpdec/umodarith.h
2132+
MODULE__ELEMENTTREE_DEPS=$(srcdir)/Modules/expat/ascii.h $(srcdir)/Modules/expat/asciitab.h $(srcdir)/Modules/expat/expat.h $(srcdir)/Modules/expat/expat_config.h $(srcdir)/Modules/expat/expat_external.h $(srcdir)/Modules/expat/internal.h $(srcdir)/Modules/expat/latin1tab.h $(srcdir)/Modules/expat/utf8tab.h $(srcdir)/Modules/expat/xmlparse.c $(srcdir)/Modules/expat/xmlrole.c $(srcdir)/Modules/expat/xmlrole.h $(srcdir)/Modules/expat/xmltok.c $(srcdir)/Modules/expat/xmltok.h $(srcdir)/Modules/expat/xmltok_impl.h $(srcdir)/Modules/pyexpat.c
2133+
MODULE__HASHLIB_DEPS=$(srcdir)/Modules/hashlib.h
2134+
MODULE__IO_DEPS=$(srcdir)/Modules/_io/_iomodule.h
2135+
MODULE__MD5_DEPS=$(srcdir)/Modules/hashlib.h
2136+
MODULE__SHA1_DEPS=$(srcdir)/Modules/hashlib.h
2137+
MODULE__SHA256_DEPS=$(srcdir)/Modules/hashlib.h
2138+
MODULE__SHA3_DEPS=$(srcdir)/Modules/_sha3/kcp/KeccakHash.c $(srcdir)/Modules/_sha3/kcp/KeccakHash.h $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-64.macros $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-SnP-opt32.h $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-SnP-opt64.h $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-SnP.h $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-inplace32BI.c $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-opt64-config.h $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-opt64.c $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-unrolling.macros $(srcdir)/Modules/_sha3/kcp/KeccakSponge.c $(srcdir)/Modules/_sha3/kcp/KeccakSponge.h $(srcdir)/Modules/_sha3/kcp/KeccakSponge.inc $(srcdir)/Modules/_sha3/kcp/PlSnP-Fallback.inc $(srcdir)/Modules/_sha3/kcp/SnP-Relaned.h $(srcdir)/Modules/_sha3/kcp/align.h $(srcdir)/Modules/hashlib.h
2139+
MODULE__SHA512_DEPS=$(srcdir)/Modules/hashlib.h
2140+
MODULE__SOCKET_DEPS=$(srcdir)/Modules/socketmodule.h
2141+
MODULE__SSL_DEPS=$(srcdir)/Modules/_ssl.h $(srcdir)/Modules/_ssl/cert.c $(srcdir)/Modules/_ssl/debughelpers.c $(srcdir)/Modules/_ssl/misc.c $(srcdir)/Modules/_ssl_data.h $(srcdir)/Modules/_ssl_data_111.h $(srcdir)/Modules/_ssl_data_300.h $(srcdir)/Modules/socketmodule.h
2142+
MODULE__TESTCAPI_DEPS=$(srcdir)/Modules/testcapi_long.h
2143+
21212144
# IF YOU PUT ANYTHING HERE IT WILL GO AWAY
21222145
# Local Variables:
21232146
# mode: makefile

‎Misc/NEWS.d/next/Build/2021-10-24-11-02-43.bpo-45595.WI_5YU.rst

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
``setup.py`` and ``makesetup`` now track build dependencies on all Python
2+
header files and module specific header files.

‎Misc/NEWS.d/next/Core and Builtins/2021-06-26-16-55-08.bpo-44511.k8sMvV.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Improve the generated bytecode for class and mapping patterns.

0 commit comments

Comments
 (0)