Skip to content

Reduce the variability of the mypy benchmark #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 27, 2023
Merged

Reduce the variability of the mypy benchmark #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
afeb47e
Make mypy results more consistent
mdboom Jan 19, 2023
41d08c8
Reduce the variability of the mypy benchmark
mdboom Jan 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion benchmarks/MANIFEST
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ gevent_hub <local>
gunicorn <local>
json <local>
kinto <local>
mypy <local>
mypy2 <local>
pycparser <local>
pylint <local>
pytorch_alexnet_inference <local>
Expand Down
0 benchmarks/bm_mypy/data/mypy_target.py → benchmarks/bm_mypy2/data/mypy_target.py
View file
Open in desktop
File renamed without changes.
0 benchmarks/bm_mypy/legacyutils.py → benchmarks/bm_mypy2/legacyutils.py
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion benchmarks/bm_mypy/pyproject.toml → benchmarks/bm_mypy2/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[project]
name = "bm_mypy"
name = "bm_mypy2"
dependencies = [
"mypy",
]
Expand Down
0 benchmarks/bm_mypy/requirements.txt → benchmarks/bm_mypy2/requirements.txt
View file
Open in desktop
File renamed without changes.
12 changes: 7 additions & 5 deletions benchmarks/bm_mypy/run_benchmark.py → benchmarks/bm_mypy2/run_benchmark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,6 @@ def _bench_mypy(loops=20, *, legacy=False):
times = []
with open(os.devnull, "w") as devnull:
for i in range(loops):
if legacy:
print(i)
# This is a macro benchmark for a Python implementation
# so "elapsed" covers more than just how long main() takes.
t0 = pyperf.perf_counter()
Expand All @@ -52,8 +50,12 @@ def _bench_mypy(loops=20, *, legacy=False):
pass
t1 = pyperf.perf_counter()

elapsed += t1 - t0
times.append(t0)
# Don't include results from the first run, since it loads the
# files from disk. Subsequent runs will use the file contents in an
# in-memory cache.
if i > 1:
elapsed += t1 - t0
times.append(t0)
times.append(pyperf.perf_counter())
return elapsed, times

Expand All @@ -67,4 +69,4 @@ def _bench_mypy(loops=20, *, legacy=False):

runner = pyperf.Runner()
runner.metadata['description'] = "Test the performance of mypy types"
runner.bench_time_func("mypy", bench_mypy)
runner.bench_time_func("mypy2", bench_mypy)