update the benchmark results with Julia 1.8

[johnnychen94]

Since we have made Julia 1.6 the new LTS version, it might make sense to update the benchmark results in https://julialang.org/benchmarks/.

[StefanKarpinski]

We just need someone willing and able to get all the programs running on a single benchmark system.

[MilesCranmer]

Could this instead be set up using GitHub actions for continuous benchmarking?

See https://labs.quansight.org/blog/2021/08/github-actions-benchmarks/ for a discussion. It might be a good option, as the performances on a GitHub action VM are pretty consistent:

If you run all benchmarks with a single action, then you would guarantee the same VM each time, and could measure the performance ratios.

+you could also see the performance comparison across a variety of versions of each language.

(This would exclude proprietary languages such as mathematica)

[StefanKarpinski]

A bunch of the benchmarked programs are proprietary such ass Matlab and Mathematica.

[MilesCranmer]

It's only those two right? I think it's very reasonable to exclude proprietary software in a reproducible benchmark. e.g., https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html excludes any proprietary languages. Yes you would lose a couple of data points, but you would have an always up-to-date benchmark, which I see as significantly more important.

In my opinion, the most important comparisons are against C, Rust, and Fortran (+ maybe numpy), since users of those packages are the ones who would look up speed comparisons - not so much Mathematica users. As long as those are included, we are good.

As an alternative option, it seems there are some free versions provided by MATLAB and Mathematica which are available for GitHub actions:

• https://github.com/matlab-actions/overview/blob/main/README.md
• https://github.com/WolframResearch/WL-FunctionCompile-CI-Template

[timholy]

the most important comparisons are against C, Rust, and Fortran

I disagree. First, it's going to be boring: any language that doesn't get in your way will be pretty fast. Second, the point of Julia is to be good at two things: easier to use than C and faster to run than Python. That's a point the benchmarks have to make, which between their source code and raw numbers, they do.

[MilesCranmer]

easier to use than C and faster to run than Python.

I think that's what benchmarks against fast languages show, no?

1. Show syntax examples to demonstrate its just as easy to use as Python
2. Show benchmarks against C/Rust to show its fast

Anyways, this is a second order effect. The more important point is updating out-of-date benchmarks. Basically I am saying that I don't think mathematica/matlab should be roadblocks to getting updated results against C/Rust/Fortran/Python.

[MilesCranmer]

See #51 which drafts a GitHub workflow for running the suite

[johnfgibson]

My feeling is that MileCranmer is right, that having up-to-date benchmarks against just open languages is better than having them held up, going back multiple Julia versions, in order to have a couple proprietary languages.

I really enjoyed doing the benchmarks back for Julia-1.0, but I haven't able to keep it up, due to the investment of time to update each language environment (many with their own peculiar set-up and build system), and also COVID, which has kept me working at home without access to campus-locked proprietary licenses. I was hoping to return this past fall, but delta and omicron have kept me from that.

So I'm supportive of your effort to do this via GitHub workflow.

[MilesCranmer]

Thanks @johnfgibson.

So, with the loss of my sanity, I finally got the workflow running in #51 - it correctly generates the various csv files. This is in spite of most languages being easy to set up since there are already GitHub actions available which stack up on the same VM. I therefore greatly empathize with @johnfgibson in knowing that you had to set these up manually each time...

The workflow runs for the following languages:

• C, fortran, java, javascript, julia, python, R, and rust.

The following benchmarks are not part of the current workflow, for the reasons given below:

• Lua, Go (install fine, but the benchmarks are out-of-date with current syntax)
• Octave (couldn't find a github action that supports version control)
• Mathematica, MATLAB (proprietary)

I think these excluded languages are lower priority, so I would vote for simply displaying the up-to-date benchmarks with the other languages. Then if/when the broken ones are fixed we can turn them back on. Thoughts?

[MilesCranmer]

Here are the actual updated benchmarks, copied from the workflow's output. Could these automatically update the website after #51 is merged?

Seems like parse_integers got a massive improvement compared to the currently displayed results, which is awesome. matrix_multiply also seems like it has put Julia clear in the lead now:

c,iteration_pi_sum,8.028984
c,matrix_multiply,43.012142
c,matrix_statistics,5.007982
c,parse_integers,0.19634
c,print_to_file,20.508051
c,recursion_fibonacci,0.025188
c,recursion_quicksort,0.422955
c,userfunc_mandelbrot,0.08167
fortran,iteration_pi_sum,8.028663
fortran,matrix_multiply,57.163952
fortran,matrix_statistics,8.046266
fortran,parse_integers,0.753935
fortran,print_to_file,113.916496
fortran,recursion_fibonacci,4.4e-5
fortran,recursion_quicksort,0.483927
fortran,userfunc_mandelbrot,7.8e-5
java,iteration_pi_sum,16.370829
java,iteration_sinc_sum,0.049201
java,matrix_multiply,788.083768
java,matrix_statistics,30.276736
java,parse_integers,0.274402
java,print_to_file,99.797282
java,recursion_fibonacci,0.0424
java,recursion_quicksort,1.006608
java,userfunc_mandelbrot,0.136501
javascript,iteration_pi_sum,10.5
javascript,matrix_multiply,2900.0
javascript,matrix_statistics,46.9
javascript,parse_integers,0.64
javascript,print_to_file,118.0
javascript,recursion_fibonacci,0.109
javascript,recursion_quicksort,1.61
javascript,userfunc_mandelbrot,0.149
julia,iteration_pi_sum,8.028063
julia,matrix_multiply,33.387676
julia,matrix_statistics,8.219065
julia,parse_integers,0.137201
julia,print_to_file,18.368588
julia,recursion_fibonacci,0.0482
julia,recursion_quicksort,0.469904
julia,userfunc_mandelbrot,0.0796
python,iteration_pi_sum,630.6591033935547
python,matrix_multiply,49.559593200683594
python,matrix_statistics,51.499128341674805
python,parse_integers,1.6732215881347656
python,print_to_file,54.22806739807129
python,recursion_fibonacci,2.522706985473633
python,recursion_quicksort,11.09170913696289
python,userfunc_mandelbrot,6.908893585205078
r,iteration_pi_sum,236.0
r,matrix_multiply,116.0
r,matrix_statistics,78.0
r,parse_integers,4.0
r,print_to_file,1325.0
r,recursion_fibonacci,10.0
r,recursion_quicksort,22.0
r,userfunc_mandelbrot,20.0
rust,iteration_pi_sum,8.029562
rust,matrix_multiply,46.196557
rust,matrix_statistics,6.52925
rust,parse_integers,0.186271
rust,print_to_file,11.194186
rust,recursion_fibonacci,0.046293
rust,recursion_quicksort,0.428904
rust,userfunc_mandelbrot,0.080522

[MilesCranmer]

#51 is merged now 🎉

How do we update the benchmarks webpage?

[acxz]

@MilesCranmer thanks for the amazing work on the ci!

Lua, Go (install fine, but the benchmarks are out-of-date with current syntax)

I have taken @sbinet's #27 and added a commit to enable the go benchmark in #55

I'll try to work on getting Lua working if I get some time later. Edit: got Lua working as well.

[acxz]

Also do you know of a way to get the system hardware specifications from the CI machine? While it doesn't necessarily matter to get the comparison, know the actual hardware might help interpret the numbers between CI runs.

[MilesCranmer]

Nice work!

I don't know how to get the hardware specs for a particular workflow. According to the docs, linux runs always use 2-core CPU, 7 GB of RAM memory, 14 GB of SSD disk space, (in a virtual machine) but they don't specify whether the CPU changes.

According to the article here - https://labs.quansight.org/blog/2021/08/github-actions-benchmarks/, the times are noisy, so times should only be interpreted relative to C, rather than absolute times.