[benchmarking] Get a machine to run benchmarks on

[ericsnowcurrently]

(We'll also run profiling there.)

It will likely sit in my home office.

The procurement process goes through our team admin, with approval going through Graham.

• decide on a name for the host (arguably the most important part 😉)
• order the machine (18 March)
• receive delivery
• set up OS
  • install Ubuntu
  • install dev dependencies
  • add limited "benchmarking" user
  • ...
• set up persistent host in Azure (for tunneling, artifacts, etc.)
  • (having a separate host will help minimize noise/load on the benchmarking machine)
  • provision a VM
  • decide on a name for the host
  • set up DNS?
  • ...
• expose the machine via the Azure host
  • set up SSH tunneling on the benchmarking machine
  • set up port forwarding (for tunneled SSH)?
• set up cpython repo
  • clone
  • add remotes
  • build it once
  • ? add a cron job or service to keep master up-to-date
• set up pyperformance
  • clone the repo
  • add remotes (at least the ericsnowcurrently fork)
  • do "run" once
  • set up a "compile" config
  • do "compile" once
• add users
• document access (and maintenance) info, and share privately
• ? set up automation (see (maybe) Add Tooling and Automation for Benchmarking/Profiling #21)

[ericsnowcurrently]

We took a while to get things sorted out but we're moving forward now. I'm asking for the following (along with a monitor):

• Dell Precision T5820 (Xeon W-2255, 10 cores, 19MB cache; 64GB ECC RAM; 1TB m.2 SSD)

I considered getting something beefier (e.g. dual-proc) but think this should be more than enough (and costs much less).

Also, FYI, the process to get a server (rather than a "desktop") is distinct, as well as slower and more painful.

[gvanrossum]

I have no idea what most of that means but it sounds good to me. :-)

[ericsnowcurrently]

FYI, there aren't any updates yet on the computer I ordered. Apparently there's a sizeable backlog on hardware right now due to global supply issues, meaning orders are getting heavily delayed. I will post once delivery is scheduled (and avoid any other updates like this).

[ericsnowcurrently]

Apparently there is a "test lab" on the Microsoft campus were teams can host computers they use for testing, with a number of benefits (e.g. internal procurement, IT staff, remote console, campus facilities). I'm going to follow-up with Derek Keeler, who should have more info. I'll update this comment with any additional info I get.

[Update 1] I spoke with Derek about the "test lab". He gave me a good explanation, some encouragement, and some recommendations. Long story short, we could benefit from using the (MS-internal) service but there are business processes involved (and a long lead time) that make it not worth it for the one or two machines we need. (FYI, I have more details on the MS-internal team, as well as the "lab" and the related business processes, that I won't post here but can provide on request.)

[Update 2] Derek has provided access to an unused machine he has under his desk at work that we can monopolize as a plan B.

[gvanrossum]

Okay, as long as there will still be a way for Mark to submit jobs and see their output.

[ericsnowcurrently]

Good news: delivery has been scheduled. The machine should arrive by EOD April 1 (tomorrow) April 2 April 5. [That was a cruel April Fools joke, Fedex!] It will take a day or two to get it set up the way we want and ready to run benchmarks. So if all goes well we can start getting reliable benchmark results Monday (or perhaps sooner).

There will be more to do after that but I'm going to focus on our basic needs first. 🙂

[ericsnowcurrently]

The machine has arrived. I'll be setting it up over the next few days.

[ericsnowcurrently]

We should be good to go on this. There's probably more to do, between tweaks and extra functionality, but we can address those separately.