Spurious 3 hour timeout on the "dist-x86_64-apple" job

[kennytm]

Symptom: The dist-x86_64-apple takes >3 hours to complete, causing it to fail.

We have investigated the timeouts and did not find any causes yet, other than "rustc becomes too slow to compile" (23 minutes → 30 minutes).

Previous instances:

Date                    PR#
2019-01-10T01:20:36Z    57004    
2019-01-10T13:15:28Z    57355    
2019-01-11T06:08:08Z    57355    
2019-01-11T09:49:53Z    57355    
2019-01-12T01:15:10Z    57234    
2019-01-12T17:09:53Z    56759    
2019-01-12T20:11:33Z    56874    
2019-01-13T02:13:40Z    51487    
2019-01-13T07:03:25Z    57666    
2019-01-17T21:47:49Z    57690    
2019-01-18T04:34:21Z    57690    
2019-01-28T19:57:48Z    57045    
2019-02-08T01:19:57Z    58213    
2019-02-08T04:21:44Z    58191    
2019-02-09T09:48:36Z    57770    
2019-02-11T19:09:53Z    58167    
2019-02-14T20:37:43Z    58406    
2019-02-15T02:57:17Z    58470    
2019-02-15T05:58:57Z    58403    
2019-02-15T09:13:39Z    58406    
2019-02-15T12:15:12Z    58470    

[Mark-Simulacrum]

FWIW, everything gets slower - I suspect something running in parallel on the same hardware.

@alexcrichton - Do you know how many dedicated Macs we have? If it's less than 5 that could explain this since presumably in that case we run on the same machine sometimes. Maybe something else in rust-lang org also has macOS builders and Travis runs that in parallel, too.

[Zoxc]

Can we not raise this time limit so we don't have to restart things and waste time?

[alexcrichton]

@Mark-Simulacrum FWIW we have 10 I believe

[Mark-Simulacrum]

@pietroalbini -- might be good to reach out to Travis and see if there's anyway we can get visibility into which Mac each job is getting scheduled; I wonder if we're sometimes scheduling onto the same mac (instead of using 5 different ones)...

[pietroalbini]

Contacted Travis Support about this.

[pietroalbini]

Travis Support replied, and while this is not the definitive answer (they didn't take a deeper look at the issue yet) it might be possible that we're overcommitting our dedicated Mac Pros if we run at full capacity, since the xcode9.3-moar image is larger than their standard images... That would explain this spurious failure.

[alexcrichton]

Build time time variance has leveled off in the recent week since this change was deployed on Travis's side, @kennytm do you know if this has come up much in the past week or so?

[kennytm]

@alexcrichton Nope, all 3-hour timeouts I knew in the past week were caused by Docker rebuild.