runtime/trace: TestTraceStressStartStop is unreliable

[davecheney]

pprof.TestTraceStressStartStop is unreliable and causes many builds to be marked as failing.

--- FAIL: TestTraceStressStartStop (0.03s)
trace_test.go:363: failed to parse trace: g 5 is not waiting during syscall exit (offset 357, time 0)
FAIL
FAIL runtime/pprof 8.453s

[davecheney]

https://go-review.googlesource.com/8962

[bradfitz]

Dup of #9729

[rsc]

Unduplicated from #9729 and assigning Dmitriy. This failure seems to be particularly tied to system calls. The old failure was due to running when the world was stopped. We fixed that. I believe this new failure is due to a bug in:

commit 089d363
Author: Dmitry Vyukov dvyukov@google.com
AuthorDate: Fri Apr 10 18:32:40 2015 +0300
Commit: Dmitry Vyukov dvyukov@google.com
CommitDate: Fri Apr 10 17:39:06 2015 +0000

runtime: fix tracing of syscall exit

Fix tracing of syscall exit after:
https://go-review.googlesource.com/#/c/7504/

Change-Id: Idcde2aa826d2b9a05d0a90a80242b6bfa78846ab
Reviewed-on: https://go-review.googlesource.com/8728
Reviewed-by: Rick Hudson <rlh@golang.org>
Run-TryBot: Dmitry Vyukov <dvyukov@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>

[gopherbot]

CL https://golang.org/cl/9132 mentions this issue.

[mewmew]

FYI, reproduced on rev 6f42b61.

$ go version
go version devel +6f42b61 Sun May 3 16:10:40 2015 +0000 linux/amd64

[~/go/src/runtime/pprof]$ go test
--- FAIL: TestTraceSymbolize (22.61s)
    trace_stack_test.go:130: failed to parse trace: p 0 is running before start (offset 669, time 6666204163)
--- FAIL: TestTraceStress (15.24s)
    trace_test.go:224: failed to parse trace: g 82 is not runnable before start (offset 1353327, time 50453959)
--- FAIL: TestTraceFutileWakeup (0.32s)
    trace_test.go:432: failed to parse trace: g 162 is not runnable before start (offset 390, time 155268)
FAIL
exit status 1
FAIL    runtime/pprof   104.363s

[davecheney]

Robert. Is this running inside a VM ? If not, what hardware ?

On Mon, May 4, 2015 at 4:17 AM, Robin Eklind notifications@github.com
wrote:

FYI, reproduced on rev 6f42b61
6f42b61
.

$ go version
go version devel +6f42b61 Sun May 3 16:10:40 2015 +0000 linux/amd64

[~/go/src/runtime/pprof]$ go test
--- FAIL: TestTraceSymbolize (22.61s)
trace_stack_test.go:130: failed to parse trace: p 0 is running before start (offset 669, time 6666204163)
--- FAIL: TestTraceStress (15.24s)
trace_test.go:224: failed to parse trace: g 82 is not runnable before start (offset 1353327, time 50453959)
--- FAIL: TestTraceFutileWakeup (0.32s)
trace_test.go:432: failed to parse trace: g 162 is not runnable before start (offset 390, time 155268)
FAIL
exit status 1
FAIL runtime/pprof 104.363s

—
Reply to this email directly or view it on GitHub
#10476 (comment).

[mewmew]

Robert. Is this running inside a VM ? If not, what hardware ?

Running on real (albeit old) hardware. Including the hardware specs below. P.S. my name is Robin :) Let me know if you need any more info.

[~]$ uname -a
Linux x61s 4.0.1-1-ARCH #1 SMP PREEMPT Wed Apr 29 12:00:26 CEST 2015 x86_64 GNU/Linux

[~]$ cat /proc/cpuinfo 
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 15
model name  : Intel(R) Core(TM)2 Duo CPU     L7500  @ 1.60GHz
stepping    : 11
microcode   : 0xba
cpu MHz     : 1200.000
cache size  : 4096 KB
physical id : 0
siblings    : 2
core id     : 0
cpu cores   : 2
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 10
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dtherm tpr_shadow vnmi flexpriority
bugs        :
bogomips    : 3193.85
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 15
model name  : Intel(R) Core(TM)2 Duo CPU     L7500  @ 1.60GHz
stepping    : 11
microcode   : 0xba
cpu MHz     : 1200.000
cache size  : 4096 KB
physical id : 0
siblings    : 2
core id     : 1
cpu cores   : 2
apicid      : 1
initial apicid  : 1
fpu     : yes
fpu_exception   : yes
cpuid level : 10
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida dtherm tpr_shadow vnmi flexpriority
bugs        :
bogomips    : 3193.85
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

[mewmew]

I applied the changeset in CL https://golang.org/cl/9132, but got the same result:

--- FAIL: TestTraceStress (3.16s)
    trace_test.go:224: failed to parse trace: g 66 is not runnable before start (offset 193346, time 51468022)
--- FAIL: TestTraceStressStartStop (0.06s)
    trace_test.go:363: failed to parse trace: g 65 is not runnable before start (offset 247, time 21113)
--- FAIL: TestTraceFutileWakeup (0.50s)
    trace_test.go:427: failed to parse trace: g 119 is not waiting before unpark (offset 6266, time 2007994)
FAIL
FAIL    runtime/pprof   11.239s