runtime: "found bad pointer in Go heap" on linux-mips64le-mengzhuo builder

[bcmills]

2019-11-12T05:18:25-194ae32/linux-mips64le-mengzhuo
2019-11-11T16:02:42-f511467/linux-mips64le-mengzhuo

Possibly related to #34835, #32324.

CC @mengzhuo @cherrymui

[mengzhuo]

I can't reproduce by all.bash nor dist test ( tried serval times)
Bad memory block?

[cherrymui]

Eyeballing the hex dump, it looks to me that the bad pointer is found in a g struct, in g._defer field.

• 2019-11-11T16:02:42-f511467/linux-mips64le-mengzhuo:

object=0xc000106300 s.base()=0xc000106000 s.limit=0xc000107f80 s.spanclass=42 s.elemsize=384 s.state=mSpanInUse
 *(object+0) = 0xc00052a000		g.stack.lo
 *(object+8) = 0xc000532000		g.stack.hi
 *(object+16) = 0xc00052a380		g.stackguard0 = g.stack.lo + _Stackguard = g.stack.lo + 896
 *(object+24) = 0xffffffffffffffff	g.stackguard1
 *(object+32) = 0x0			g._panic
 *(object+40) = 0xc0005313e0 <==	g._defer
 *(object+48) = 0xc000080380		g.m
 *(object+56) = 0xc0005310b8		g.sched.sp
 *(object+64) = 0x87038			g.sched.pc
 *(object+72) = 0xc000106300		g.sched.g
 ...

The stack bounds matches the stack of goroutine 67, as well as g.sched.pc and g.sched.sp.

What's interesting is that the bad pointer 0xc0005313e0 is also within the stack bound, in particular, within this frame

io/ioutil.readAll(0x3fdc60, 0xc000203a40, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/io/ioutil/ioutil.go:36 +0xd8 fp=0xc000531360 sp=0xc0005312e8 pc=0x170790

Presumably it is a stack-allocated defer. I don't know why spanOf does not find the span of the stack, returns a different span instead...

• 2019-11-12T05:18:25-194ae32/linux-mips64le-mengzhuo

This also looks like the g._defer field in a g. The stack bounds match goroutine 1.

What's more interesting is that findObject and badPointer were complaining about pointer 0xc000346ef8, but in the object dump the field is actually containing 0xc00034a688, a different value.

runtime: pointer 0xc000346ef8 to unallocated span span.base()=0xc000346000 span.limit=0xc000355e00 span.state=0
runtime: found in object at *(0xc000000180+0x28)
object=0xc000000180 s.base()=0xc000000000 s.limit=0xc000001f80 s.spanclass=42 s.elemsize=384 s.state=mSpanInUse
 *(object+0) = 0xc00033c000		g.stack.lo
 *(object+8) = 0xc00034c000		g.stack.hi
 *(object+16) = 0xc00033c380		g.stackguard0 = g.stack.lo + _Stackguard = g.stack.lo + 896
 *(object+24) = 0xffffffffffffffff	g.stackguard1
 *(object+32) = 0x0			g._panic
 *(object+40) = 0xc00034a688 <==	g._defer

I think g._defer can only change synchronously, right? But the goroutine is not running...

Also, the span's address range, [0xc000346000, 0xc000355e00], overlaps with the stack bounds [0xc00033c000, 0xc00034c000], assuming my interpretation of the hex dump is right. How could this be possible?

@aclements any idea how g._defer could change this way, or how span tracking could go weirdly?

[cherrymui]

Another one today https://build.golang.org/log/a810adac990353a9918a366fc13538c03649d206
Also looks like g._defer pointing to the stack.

[mengzhuo]

@cherrymui
The PAGESIZE setting of this builder is 16384 bytes and
0xc000346000 % 16384 = 8192

It looks like some bug of mpagealloc?

[mknyszek]

@mengzhuo Only the scavenger (mgcscavenge.go) operates in terms of physical pages, the runtime page size is always 8 KiB on all platforms, so 8 KiB alignment is exactly what we want.

It could be related to the page allocator, but the physical page size here doesn't matter, I don't think.

[ianlancetaylor]

2019-12-05T17:23:50-50535e6/linux-mips64le-mengzhuo
2019-12-05T01:48:12-fa3a121/linux-mips64le-mengzhuo
2019-12-03T15:26:50-2ac1ca9/linux-mips64le-mengzhuo
2019-11-19T19:59:21-8b1e8a4/linux-mips64le-mengzhuo
2019-11-18T22:08:03-fff4502/linux-mips64le-mengzhuo
2019-11-15T20:56:35-9af8794/freebsd-arm64-dmgk
2019-11-15T02:31:58-498eaee/linux-mips64le-mengzhuo
2019-11-14T00:58:33-e77106c/linux-mips64le-mengzhuo
2019-11-12T22:28:47-13fb147/linux-mips64le-mengzhuo

[bcmills]

Still occurring regularly. Does this need to be addressed for the 1.14 release? Should we mention it as a known issue in the release notes?
(CC @cagedmantis @dmitshur @toothrot)

2020-01-21T23:58:54-fb8cdb7/linux-mips64le-mengzhuo
2020-01-07T18:40:18-7340d15/linux-mips64le-mengzhuo
2019-12-20T23:07:52-4d5bb9c/linux-mips64le-mengzhuo
2019-12-18T19:09:23-87a5467/linux-mips64le-mengzhuo
2019-12-10T21:11:20-207a0b7/linux-mips64le-mengzhuo

[toothrot]

I think we would need input from the runtime team to make that decision, or anyone else on this thread with sufficient expertise (@cherrymui @ianlancetaylor @aclements @rsc @randall77)

It's interesting that the failure also happened on freebsd-arm64, as @ianlancetaylor linked. Should the title be updated?

[bcmills]

I'm not sure whether the freebsd-arm64-dmgk failure is the same underlying cause.

(I'd prefer to keep this issue focused on the mips64le builder unless we have a specific reason to suspect the same root cause for both.)

[cherrymui]

I looked at a few new MIPS failures and they all look like the ones before, and are consistent with my previous comment. The "bad pointer" seems from g._defer, which points to the stack (within the stack bounds), but spanOf returns a different span. I don't yet understand how this could happen.

The freebsd-arm64 one is different, and may be fixed by CL https://go-review.googlesource.com/c/go/+/212626 .

[mengzhuo]

I think it’s unnecessary. The kernel development of Loongson is out of tree and not stable. I’ve tried to use Linux 5.4, but it has an unknown bug in vsyscall and causing segment fault. Bryan C. Mills <notifications@github.com>于2020年1月22日 周三22:32写道：

…

Still occurring regularly. Does this need to be addressed for the 1.14 release? Should we mention it as a known issue in the release notes? (CC @cagedmantis <https://github.com/cagedmantis> @dmitshur <https://github.com/dmitshur> @toothrot <https://github.com/toothrot>) 2020-01-21T23:58:54-fb8cdb7/linux-mips64le-mengzhuo <https://build.golang.org/log/fa85cfe17b4648348e5b5de9a069f2afceabaeb1> 2020-01-07T18:40:18-7340d15/linux-mips64le-mengzhuo <https://build.golang.org/log/6f063f52e50c29c5afb522fa6d8a0f340cfd7418> 2019-12-20T23:07:52-4d5bb9c/linux-mips64le-mengzhuo <https://build.golang.org/log/7592f9e2accda55b972da33ac348f133bf6fdc69> 2019-12-18T19:09:23-87a5467/linux-mips64le-mengzhuo <https://build.golang.org/log/26562517daa862c4feabd0c1daa0ea3f6e6674f1> 2019-12-10T21:11:20-207a0b7/linux-mips64le-mengzhuo <https://build.golang.org/log/2ee2b502cf3b7af0c3f495e46655eb59e82d05b4> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35541?email_source=notifications&email_token=AAGYHHWHUFDNI66XBRV3QGTQ7BKJTA5CNFSM4JMGUVW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJTYB2Y#issuecomment-577208555>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGYHHS2HWBNR7W726HEPLDQ7BKJTANCNFSM4JMGUVWQ> .

[bcmills]

2020-02-20T03:27:46-715e8bb/linux-mips64le-mengzhuo
2020-02-19T21:00:02-1bf9d3c/linux-mips64le-mengzhuo
2020-02-17T10:30:03-88e564e/linux-mips64le-mengzhuo
2020-02-12T17:32:01-7385947/linux-mips64le-mengzhuo
2020-01-31T00:36:45-25d0ffe/linux-mips64le-mengzhuo
2020-01-29T02:43:46-a6701d8/linux-mips64le-mengzhuo
2020-01-24T19:37:58-67539f6/linux-mips64le-mengzhuo