runtime: linux/arm64 runtime.Breakpoint times out

[9nut]

Please answer these questions before submitting your issue. Thanks!

1. What version of Go are you using (go version)?
   go 1.6 and current tip (go version devel +fc6bcde Fri Mar 18 23:26:36 2016 +0000 linux/arm64)
2. What operating system and processor architecture are you using (go env)?
   GOARCH="arm64"
   GOBIN=""
   GOEXE=""
   GOHOSTARCH="arm64"
   GOHOSTOS="linux"
   GOOS="linux"
   GOPATH=""
   GORACE=""
   GOROOT="/home/odroid/go"
   GOTOOLDIR="/home/odroid/go/pkg/tool/linux_arm64"
   CC="gcc"
   GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build271441235=/tmp/go-build -gno-record-gcc-switches"
   CXX="g++"
   CGO_ENABLED="1"
3. What did you do?
   If possible, provide a recipe for reproducing the error.
   A complete runnable program is good.
   A link on play.golang.org is best.

Using a bootstrap environment built on linux/amd64, built Go from sources on linux/arm64.

1. What did you expect to see?
   ALL TESTS PASSED
2. What did you see instead?

runtime.Breakpoint test times out. all other tests succeed. as minux suggested, here's the gdb backtrace of runtime/testdata/testprog Breakpoint when it hangs:

$ pwd
/home/odroid/go/src/runtime/testdata/testprog
$ go version
go version devel +fc6bcde Fri Mar 18 23:26:36 2016 +0000 linux/arm64
$ gdb ./testprog
GNU gdb (Ubuntu 7.11-0ubuntu1) 7.11
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./testprog...done.
warning: File "/home/odroid/go/src/runtime/runtime-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /home/odroid/go/src/runtime/runtime-gdb.py
line to your configuration file "/home/odroid/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/odroid/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
(gdb) r Breakpoint
Starting program: /home/odroid/go/src/runtime/testdata/testprog/testprog Breakpoint
[New LWP 23014]
[New LWP 23015]
[New LWP 23017]
[New LWP 23016]
[LWP 23011 exited]
^C
Thread 2 "testprog" received signal SIGINT, Interrupt.
[Switching to LWP 23014]
runtime.usleep () at /home/odroid/go/src/runtime/sys_linux_arm64.s:141
141 SVC
(gdb) bt
#0 runtime.usleep () at /home/odroid/go/src/runtime/sys_linux_arm64.s:141
#1 0x00000000000432c8 in runtime.mstart1 ()

at /home/odroid/go/src/runtime/proc.go:1082

#2 0x0000000000000001 in ?? ()

Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[minux]

What is the output of "thread apply all bt" in gdb?

[9nut]

$ gdb ./testprog
GNU gdb (Ubuntu 7.11-0ubuntu1) 7.11
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./testprog...done.
warning: File "/home/odroid/go/src/runtime/runtime-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /home/odroid/go/src/runtime/runtime-gdb.py
line to your configuration file "/home/odroid/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/odroid/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
(gdb) r Breakpoint
Starting program: /home/odroid/go/src/runtime/testdata/testprog/testprog Breakpoint
[New LWP 13288]
[New LWP 13289]
[New LWP 13290]
[New LWP 13291]
[New LWP 13292]
[New LWP 13293]
[LWP 13285 exited]
^C
Thread 2 "testprog" received signal SIGINT, Interrupt.
[Switching to LWP 13288]
runtime.usleep () at /home/odroid/go/src/runtime/sys_linux_arm64.s:141
141 SVC
(gdb) thread apply all bt

Thread 7 (LWP 13293):
#0 runtime.futex () at /home/odroid/go/src/runtime/sys_linux_arm64.s:308
#1 0x000000000002021c in runtime.notesleep (n=0x1616b0 <runtime.sched+16>)
at /home/odroid/go/src/runtime/lock_futex.go:145
#2 0x0000000000044438 in runtime.stopm ()
at /home/odroid/go/src/runtime/proc.go:1519
#3 0x000000000004448c in runtime.stopm ()
at /home/odroid/go/src/runtime/proc.go:1522
#4 0x0000000000000001 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 6 (LWP 13292):
#0 runtime.futex () at /home/odroid/go/src/runtime/sys_linux_arm64.s:308
#1 0x000000000002021c in runtime.notesleep (n=0x1616b0 <runtime.sched+16>)
at /home/odroid/go/src/runtime/lock_futex.go:145
#2 0x0000000000044438 in runtime.stopm ()
at /home/odroid/go/src/runtime/proc.go:1519
#3 0x000000000004448c in runtime.stopm ()
at /home/odroid/go/src/runtime/proc.go:1522
#4 0x0000000000000001 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

---Type to continue, or q to quit---
Thread 5 (LWP 13291):
#0 runtime.futex () at /home/odroid/go/src/runtime/sys_linux_arm64.s:308
#1 0x000000000002021c in runtime.notesleep (n=0x1616b0 <runtime.sched+16>)
at /home/odroid/go/src/runtime/lock_futex.go:145
#2 0x0000000000044438 in runtime.stopm ()
at /home/odroid/go/src/runtime/proc.go:1519
#3 0x000000000004448c in runtime.stopm ()
at /home/odroid/go/src/runtime/proc.go:1522
#4 0x0000000000000001 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 4 (LWP 13290):
#0 runtime.futex () at /home/odroid/go/src/runtime/sys_linux_arm64.s:308
#1 0x000000000002021c in runtime.notesleep (n=0x1616b0 <runtime.sched+16>)
at /home/odroid/go/src/runtime/lock_futex.go:145
#2 0x0000000000044438 in runtime.stopm ()
at /home/odroid/go/src/runtime/proc.go:1519
#3 0x000000000004448c in runtime.stopm ()
at /home/odroid/go/src/runtime/proc.go:1522
#4 0x0000000000000001 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 3 (LWP 13289):
---Type to continue, or q to quit---
#0 runtime.futex () at /home/odroid/go/src/runtime/sys_linux_arm64.s:308
#1 0x000000000002021c in runtime.notesleep (n=0x1616b0 <runtime.sched+16>)
at /home/odroid/go/src/runtime/lock_futex.go:145
#2 0x0000000000044438 in runtime.stopm ()
at /home/odroid/go/src/runtime/proc.go:1519
#3 0x000000000004448c in runtime.stopm ()
at /home/odroid/go/src/runtime/proc.go:1522
#4 0x0000000000000001 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 2 (LWP 13288):
#0 runtime.usleep () at /home/odroid/go/src/runtime/sys_linux_arm64.s:141
#1 0x00000000000432c8 in runtime.mstart1 ()
at /home/odroid/go/src/runtime/proc.go:1082
#2 0x0000000000000001 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

[quentinmit]

@minux Any clues from that backtrace?

[minux]

I didn't see anything special in the backtrace. Somehow the BKPT instruction doesn't trigger a SIGTRAP signal and make program stuck.

[ianlancetaylor]

We have a builder for arm64 GNU/Linux, and it is running the tests, and is not showing any problem.

That said, according to https://gcc.gnu.org/ml/gcc-patches/2013-11/msg02228.html we should be using BRK $0x1000. Which unfortunately does not seem to work with our assembler.

Could you try running this C program with and without an argument and tell us the output and exit status for each case? Thanks.

#include <stdio.h>

int main(int argc, char **argv) {
    if (argc > 1) {
        asm("brk #0x1000");
        printf("continuing after brk #0x1000\n");
    } else {
        asm("brk #0");
        printf("continuing after brk #0\n");
    }
}

[josharian]

FWIW, I can't reproduce this on a pine64 board running linux/arm64.

Using tip @ c83e6f5, linux jessie, GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1. I did:

$ cd src/runtime/testdata/testprog
$ go build
$ gdb ./testprog
// ...
(gdb) r Breakpoint
Program received signal SIGTRAP, Trace/breakpoint trap.
runtime.breakpoint () at /home/debian/go/src/runtime/asm_arm64.s:97
97      BRK

In addition to trying the C program from @ianlancetaylor, you might also try manually sending a SIGTRAP (kill -s SIGTRAP <pid>) when the process is hung inside gdb, just to confirm that that piece of it works.

[jpoirier]

fyi - not able to reproduce it on my Odroid-C2

Using tip @ a65a96, UbuntuMate, GNU gdb (Ubuntu 7.11-0ubuntu1) 7.11

$ gdb ./testprog

This GDB was configured as "aarch64-linux-gnu".

(gdb) r Breakpoint
Starting program: /root/go/src/runtime/testdata/testprog/testprog Breakpoint
[New LWP 3269]
[New LWP 3270]
[New LWP 3272]
[New LWP 3271]

Thread 1 "testprog" received signal SIGTRAP, Trace/breakpoint trap.
runtime.breakpoint () at /root/go/src/runtime/asm_arm64.s:97
97 BRK

[ianlancetaylor]

Thanks for trying.

@9nut I'm going to close this issue until we get more information about your system. See questions above.