runtime: "runtime·unlock: lock count" on Go 1.10.1 when cgo is enabled

[SuperQ]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

Go 1.10.1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

amd64

What did you do?

We have an application that makes use of CGO to collect kernel metrics. When we compile with Go 1.9.5 or with 1.10.1 and CGO_ENABLED=0, we see no crashes. With CGO enabled an Go 1.10.1 we see crashes.

Our test setup is documented here.

What did you expect to see?

No crashes

What did you see instead?

fatal error: runtime·unlock: lock count

[SuperQ]

/CC @ianlancetaylor

[bcmills]

runtime·unlock: lock count is a throw. Where is the rest of the stack trace?

Beyond that, this sounds like memory corruption. You should try to eliminate that as a cause first.

• Are your C dependencies installing signal handlers? If so, are they setting SA_ONSTACK as required by the Go runtime?
• Are you using finalizers in your cgo wrapper? If so, you may be missing some needed calls to runtime.KeepAlive. (I seem to recall that the garbage collector got a little more aggressive in 1.10.)
• Does your program report any races when compiled with -race?
• Does your program report any failures when you compile using -fsanitize=memory (for the C compiler) and -msan (for the Go compiler)?
• Can you construct a smaller reproducer?

[daenney]

@bcmills Unforutunately the rest of the stack trace doesn't seem to get logged. I have no idea why. I ran tests with debug level logging enabled and eventually all I get is that throw (sometimes just a few times, sometimes hundreds of times that same line gets emitted). Aside from the normal logging done by the node exporter itself, part of the stack trace seems to just go missing. I don't know if that's a consequence of running this under systemd but I'm having a hard time seeing how that would cause part of the stack trace to not get captured.

Given that the same code compiles and runs fine (i.e doesn't crash) on Go 1.9.5 but does crash on 1.10.0, 1.10.1 I'm wondering what changes have been made to cgo between those releases that could point us in the right direction? I also confirmed that building under Go 1.10.0, 1.10.1 with CGO_ENABLED=0 results in non-crashing builds.

[daenney]

• I did a go build -race for the node exporter, no errors reported.

• I tried a go build -msan and simply got this:

  # runtime/cgo
  gcc: error: unrecognized argument to -fsanitize= option: 'memory'
  

  $ gcc --version
  gcc (GCC) 7.3.1 20180406
  

  I tried this with both Go 1.9.5 and 1.10.1, same story/error. Afaik fsanitize=memory is only valid for clang and has no equivalent to gcc. According to go env it correctly detected GCC: CC="gcc". I'm guessing the Go tooling gets confused by the fact that /usr/bin/cc is symlinked to /usr/bin/gcc on Arch and decides to call that and assume it's clang essentially passing in the wrong flags? Or it simply defaults to caling cc under the assumption that's clang when -msan is passed? It shouldn't though given it correctly detected CC as gcc and CXX as g++. I would expect it to raise an error if a flag is passed that's invalid for the detected CC and CXX.

• env CC=clang env CXX=clang++ go build -msan resulted in no errors

• Wrt constructing a smaller reproducer, I honestly have no idea where to start. Don't even know if it's in the node exporter itself or one of its dependencies that use a CGO thing. Any pointers on how one could go about tracking this down would be appreciated

• Given it's related to CGO I did a quick search for "C" in the code base, the only possibly relevant ones I can find in node exporter itself are:

  • collector/timex.go
  • collector/loadavg_unix.go though I suspect that doesn't get used in my builds since there also is a collector/loadavg_linux.go. I'm unsure as to what happens in this case, does _linux take precedence over _unix or is this a game of chance?

  I can't find anything immediately obviously wrong with those collectors. All other collectors leveraging C wouldn't/shouldn't have gotten compiled since I did the builds on a Linux system:

  $ ag '"C"'
  
  timex.go
  20:import "C"
  
  cpu_openbsd.go
  30:import "C"
  
  filesystem_bsd.go
  32:import "C"
  
  loadavg_unix.go
  24:import "C"
  
  loadavg_solaris.go
  29:import "C"
  
  netdev_openbsd.go
  32:import "C"
  
  sysctl_bsd.go
  29:import "C"
  
  cpu_dragonfly.go
  82:import "C"
  
  meminfo_openbsd.go
  41:import "C"
  
  meminfo_darwin.go
  19:import "C"
  
  cpu_darwin.go
  46:import "C"
  
  netdev_darwin.go
  33:import "C"
  
  devstat_freebsd.go
  29:import "C"
  
  netdev_bsd.go
  35:import "C"
  
  interrupts_openbsd.go
  95:import "C"
  
  kvm_bsd.go
  26:import "C"
  
  devstat_dragonfly.go
  89:import "C"
  

  I did a similar search in the vendor/ directory. None of them would appear to apply to builds done on a Linux system:

  golang.org/x/sys/unix/mkerrors.sh
  331:	echo 'import "C"'
  
  github.com/lufia/iostat/iostat_darwin.go
  9:import "C"
  
  github.com/godbus/dbus/transport_unixcred_dragonfly.go
  13:import "C"
  
  github.com/godbus/dbus/transport_unixcred_freebsd.go
  15:import "C"
  

[bcmills]

Given it's related to CGO I did a quick search for "C" in the code base,

Maybe try searching for runtime.SetFinalizer? Given that the behavior changed between 1.9 and 1.10, a bad finalizer seems like one of the more likely causes. (I seem to recall that we had to fix several in the Google codebase during the internal 1.10 rollout.)

[daenney]

Thanks. The only call I could find was in logrus:

vendor/github.com/sirupsen/logrus/writer.go
44:	runtime.SetFinalizer(writer, writerFinalizer)

The code is here: https://github.com/sirupsen/logrus/blob/v1.0.4/writer.go#L21-L47. I have no idea if this looks sane or not. I don't see any import statements for C in logrus. The only thing mentioning runtime in Logrus is writer.go, in the import statements and that one call.

[daenney]

Can anyone theorize as to what might be causing the stack trace to just not show up? It's not really helping the debugging efforts and I'm wondering if finding the reason the stack trace doesn't show up might be able to point us in the direction of the bug.

For example: prometheus/node_exporter#870 (comment)

Well, that didn't help.. the notimex variation crashed.

fatal error: runtime·unlock: lock count
fatal error: fatal: morestack on g0

And again, no other useful stack log.

[austin987]

FWIW, two users (myself and one more) have been able to get stacktraces. I've seen crashes a couple dozen times, not sure why this time was special.

My stack trace is here:
https://github.com/prometheus/node_exporter/files/2032750/node_exporter.log

The other is here:
prometheus/node_exporter#870 (comment)