os: on unix Process.Kill() can kill the wrong process

[ibukanov]

This is somewhat similar to the issue #9382. On Unix Process.Kill() just calls Process.Signal(Kill). As https://golang.org/src/os/exec_unix.go#L39 indicates, the Signal function invokes syscall.Kill(p.Pid) outside any lock after checking if the process still runs. Thus at the point when the signal is called the process could finish and Process.Wait() from another thread return. Thus OS is free to reuse the pid for another unrelated process. If this happens, Process.Kill() kills that process.

Due to this race it is impossible to write a correct platform-independent code in Go that kills the process if it does not terminate after a pause. I.e. the code fragment from #9382 is not correct on Unix:

func spawnAndKill(exePath string, counter int) error {
    cmd := exec.Command(exePath, fmt.Sprintf("%d", counter))
    err := cmd.Start()
    if err != nil {
        return err
    }
    go func() {
        time.Sleep(1000 * time.Millisecond)
        cmd.Process.Kill()
    }()
    cmd.Wait()
    return nil
}

[minux]

The spwanAndKill func is racy. The cmd.Process.Kill() races with cmd.Wait. I think this is working as intended, there is no way for Go to prevent this race (say, we make Kill check if the waitpid has returned, but that is still racy, as waitpid could finish anytime after the check and before the kill syscall). It's the programmer's responsbility to eliminate the race. If you want to kill the process after a timeout, just wait for the timeout, kill it unconditionally, and then Wait.

[ibukanov]

@minux On Windows the above code is not racy and works correctly due to fixes that went into #9382. On Linux I can also implement the same semantics correctly via waiting for SIGCHLD and using non-blocking wait4 syscall. That allows to synchronize with kill attempts. However, this is non-portable.

What I would like to see is an option to write a portable Go code implementing the kill-if-not-finished-after-timeout pattern. If this requires new API, so be it, but at least document loudly that calling Process.Kill and Process.Wait from different threads at the same time could race and should be avoided in portable code.

[minux]

There is a portable way to implement the functionality. Just wait for the timeout, kill the process unconditionally and then wait. The suggested workaround for unix is too complicated (not to mention it needs one goroutine to pull with non- blocking wait4) when there is easy high-level workaround.

[ianlancetaylor]

I think we can do it without a race, and without an extra goroutine, if we add something to the runtime package to count SIGCHLD signals, and provide a way for a goroutine to retrieve the current SIGCHLD count and a way to sleep until the SIGCHLD counter is greater than N.

[ibukanov]

@minux The point is precisely not to wait to timeout as typically the process finishes much faster. In my case the timeout is really for runaway cases when something bad happens or the system is too busy. I also cannot just start another go routine that does kill/wait as I need to get the process return status to decide if everything is OK.

@ianlancetaylor Note I do not suggest to change the current implementation. Rather I suggest to document loudly that one should avoid Wait()/Kill() race in portable code. Then add some new API that allows to implement the desired functionality. For example, a version of exec.Cmd.Wait that takes a channel and waits until the child finishes or there is input on the channel is enough to implement all my use cases.

[ianlancetaylor]

If we can change the existing API so that it avoids a race, I think that is clearly better than introducing a new API.

[minux]

I still think the user code should be able to avoid this race
as much as possible.

For example, something like this should be pretty safe
without any complications in runtime and it also have
the benefit that as soon as the process exits, the timeout
goroutine will exit soon, rather than wait until the timer
expires.

func spawnAndKill(exePath string, counter int) error {
cmd := exec.Command(exePath, fmt.Sprintf("%d", counter))
err := cmd.Start()
if err != nil {
return err
}
ch := make(chan bool)
go func() {
select {
case <-time.After(1000 * time.Millisecond):
cmd.Process.Kill()
case <-ch:
return
}
}()
cmd.Wait()
close(ch)
return nil
}

[ibukanov]

The above code just minimizes the chances of race. It is still there. As it is impossible to make its probability arbitrary small, one should avoid it, say, in production code.

[gopherbot]

CL https://golang.org/cl/23967 mentions this issue.

[ianlancetaylor]

I believe this race is now fixed on GNU/Linux.

Leaving the issue open as the fix will have to be implemented on other Unix systems. Systems that provide waitid (which should be most of them, since it is in POSIX.1-2008) need to implement that; it's slightly painful since it's not in the syscall package.