testing: T.Context usability concerns

[quentinmit]

https://groups.google.com/d/topic/golang-dev/rS6psxIf17Q/discussion

Decide on whether to keep/change/remove the API before the 1.8 release.

[bradfitz]

Issue #18182 was another person asking for this last night.

Their motivation: #18182 (comment)

[dsnet]

#16221 is the proposal that added T.Context

[niemeyer]

Copying here the point made in golang-dev, for clarity:

Such a flagging mechanism that by definition will only be useful when things will continue running in the background past the test completion does feel like a wart to me. We should be discouraging people from doing that in the first place, because arbitrary background logic delayed for arbitrary periods of time while other things are being tested is a common source of testing pain, breaking in curious and hard to reproduce ways.

As people pointed out, it's not a mere case of putting logic locally. The context, by definition, will only trigger code to start aborting after it's too late. An aborting yet running goroutine can do whatever it wants in order to attempt to stop cleanly.

In my own testing, services generally have a Stop method because of that. Internally, they leverage a tomb to flag the stopping and block until everything is done cleaning themselves up. Stop also usually returns an error, so one can also check that the stop was itself clean.

[dsnet]

Summary of my thoughts.

We should either:

• Add both (a mechanism to signal for termination AND a mechanism to wait for all to be terminated)
• Not add anything new at all

I understand most uses of Context don't care about waiting for termination, but I believe it is important enough in a significant number of testing purposes.

Some arguments against the status quo (i.e., only adding Context):

• The same argument for adding Context for convenience can also be made for adding some wait mechanism. Adding Context saves 2 lines of code. Adding some mechanism for waiting saves another 2 lines. It's a slippery slope. Where do we stop?
• Adding Context alone makes it more confusing how to add your own wait mechanism since there is no way to insert a post-cancelation wait. That is, the testing framework will only call cancel when the Test function returns, but using your own WaitGroup.Wait will prevent the Test function from even returning.
• You could resolve the above issue by just not using T.Context and roll with your own Context and WaitGroup, but now you have to deal with remembering whether you should be using your own ctx variable that you pass around or T.Context.
• We already have a problem with many tests calling T's method when the Test function returns. The lack of a built-in waiting mechanism encourages people to do this incorrectly. Imagine the following:

  go func() {
      select {
      ...
      case <-t.Context().Done(): // Closed when Test returns
          if err := something.Close(); err != nil {
              t.Errorf("unexpected Close error: %v", err) // This is problematic!
          }
      }
  }()

If we go with adding a wait mechanism, I don't feel strongly whether it's by exposing a WaitGroup, or by adding a Go method as @bcmills suggested in #16221, or something else.

[bcmills]

There is one benefit of adding Context that does not apply to a mechanism for waiting for cleanup: it is difficult to construct a Context which is canceled when the test has failed but the test function is still running (e.g. when t.Failed() transitions to true), but it's fairly easy to defer func() { cancel(); wg.Wait() }().

So a minimal fix might be a Context which is canceled only on failure (and which otherwise explicitly leaks goroutines), or perhaps a Context which is canceled on failure and explicitly panics if its methods are called after the test function has returned. That would address the "finding out when the test has failed" case but leave the "cleaning up the test" boilerplate in user code (where it is today).

[dsnet]

I think canceling the Context upon first failure is inconsistent with how tests work. When t.Error is called, it is normal to continue moving on and perform other tests. If the first t.Error caused the context to be canceled, then that would affect the results of following tests within the same Test function. I'm okay with cancelation occuring when t.Fatal is called, but that is included in the current behavior.

The fact that people have differing expectations of when T.Context gets canceled is an argument against adding it. At least with rolling with your own context, the expectations are clear. So far there have been at least 3 expressed ways people expect it gets canceled:

• (current behavior) Only when the Test ends (either the Test returns or t.Fatal is called)
• Only when the Test fails
• Only when the Test times out (however that is defined)
• Some combination of the above...

[bcmills]

When t.Error is called, it is normal to continue moving on and perform other tests.

Good point. Still, especially given the ability to create subtests I think there are a number of situations in which "cancel on first failure" really is what you want. But perhaps that argues for something more like errgroup, combining "cancel on first failure", "ignore subsequent errors", and "wait for cleanup" into a single API.

[dsnet]

Another data point. There are use-cases where we want to add graceful timeout to tests. The current implementation of t.Context provides no way for the user to manually cancel the context other than returning from the Test function.

[bradfitz]

If you want a timeout, you can make a derived context from t.Context(). Again, this isn't supposed to be the end-all Context for all imaginable use cases. It's supposed to be a convenience thing.

And having a context available would've helped a CL I reviewed yesterday (https://go-review.googlesource.com/c/33972/1/http2/transport_test.go) if the context were available. But it was a few lines of code without t.Context, so we survived.

But obvious you dislike it, so feel free to send a CL removing it and the documentation from go1.8.html.

Perfect is the enemy of good.

[dsnet]

I'm uploading a CL to revert this change. There are a fair number of people who have reservations about this.

The main argument for adding t.Context seems to be for convenience (saving ~2 lines) and for consistency with the rest of standard library (which is moving towards adding Context where it makes sense). However, the benefit of convenience and consistency is offset by increased risk of users calling t.Error after the Test has returned. Overall, I don't believe a 2-line convenience is worth the real risk of improper t.Error usage (which is already a fairly common problem in tests).

To be fair, I originally voted in favor of this change in #16221, but changed my position on this after trying to use it on several unit tests.

I am personally still in support adding t.Context in conjunction with a wait mechanism as this will counter the current detriment of adding t.Context only. However, I don't believe that we should rush on more API design and we can revisit this for Go 1.9.