proposal: errors: add With(err, other error) error

[natefinch]

Background

Right now, the only way to wrap an error in the standard library is with fmt.Errorf which means that all you can do to an error is add text to its .Error() output.

When you receive an error from a function and want to return it up the stack with additional information, you have 3 options

1. you can return some other error, which loses the original error's context
2. you can wrap the error with fmt.Errorf, which just adds textual output, and therefore isn't something callers can programmatically check for
3. you can write a complicated error wrapping struct that contains the metadata you want to check for, and behaves correctly for errors.Is, .As, and .Unwrap to allow callers to access the underlying cause of the error.

Proposal

This proposal offers a much easier way to do number 3, by implementing a new function in the stdlib errors package that supports wrapping any error with any other error, such that they are both discoverable with errors.Is and errors.As.

// With returns an error that adds flag to the list of errors that can be
// discovered by errors.Is and errors.As.
func With(err, flag error) error {
	if err == nil {
		return nil
	}
	if flag == nil {
		return err
	}

	return flagged{error: err, flag: flag}
}

type flagged struct {
	flag error
	error
}

func (f flagged) Is(target error) bool {
	return errors.Is(f.flag, target)
}

func (f flagged) As(target interface{}) bool {
	return errors.As(f.flag, target)
}

func (f flagged) Unwrap() error {
	return f.error
}

This means that:
errors.With(err, flag).Error() returns err.Error()
errors.Unwrap(With(err, flag)) returns err
errors.Is(With(err, flag), target) is the same as calling errors.Is(flag, target) || errors.Is(err, target)
errors.As(With(err, flag), target) is the same as calling errors.As(flag, target) || errors.As(err, target)

Why This Approach?

By reusing the existing error wrapping pattern, we make the smallest possible change to the standard library, while allowing the broadest applicability of the functionality. We introduce no new interfaces or concepts. The function is general enough to support many different use cases, and yet, its use would be invisible to anyone who is not interested in using error wrapping themselves. It imposes no added burden on third parties that check errors (beyond what is already standard with fmt.Errorf wrapping), and can be ignored by authors producing errors, if that is their wish.

Use Cases

The main use case for this function is incredibly common, and I've seen it in basically every go application I've ever written. You have a package that returns a domain-specific error, like a postgres driver returning pq.ErrNoRows. You want to pass that error up the stack to maintain the context of the original error, but you don't want your callers to have to know about postgres errors in order to know how to deal with this error from your storage layer. With the proposed With function, you can add metadata via a well-known error type so that the error your function returns can be checked consistently, regardless of the underlying implementation.

This kind of error is often called a Sentinel error. fs.ErrNotExist is a good example of a sentinel error that wraps the platform-specific syscall error.

// SetUserName sets the name of the user with the given id. This method returns 
// flags.NotFound if the user isn't found or flags.Conflict if a user with that
// name already exists. 
func (st *Storage) SetUserName(id uuid.UUID, name string) error {
    err := st.db.SetUser(id, "name="+name)
    if errors.Is(err, pq.ErrNoRows) {
       return nil, errors.With(err, flags.NotFound)
    }
    var pqErr *pq.Error
    if errors.As(err, &pqErr) && pqErr.Constraint == "unique_user_name" {
        return errors.With(err, flags.Conflict)
    }
    if err != nil {
       // some other unknown error
       return fmt.Errorf("error setting name on user with id %v: %w", err) 
    }
    return nil
}

This keeps the error categorization very near to the code that produces the error. Nobody outside of SetUserName needs to know anything about postgres driver errors.

Now in the API layer, you can translate this error to an HTTP error code trivially:

func (h *Handlers) HandleSetName(w http.ResponseWriter, r *http.Request) {
    name, id := getNameAndID(r)
    err := h.storage.SetUserName(id, name)
    if err != nil {
        handleError(err, w)
        return
    }
    // other stuff
}

func handleError(err error, w http.ResponseWriter) {
    switch {
    case errors.Is(err, flags.NotFound):
        http.Error(w, 404, "not found")
    case errors.Is(err, flags.Conflict):
        http.Error(w, 409, "conflict")
    default:
        // other, uncategorized error
        http.Error(w, 500, "internal server error")
        // probably log it, too
    }
}

This code doesn't know anything about postgres. It uses the standard errors.Is to check for errors it knows about. But if it then decides to log that error, it has full access to the original error's full context if it wants to dig into it.

This code is very insulated from any implementation changes to the storage layer, so long as it maintains its API contract by continuing to categorize the errors with the same error flags using errors.With.

What This Code Replaces

Without code like this, the handleError functions above would have to do something like this itself:

    var pqErr *pq.Error
    if errors.As(err, &pqErr) && pqErr.Constraint == "unique_user_name" {
        http.Error(w, 409, "conflict")
        return
    }

Not only is that super gross to be doing in the API layer, but it would silently break if someone decided to change database or even just the name of the constraint, and didn't realize that this far away method was digging deep into that error. I've seen and written this kind of code many times in my career, and it feels really wrong, but without stdlib support, it's hard to know what the alternative is.

Production Experience

I used this kind of error wrapping at Mattel for a long time, and now I'm introducing it to GitHub's codebase. It VASTLY improved our error handling in both cases, with very little cognitive or processing overhead. It has made it much easier to understand what an error return really means. It has made writing correctly-behaving API code much simpler by breaking the coupling between two disparate parts of the system.

Community Examples

• https://pkg.go.dev/go.uber.org/multierr
• https://pkg.go.dev/github.com/hashicorp/go-multierror
• https://pkg.go.dev/github.com/go-openapi/errors#CompositeError
• https://pkg.go.dev/v2ray.com/core/common/errors#Combine
• https://pkg.go.dev/github.com/hashicorp/errwrap#Wrap
• https://pkg.go.dev/github.com/juju/errors#Wrap
• https://pkg.go.dev/github.com/cosmos/cosmos-sdk/types/errors#Wrap
• https://pkg.go.dev/github.com/cockroachdb/errors

Conclusion

Adding this method to the standard library would encourage the use of this pattern. Packages using this pattern have better-defined and more stable contracts about how callers can programmatically respond to different categories of errors, without having to worry about implementation details. This is especially true of application code, where error categorization has traditionally been difficult.

Special thanks to @rogpeppe for his suggested improvement to my original error flags idea.

[ianlancetaylor]

This has some similarities to #48831.

[seankhliao]

Note we've declined multiple errors in #47811 and error marks (which looks like the same as this one?) #50819

This seems like it would make discoverability slightly worse as the extra error matching is shifted to point of construction vs having the error type implement errors.Is

[natefinch]

I understand that the bar is high to put something new in the standard library, but I think the fact that there are at least 3 prior proposals about this (and probably more, I know I've seen multierror support requested a few times) speaks to how many people need this behavior.

The comment that it needs to exist in an external library seems to be satisfied by these:

• https://pkg.go.dev/go.uber.org/multierr
• https://pkg.go.dev/github.com/hashicorp/go-multierror
• https://pkg.go.dev/github.com/go-openapi/errors#CompositeError
• https://pkg.go.dev/v2ray.com/core/common/errors#Combine
• https://pkg.go.dev/github.com/hashicorp/errwrap#Wrap
• https://pkg.go.dev/github.com/juju/errors#Wrap
• https://pkg.go.dev/github.com/cosmos/cosmos-sdk/types/errors#Wrap

As for why it needs to be in the standard library, I think because it is not an obvious or easy function to create, and it's hard to write correctly. Having a function like this in the errors package gives everyone an answer to "how do I decorate an existing error with more metadata without losing the original error information?" And the answer is right there next to the rest of the error handling code. Right now, there's no real answer to that.

Just as we have put error wrapping code into the stdlib to be able to decorate an error's message without losing the original error, adding this function could make it so that we can decorate an error with more than just more text. This will help avoid leaking implementation details and distant coupling that can happen when your only options are "add some text to the existing error" or "throw away that error and return your own".

This seems like it would make discoverability slightly worse as the extra error matching is shifted to point of construction vs having the error type implement errors.Is

I'm not sure this means. Discoverability is worse than what? The error returned does implement errors.Is. And adding metadata at the point where an error is first encountered in your code is the best place to do that. That's where you have the most context about what an error means and what it represents.

[tv42]

As for why it needs to be in the standard library, I think because it is not an obvious or easy function to create, and it's hard to write correctly.

Didn't you just disprove this with https://go.dev/play/p/usV6mFRjxxb? 30 lines of very straight forward implementation.

I'd like to point out that nothing that consumes this needs to know how such an error was constructed; the consumer doesn't even know if you used an stdlib implementation or 3rd party one.

As far as I know, the listed 3rd party projects largely predate errors.Is/errors.As and have dragged their ugly old APIs around, or are doing something else that's much uglier than the examples in this proposal.

I really do appreciate that this proposal explicitly does not attempt to display multiple errors. This is more like decorating the error with more data for programmatic access, which if not used is ignored in the final display. I find that much saner.

[rittneje]

you can write a complicated error wrapping struct that contains the metadata you want to check for, and behaves correctly for errors.Is, .As, and .Unwrap to allow callers to access the underlying cause of the error.

FWIW you don't have to do anything special for Is or As. You just need to implement Unwrap. For example:

type NotFoundError struct {
    Err error
}

func (e NotFoundError) Error() string {
    return e.Err.Error()
}

func (e NotFoundError) Unwrap() error {
    return e.Err
}

Then the client can do:

var nfe NotFoundError
if errors.As(err, &nfe) {
    ...
}

The only reason a more complicated implementation is needed is if you specifically try to return another sentinel error.

[natefinch]

Didn't you just disprove this with https://go.dev/play/p/usV6mFRjxxb? 30 lines of very straight forward implementation.

30 lines of code that had two subtle bugs in it that I missed even after writing the initial tests.... and I've written almost this exact library before. The first time I wrote it, it was a couple days of work to wrap (heh) my head around errors.Is and errors.As and what the interfaces really expected. And even then, since I was new to the idea, I came up with something much more complicated than this.

One of the major benefits of having this in the standard library is that it makes the functionality discoverable. By having it in the standard library, we're saying "hey, this is a thing you might want to do". We give devs a standard and easy way to let people programmatically check their errors, without having to throw away the underlying implementation errors.

[hampgoodwin]

While I agree with the argument "it's simple to do", I would suggest we're overestimating engineers. I would prefer everyone be aware of a convention like in the aforementioned counter arguments, but I do not think it's realistic and what follows is lower quality error handling and discovery than what could exist with this change.

If we intend to have ubiquitous adoption of the aforementioned convention, I would argue it makes sense to include in the stdlib?

Thank you for the proposal and looking forward to the discussion!

[ChrisHines]

I have some support for this proposal, but also some concerns and at least one design question.

In support

I can see the utility of something like this. I believe there is a real challenge to maintain the discipline to raise the abstraction level of errors as they are returned up the call stack while at the same time not obliterating the useful debugging information contained in the original lower level errors. In particular, API services I've worked on—which have typically used github.com/go-kit/kit packages and design patterns—often have error handling complexities that fit the use cases described in this proposal. Here is some code taken from one of the Go kit examples, it is akin to the handleError function shown above.

https://github.com/go-kit/examples/blob/e18c82edc2f9c7de3aff9748751d63270c052c1b/shipping/booking/transport.go#L181-L195

// encode errors from business-logic
func encodeError(_ context.Context, err error, w http.ResponseWriter) {
	w.Header().Set("Content-Type", "application/json; charset=utf-8")
	switch err {
	case cargo.ErrUnknown:
		w.WriteHeader(http.StatusNotFound)
	case ErrInvalidArgument:
		w.WriteHeader(http.StatusBadRequest)
	default:
		w.WriteHeader(http.StatusInternalServerError)
	}
	json.NewEncoder(w).Encode(map[string]interface{}{
		"error": err.Error(),
	})
}

That code predates errors.Is, but a newly written one would likely use it or maybe errors.As. Go kit has an FAQ about error encoding that hints at some of the dilemmas involved. What I always found unsettling was that despite the nice degree of decoupling that a typical Go kit application has between the various pieces, these error handling functions were the most likely to have the coupling skeletons hidden in them.

I believe a well known way to transform a system-domain error into a business-domain error without losing the ability to log the original system-domain details for triaging and debugging purposes would add value. I believe that's what this proposal is aiming for.

Concerns

I am not confident the approach taken here is general enough for the standard library. #52607 (comment) references several existing packages providing some form of composite errors. Are we seeing convergence on a common approach across those implementations? I've always been reluctant to use any of those that I've researched in the past because some aspect of their semantics wasn't quite what I needed at the time.

Design Questions

A portion of the errors.As docs for reference:

// As finds the first error in err's chain that matches target, and if one is found, sets
// target to that error value and returns true. Otherwise, it returns false.
...
// An error matches target if the error's concrete value is assignable to the value
// pointed to by target, or if the error has a method As(interface{}) bool such that
// As(target) returns true. In the latter case, the As method is responsible for
// setting target.
...
func As(err error, target any) bool

Restating this proposal:

// With returns an error that represents the union of the two errors. 
// It will report true from errors.Is and errors.As if calling
// that function would return true on either err or other. 
// Calling errors.Unwrap will return err.
func With(err, other error) error

The docs for With do not address which of the two errors that As sets target to when it returns true. The choice becomes ambiguous if both err and other are assignable to the value pointed to by target. The only hint is the last sentence which says that Unwrap returns err which suggests that perhaps err is given priority over other in general and one might guess that As would follow suit. But the linked implementation prioritizes other in its As method:

func (u union) As(target any) bool {
	if errors.As(u.other, target) {
		return true
	}
	return errors.As(u.error, target)
}

What is the rationale for Unwrap and As to behave "differently" here? I also wonder how weird this might get with multiple layers of With wrappings.

[natefinch]

existing packages providing some form of composite errors. Are we seeing convergence on a common approach across those implementations?

Not across those implementations, but also those are mostly older than error wrapping, which was introduced only 2.5 years ago. Since this is often application-specific code, wrapping may often be done as a one-off library inside application code, which can be hard to find and in many cases is in private repos (like the code I've written at Mattel and GitHub).

What is the rationale for Unwrap and As to behave "differently" here?

Honestly, almost no one is going to call Unwrap directly. The Unwrap interface mostly exists to support Is and As. Is and As, because there is a notion of failing, can effectively operate on both errors, checking one and falling back to check the other value. But for Unwrap, there's no way to fall back (as far as I can tell). There's no way to say "ok, unwrap until you get to the bottom of u.other and then start unwrapping u.error".

For Is and As, I decided to check the other error first, because it is newest in the stack and also most likely to just be a single sentinel error, avoiding the need to repeatedly unwrap. So it's both logically the best choice and likely less work than unwrapping the base error, which may well be several layers of wrapped errors.

For Unwrap, I had to choose something, and the idea is that u.error is more representative of the true underlying cause than the added context in u.other.

I also wonder how weird this might get with multiple layers of With wrappings.

That should actually work just fine. Each errors.With effectively adds one error to the top of the stack. Is and As would detect any errors in any branches of the tree.

[darkfeline]

Is the actual problem here that you want error unwrapping that does not escape to external clients? If so, this feels like a roundabout way of addressing it.

[jimmyfrasche]

So this is fmt.Errorf("msg: %w", err) but

• msg can be a custom error type
• err does not contribute to the returned error's .Error()

Is that correct?

There's no way to say "ok, unwrap until you get to the bottom of u.other and then start unwrapping u.error".

Challenge accepted: https://go.dev/play/p/be3AVWRAoTO

[natefinch]

What I want is to solve this:

func (l *Loader) LoadConfig(name string) (*Config, error) {
   b, err := os.ReadFile(filepath.Join(l.configDir, name))
   if os.IsNotExist(err) {
       return nil, fmt.Errorf("can't find config: %w", err)
   }
   if err != nil {
       // I don't know, something went wrong.
       return nil, fmt.Errorf("error opening config: %w", err) 
   }
   cfg, err := parseConfigFile(b)
   if err != nil {
       // malformed config
       return nil, fmt.Errorf("malformed config: %w", err)
   }
   return cfg, nil
}

If you're calling LoadConfig, how do you differentiate programmatically between "config not found" and other errors, so that you can handle that error differently?

Without errors.With, you have two choices....

1.) look for the string "can't find config" in err.Error() (gross)
2.) call os.IsNotExist(err) on the returned error. (also gross, but less obviously so)

Most people do #2. The problem is that calling os.IsNotExist is causing your code to rely on the hidden implementation details of LoadConfig. That method intentionally hides how it stores configs. If it changes so it's getting the config from a database instead of off of disk, os.IsNotExist will stop detecting the not found condition, and your code that is supposed to do something in that case won't trigger.

If you instead write this code with errors.With, you can wrap the error with a sentinel error in the not found branch. Then it is detectable by callers, without losing the possibly valuable information in the underlying error that would happen if you just returned the sentinel error instead:

var ErrConfigNotFound = errors.New("config not found")
var ErrConfigNotValid = errors.New("config not valid")

func (l *Loader) LoadConfig(name string) (*Config, error) {
   b, err := os.ReadFile(filepath.Join(l.configDir, name))
   if os.IsNotExist(err) {
       return nil, errors.With(err, ErrConfigNotFound)
   }
   if err != nil {
       // I don't know, something went wrong.
       return nil, fmt.Errorf("error opening config: %w", err) 
   }
   cfg, err := parseConfigFile(b)
   if err != nil {
       // malformed config
       return nil, errors.With(err, ErrConfigNotValid)
   }
   return cfg, nil
}

Now when you want to check for when the config is not found, you just do

cfg, err := l.LoadConfig(name)
if errors.Is(err, loader.ErrConfigNotFound) {
    // handle missing config
}
if errors.Is(err, loader.ErrConfigNotValid) {
    // handle invalid config
}
if err != nil {
    // handle unknown error
}