Improvements to error messages and their infrastructure

proposals/000-error-messages.md

@@ -0,0 +1,181 @@
+# Coordinate Improvements to Error Messages
+
+## Introduction
+
+This proposal invites the Haskell Foundation to coordinate improvements
+to the error messages users encounter while working with Haskell. It
+outlines several different ways in which error messages are challenging
+today, and suggests ways forward. The proposal seeks technical coordination
+and a driving force behind making these improvements to the ecosystem.
+
+## Background
+
+An oft-repeated challenge in learning Haskell is the quality of the error messages, along at least these dimensions:
+
+1. An error message may have an unknown provenance: is it generated by stack? by cabal? by GHC? by haddock? Even for experts, this is a real challenge sometimes. (My example: trying to compile a program using a GHC plugin. I think the message I grappled with was from haddock in the end, but it was produced by the cabal executable.)
+2. An error message may mention concepts unfamiliar to the programmer and unrelated to the problem. (Example: if I write `myFirstAttemptAtMaybe :: Just Int; myFirstAttemptAtMaybe = Just 5`, you probably don't want to enable `-XDataKinds`.)
+3. An error message may mention concepts unfamiliar to the programmer and related to the problem. (Example: if I write `not :: a -> a; not True = False`, then I see something about "rigid type variables". This information *is* relevant, but it's still unhelpful if the user doesn't know about rigid type variables.)
+4. Error messages may contain too much information. (Example: given that the erroneous line is now repeated under the error message itself, the "In the expression" lines may be redundant.)
+5. Error messages may contain too little information. (Example: It is possible to get GHC to say that `Proxy` does not match `Proxy`, because the invisible argument to `Proxy` is different. GHC usually suggests `-fprint-explicit-kinds` in these scenarios, but not always.)
+6. Error messages may say too little about what caused the error to occur. (Examples: 1. If I `import M` where `M` exports `not` and then use `not` in my code, I'll get an ambiguity error between `Prelude.not` and `M.not`. Understanding this error requires knowing that the `Prelude` is imported implicitly. 2. GHC might report a missing instances, but only after simplifying a goal via several available (and perhaps wrong) instances. By the time the error is reported, the programmer has to re-create GHC's thinking process, which can be challenging.)
+7. Error messages do not integrate well with tooling. (Example: many error messages suggest extensions to enable, but IDEs currently have to parse strings in order to find these suggestions, which are not at all standardized across error messages.)
+
+## Motivation
+
+While I do not have well-curated data to support the notion, it would seem the problems above hinder adoption. People malign error messages in Haskell routinely. This proposal charts a course to a future where this
+would happen less. Presumably, Haskellers would be happier and more productive, thus amplifying Haskell's
+impact on humanity.
+
+A slightly less direct benefit is that taking on this project at all signals to the world that central
+resources in the Haskell community are being invested in making the language more welcoming (via improved
+communication from tooling to users). This can serve as a counterpoint to claims in various fora that
+Haskell is unwelcoming.
+
+
+## Goals
+
+This proposal will have been successful when we see spontaneous posts online from members of our community
+describing how much better error messages have become and claiming that Haskell is an exemplar in good,
+clear communication between tooling and users.
+
+On a more technical level, full achievement of the goals of this proposal would entail all the following:
+ - Every time a central Haskell tool goes wrong, the user gets a message telling them what component went wrong and why.
+ - These error messages contain a way to get more information, for example, by having a searchable error code or link to a web page with more information.
+ - More complex error messages support interactivity, where users can query for more information about certain details. This allows users to focus on the information relevant to them.
+ - All technical jargon (as well as we can define "technical jargon") is defined in an easily accessible glossary of terms.
+
+But these technical achievements are not the goal: community happiness is, as imperfectly measured by
+the first goal, above.
+
+## Proposed Tasks
+
+This section includes more detail about concrete directions for achieving the goals above.
+
+### The Technical Solution: Add More Structure
+
+In order to fix some of the problems above, I've been on a multi-year campaign (aided critically by @alpmestan and @adinapoli) to add more structure to error messages, along two main axes.
+
+#### Using error datatypes
+
+This was first written up as https://github.com/ghc-proposals/ghc-proposals/pull/306, is mostly implemented as described by https://gitlab.haskell.org/ghc/ghc/-/wikis/Errors-as-(structured)-values, and is the subject of a recent blog post at https://well-typed.com/blog/2021/08/the-new-ghc-diagnostic-infrastructure/.
+
+The central idea is that, instead of GHC producing (essentially) a string to describe an error message, it should use a data constructor to state the nature of the error and store any auxiliary information necessary to print the error. Then, separately, GHC should render the error into a (structured) string.
+
+After much work, this is now possible, as described in the pages linked above. Not all error messages have been ported over to the new architecture yet. But for messages that have been, tools no longer need to parse messages to know what they say, and it's possible to, say, enumerate all the possible messages that GHC could produce. With the extra structure here, it's now feasible to create, say, a wiki or other website that explicates each error message. This way, when a user hits an error, they are linked to a page with more information, and quite possibly tips from others who have been there before. Relatedly, we could assign error codes to messages (https://github.com/ghc-proposals/ghc-proposals/pull/325) to make this all even easier.
+
+This solution addresses problems 3 and 7. We see above how this will integrate better with tooling. This step also addresses 3 (to some degree) by allowing for a simple process for making web pages related to specific errors.
+
+#### Using an inspectable document type
+
+Today, GHC produces pretty-printed strings ("ppstrings") for errors. A ppstring is just like a string, but with a bit of structure to indicate how lines should be wrapped and/or indented. A ppstring is rendered into a string using a few settings, including the desired line width of the output. However, a ppstring's internal structure is all about presentation, not about semantic content.
+
+The next technical part of this effort is adding structure to the error message texts. This is https://github.com/ghc-proposals/ghc-proposals/pull/307. The additional structure would allow us to embed, say, a `Type` into some error text, in a way that remembers it's a `Type`. This would allow for an IDE to support clickable error messages, where the user could click on the type mentioned in an error and, say, show its kinds explicitly. Or jump to its definition. Or show how GHC decided some expression had that type. The sky is the limit here -- but, critically, we need our error texts to remember what parts of the text are types (or expressions, or language extensions, or other goodies).
+
+The conversation on the [GHC proposal](https://github.com/ghc-proposals/ghc-proposals/pull/307) linked above
+is interesting. It points out that an inspectable document type mixes content and presentation, which is usually
+a bad idea. This is accurate, but I still think doing so would be useful: it allows tooling to rely on GHC to
+render some parts of error messages while allowing custom processing for other parts. Or maybe this really is
+a bad idea, overall -- I think the best way to find out is to work with potential clients of this facility to
+work out what the benefits would be.
+
+There is some design work here, but no implementation work that I'm aware of.
+
+This solution addresses problems 4, 5, and 6, by creating the possibility of fine-grained user control over the level of detail in error messages.
+
+### The Social Solution: design better error messages
+
+The above technical solutions are simply about installing plumbing. The real work is around crafting error messages that make more sense to users. While not a social problem, crafting better error messages seems to require a social solution, in that we absolutely need a diversity of voices working together in order to craft the best messages. In particular, implementors are very poorly poised to write error messages that will be helpful to newcomers, because of the great chasm between them in the amount of information they respectively know.
+
+The current attempt to address this problem is at https://github.com/haskell/error-messages. This repo, the brainchild of @ketzacoatl and me, is meant to be a clearinghouse of ideas around improving individual error messages, by writing better text. This work will be enhanced by the technical solutions above, but there are great strides we can make without rewriting parts of GHC.
+
+This solution is mostly about problem 2, but good solutions to all the problems will require careful collaboration across our user community.
+
+### The Ecosystem Solution: identifying the speaker
+
+As we improve the error messages, it would be good to come to a community standard around being able to track where a message comes from -- and even having a uniform way of presenting errors to the user. Currently, GHC has a particular style to its errors, listing the filename, line, and column number, then writing out the message, and lastly including the error-producing line with a certain span highlighted. Should other tools adopt the same style? Should GHC adopt other tools' style? Maybe, if we're integrating with IDEs, this doesn't matter -- but then, the question becomes how different parts of our tooling interact with IDEs. The more we standardize, the easier it will be to flexibly work with a variety of tools.
+
+As we standardize, we should also have some standard system for identifying the provenance of an error message. If we use error codes, for example, maybe the first character of the code could define the generating tool. e.g. `G` for GHC, `C` for cabal, `S` for stack, `H` for haddock, `L` for hlint, and perhaps others. Given its position in our community, the HF would police these prefixes and allocate them for the common good. One advantage to having clear error message provenance is that, if there is a bug or infelicity in a message, users will know where to post! This invites users into our ecosystem, and turns users into contributors.
+
+This addresses problem 1.
+
+### What we need
+
+While volunteers in the community are providing some of this already, I am proposing for the
+HF to coordinate and promote these efforts, along the following lines:
+
+1. The error-messages repo (https://github.com/haskell/error-messages) is having some good conversations, but there would ideally be a much higher level of participation. In addition, making this repo work to produce good messages will require thoughtful care and someone leading conversations to conclusions. Currently, @ketzacoatl and I have been serving this need. According to this proposal, the HF would arrange for someone to give consistent love
+to this repo, moving conversations toward conclusion and inviting broad attention from the community. There is
+not a specific task to complete here, but instead just steady attention.
+2. Relatedly, once a new message is decided upon in the error-messages repo, someone has to implement the change in GHC (or other tool). Sometimes, the work is just around wording, and is suitable for a GHC newcomer. The HF
+would push this work forward, finding a suitable implementor.
+3. The error-datatypes work in GHC has already created the new infrastructure for datatype-based error messages. But we still need to have volunteers convert the many messages scattered throughout GHC to use this new structure. This is harder work and requires some familiarity with GHC. The HF would source and coordinate the volunteers and track
+progress.
+4. The error-datatypes work has already created lots and lots of error-message constructors. Not all of these are well documented or exemplified. For example, see https://gitlab.haskell.org/ghc/ghc/-/blob/master/compiler/GHC/Parser/Errors/Types.hs, which contains a few constructors with lovely, detailed documentation and examples, and many, many more that lack these niceties. A small army of volunteers could fix this! This work does not require nearly as much familiarity with GHC (really, you'd just need to build it and operate the testsuite). The HF would
+source and coordinate the volunteers and track progress.
+5. Structured error text would be a major step forward, but it would need someone dedicated to designing a great system and, likely, implementing it. The first step for the HF would be to facilitate an exploratory conversation
+among stakeholders to figure out whether this (= structured error text) is a good idea or a bad one. (The idea has been floating within GHC
+for many years, and it powers the interactivity behind Idris's error messages, so it would seem to have *some*
+merit.) Then, if the design exploration agrees that this is worth pursuing, the HF would either find a volunteer (or small team of volunteers) to complete this or would consider
+paying to complete this task, given the expertise required to do it well and the difficulty of breaking it
+down.
+6. Now that we have lots of error-message constructors, we can start assigning ID codes to them and then creating web pages that describe each one. (This is related to volunteer opportunity 4, documenting the constructors.) Once we have structured error text, we can even imagine having special glossary-item components to messages, where users could click on terminology to get linked to a page explaining the term. (Examples: "rigid", "infinite type", "superclass", etc.) Setting up a space where this content could be hosted and kept up-to-date would be a great job for a volunteer, as would curating the site generally. The HF could establish this website, perform quality
+control, control the error-code namespace, and source and coordinate volunteers to do the writing.
+
+While there's synergy among these different opportunities, they really are (I believe) mostly independent, and we could do all of them at once.
+
+## People
+
+This section should detail the following people:
+
+-   **Performers:** This proposal seeks a Haskell-Foundation-supported (not necessarily supported by
+    money) individual to push this project forward. There will also likely be a small army of volunteers
+    carrying out individual tasks.
+
+-   **Reviewers:** I (Richard Eisenberg) am happy to review progress. More importantly, I would hope we
+    can recruit community volunteers at various stages of Haskell expertise to assess our progress. Are the
+    improvements making them happy?
+
+-   **Stakeholders:** This really affects everyone. Implementors would include people working on core Haskell
+    tooling, including (but not limited to) GHC, Cabal, Stack, HLS, ghcup, Haddock, and Hackage.
+
+## Resources
+
+This is hard to estimate. Coordination probably requires 5 hours per week. Beyond that, it would take a lot
+of volunteers to do this well. One particular piece that might be worth addressing specifically is to support
+someone (likely with money) to design and build a structured error message type. This will be hard, expert-level
+work.
+
+## Timeline
+
+This really depends on how many volunteers we can muster. I would hope for concrete improvements to be
+available within 4-6 months, and perhaps completion of the project in a year.
+
+## Lifecycle:
+
+I don't think this really applies here. There would be a warm-up period at the beginning where the goal
+is to source volunteers, but afterwards, it's all about keeping people moving forwards.
+
+## Deliverables
+
+1. Error messages with error codes, merged into our major tools.
+1. A website where users can look up error codes and learn more about error messages.
+1. A vibrant place where the community participates in crafting excellent error messages. ("Vibrant"
+is a poor metric, but we could be more quantitative if we like: 50+ different users per year might
+be such a metric.)
+1. Support within HLS for navigating an interactive error message -- for example, pointing at a type
+   mentioned in an error message and getting its kind.
+1. A blog post on the HF blog about this project and how it has helped improve things. We want to take
+   credit for facilitating this work, and to highlight the contributions of all the volunteers.
+
+## Outcomes
+
+1. Happier community around error messages.
+1. An extensible infrastructure for creating new error messages of similar quality.
+
+## Risks
+
+There is always a risk around opportunity cost (that our time might be better spent elsewhere), but
+I see few other risks in this project. It is incremental, so even if we cannot finish, the parts that
+are complete should still benefit our ecosystem. I suppose another risk is that if we have a website
+around error messages, that makes it harder to make edits to error messages or to change how errors
+are reported.