Coordination for structured error messages

[goldfirere]

This proposal, a complete rewrite of what was originally #21, seeks coordination from the HF to convert GHC's error messages to use a structured format, benefiting consumers such as HLS. It furthermore seeks to create a website explaining the error messages.

Rendered

[Ericson2314]

I am very happy to see this! These are indeed the areas I wish to see worked on.

I am skeptical that volunteers will be enough to get the job done, but I don't hold this against the proposal. My gut says that most potential volunteers rather work on end features than these prerequisites which aren't so exciting and of themselves, but gut instincts are not good enough. Either way this goes, I am happy with the outcome:

1. If our experience here proves me wrong, that's fantastic --- I would love to see more people come out of the word works interested in these sorts of yak shaves!

2. If we indeed do run into trouble, then we have a good empirical argument that the HF should prioritize doing this deep infrastructural work itself over "last mile" projects which volunteers can tackle.

So big 👍 from me!

[goldfirere]

Your point (2) is a very interesting one, @Ericson2314. We've been operating under a general assumption that the HF should primarily coordinate, plan, excite, publicize, etc... and not do so much. But maybe this assumption is wrong, and trying (and failing) to simply coordinate on something like this proposal would teach us where we have to do.

Of course, the HF may not have the resources to be the primary agent on a project like this, in which case this proposal becomes more aspirational, in the end, than practical. That would be an understandable outcome, and if it fuels a desire e.g. to fundraise in order to support initiatives like this one, that wouldn't necessarily be so bad.

[michaelpj]

This is great, it would be wonderful to get this effort over the line. My main suggestion is that I think this would be better just scoped to GHC for now. Get GHC's error messages structured and in the User's Guide and that would be a big win on its own.

[michaelpj]

GHC has a nice website of documentation for users: the User's Guide. It's versioned, built nicely from source, and the source lives alongside GHC where the error definitions go. For that reason I think it would be a good proof-of-concept/low-hanging-fruit to start with a directory of GHC's error messages in the User's Guide. If that went well, we could think about how to re-present the same content in another place if that seemed useful.

[goldfirere]

There are pluses and minuses to using the user manual. (I've added a note later in the document that the manual might be a good place.)

Pluses:

• It seems easier to ensure that the documentation keeps up with the implementation.
• It seems easier to ensure a high quality of documentation (via the normal PR review).
• The user manual is already web established, with all of its infrastructure in place.

Minuses:

• Higher barrier-of-entry to edit the manual, due to GHC's review process.
• No easy way to just leave a comment. I would see the error-message resource as benefiting from user comments, providing extra examples and insights.
• Hard to generalize beyond GHC (when that is ready).

[cdornan]

I would have thought the source of truth would belong in a code repository -- either GHC itself or a dedicated repo. But of course it would be ideal if we could run a tool to extract docs for inclusion in the GHC manaual, which could be incorporated into the GHC manual build pipeline.

Would this not answer to everyone's needs?

[goldfirere]

I think we may be agreeing. GHC's manual is in its code repository -- and in a format that can be extracted to the GHC manual website -- so that conforms to your suggestion. Or a separate website would have its content generated from some source repository ("a dedicated repo"), so that also conforms. But I think maybe there is something deeper you're suggesting, but I've missed. Have I captured your intent here?

[cdornan]

I thought as much -- looks like we are all in violent agreement.

[michaelpj]

It's was a little unclear to me what the scope should be. Here you say "Haskell tooling" error messages, but the proposal talks a lot about GHC, and mostly gestures at other Haskell tooling. Extending this project to other tools makes it much bigger and more difficult. I'd be keener to have this really sorted for GHC, than to have a broader effort that got less far.

[goldfirere]

Yes. I started writing with loftier goals, but I realized as I wrote that HLS <--> GHC was the sweet spot. I've toned down the introduction.

[michaelpj]

The HLS/GHC integration is different in kind to, say HLS's integration with other tools. Many of the most difficult ones (mostly the build tools) are invoked as binaries, which makes passing structured errors more difficult. Of course, this could be overcome (many tools these days have a --json flag for giving structured output as JSON), but it's much easier when the interface is just a Haskell library interface.

[michaelpj]

I'm unsure if this is a good idea. We should see if other ecosystems have every managed anything like this. I think maybe Microsoft tried this at some point? There's a risk that this would be quite burdensome for developers and become ignored or obsolete.

[goldfirere]

I'd be fine with dropping this from the proposal, but I'm not quite ready to drop it yet, as I really do think that tools should identify the speaker. The "established global namespace" might be something as simple as a two-letter code (or longer) that tools could use if they decide to produce error messages, where the tool decides how to proceed after that prefix.

That said, if others agree that this should be dropped, I'm OK with that.

[michaelpj]

I suspect that using the new structured errors will be straightforward. What will probably be non-straightforward and unpleasant is managing the compatibility between versions of GHC that don't have structured errors and those that do. Could be a good GSOC project.

[goldfirere]

Yes, that's a good point. But maybe some design work within GHC will avoid this problem? I'm not sure. And we need to make sure that the errors are delivered in the right way. Maybe part of the reason we didn't get much feedback in the past is that it's obviously going to be easy.

[michaelpj]

This is happening incrementally, right? Which version of GHC do they start to appear in?

[goldfirere]

9.4 will be the first.

[michaelpj]

Oh no, it will be so much better than the awful error message parsing that's going on now. It's buggy, painful, unnecessarily complicated code.

[goldfirere]

I agree. But having not heard voices from HLS say this loudly, I didn't want to be over-presumptuous.

[michaelpj]

Another reason why the User's Guide is a good location. If we don't think users find it easy to find the User's Guide, then fixing that is great anyway!

[michaelpj]

My gut says that most potential volunteers rather work on end features than these prerequisites which aren't so exciting and of themselves, but gut instincts are not good enough.

I think it's worth noting that several of the big PRs to convert GHC errors to structured formats have been from volunteers. I think it's actually quite an appealing thing to work on: you can make a nice difference to the project, and you don't have to know what skolemization is to do it!

[Ericson2314]

My gut says that most potential volunteers rather work on end features than these prerequisites which aren't so exciting and of themselves, but gut instincts are not good enough.

I think it's worth noting that several of the big PRs to convert GHC errors to structured formats have been from volunteers. I think it's actually quite an appealing thing to work on: you can make a nice difference to the project, and you don't have to know what skolemization is to do it!

Oh that's great to hear! Thanks for mentioning it.

[gbaz]

I mentioned elsewhere that this reminds me of the polymath project: https://polymathprojects.org/

I don't remember where I read it, perhaps Terry Tao's blog, but there's a few good retrospectives about what makes a good polymath project and not, emphasizing the key role of the "coordinator" in managing to divide up worthwhile subproblems that make sense to be tackled by individual volunteers, as well as the need for some sort of "hub" or forum (which in the polymath case is, or at least has sometimes been, just a roundup blogpost with people then jumping in the comments, if i recall). The latter, which i stress can be simple -- anything from a github ticket to a blogpost, to a reddit or discourse post -- is a way to sync everyone up, generate enthusiasm and maintain momentum, and give a place for everyone to share and congratulate one another on their individual contributions. Just some food for thought -- it seemed to make a big difference for polymath, and it may here too.

[Abastro]

Maybe I am polluting this issue, but I don't know where to ask.
Can I browse the involved issues/PRs in the GHC repo? Also is it happening for GHC 9.4 - any validation(backing calculations on ETA) for this?

[goldfirere]

@Abastro The main GHC ticket tracking all of this is https://gitlab.haskell.org/ghc/ghc/-/issues/18516. GHC 9.4 will release with some of this work complete, but (presumably) not all of it.

[gbaz]

We met today, but did not have a sufficient quorum to take a decisive vote on this proposal. However, all of us that met were generally in favor of this proposal and think it should pass, and would like to call for an async vote to pass this.

That said, there are some tweaks that we think could be welcome:

• The stretch goal for a "global namespace" is unclear -- is this just adding a tag to each message, identifying which tool it comes from, or is it prefixes to the error codes, or is it a registry between all error codes and tools? Some more detail could be helpful.
• The coordinator is specified to be an "HF employee" but later its specified to be either an employee or another resource (volunteer, in-kind donation, etc). This should be clarified everywhere to the latter.
• An important job of the coordinator will be to shepherd through GHC PRs.
• There's some general concern about who the Coordinator will be -- even if we accept, the HF will need to figure out who to provision and how to get the time commitment.
• There's also some general dubiousness as to if we can actually source the volunteers necessary for this -- however, we think it is a good exercise to try and see how well it can be done!

[goldfirere]

Thanks for this feedback, @gbaz. I've updated the proposal accordingly.

[cdornan]

I have only just caught sigt of this proposal, and I think it is a great initiative.

I have been a lot of work on structured error messages as part of a wider project that involves looking at the general problem of better structured errors in Haskell. I can't share this work right now, but if folks were interested I could be pushed to speed up plans for publication nd in the meantime provide feedback or help out.

[gbaz]

Reminder to the committee that we would like an offline vote on this. Committee members only: please reply "yea" or "nea" on this proposal.

[kozross]

yea

[nomeata]

Yes

[Ericson2314]

yea

[gbaz]

While we didn't have enough purely offline votes to make quorum, the March 2 meeting of the tech committee (which did have quorum) approved this proposal. The next step, I believe, is to coordinate with the HF to source and appropriate coordinator role to begin implementation.