The cpython github merge webhook to bugs.python.org is broken

[ned-deily]

The github webhook that sends cpython merge updates to individual issues on bpo is failing (https://bugs.python.org/pull_request). I'm guessing it may have something to do with the recent upgrade and move of bpo to a different system. I don't have time to track it down myself at the moment but this is a pretty critical item as we are losing documentation in bpo of pushed merges.

[Mariatta]

GitMate.io thinks possibly related issues are python/devguide#4 (Migrate over issues from bugs.python.org), #28 (Add links back to bugs.python.org), #15 (Shut down Rietveld instance on bugs.python.org), #292 (Links to www.bugs.python.org), and python/devguide#79 (https://docs.python.org/devguide/ is broken?).

[Mariatta]

Webhook are returning 400 error from bpo.

[Mariatta]

I don't have access to bpo server log. I don't know who would be able to look into this.

[ned-deily]

Thanks, @Mariatta. @ezio-melotti and/or @ewdurbin have access, I believe.

[ewdurbin]

Fairly clear error message:

ERROR:root:Missing SECRET_KEY environment variable set!

Appears configuration is the only non ini file based config in the roundup instance, and was missed.

I'm working to resolve it now and will replay all backlogged web hooks after its restored.

[ewdurbin]

https://bitbucket.org/python/roundup/commits/59d6d4284aea7df30344758fe391478a47609c2e resolves, it's deployed and hooks are flowing. I'm trying to determine how to mass replay the hooks now.

[ewdurbin]

I've submitted a support request to GitHub... it appears our only option for redelivery of the failed hooks is to click the buttons in the GitHub UI (or script something to do so)... but not too fast! if you click them too fast you'll get rate limited :)

[ned-deily]

Thanks for getting it working again, @ewdurbin! But, yikes, I hope you don't have to push every button!

[ewdurbin]

Welp, clicked and tapped and it’s done. 😅

Gonna leave this issue open until we confirm things are flowing/behaving as expected.

[ewdurbin]

Got confirmation from GitHub that there is no automated way to mass redeliver failed hooks.

[ewdurbin]

Okay, looks like we had some kerfuffle in delivery over the past few days.

To address this I've refactored each roundup instance on the new host to run its own gunicorn stack and gracefully reload to maintain availability: python/psf-salt@ebacc0a

Will continue to watch for a few days :)

[ned-deily]

Thanks, @ewdurbin! I think the merge messages are now showing up as expected. But there is still at least one remaining transition issue: the multiple identical PR links showing up in b.p.o issues as referenced above.