v0.13.0 is generating a large CPU load and starving the nginx process during renewals

[mklauber]

Hi, I'm working with a large system with 10's to 100's of thousands of domains, and something between version 0.12.0 and 0.13.0 is resulting in a large amount of CPU churn as we try to renew certificates. This appears to completely stop openresty from serving requests.

My guess is that this is related to the shuffle code added into the v.013.0 release, but I can't 100% confirm that. Can anyone provide advice on confirming or addressing that, or if there's something else people think I should look into?

[mklauber]

To provide additional information, about once every 24 hours, a large number (30-40) of dehydrated processes would be started simultaneously, and the ability of our server to process requests would become degraded, eventually resulting in logs grinding to a complete halt. A restart of openresty would fix the issue, but also stop the renewals.

[mklauber]

I can say the issue is coming from dehydrated. Each dehydrated process appears to enter a state where it sits, waiting, consuming 15-20% cpu. Advice is welcome if you have any, but I'll also understand if you want to close this and point me upstream to https://github.com/dehydrated-io/dehydrated

[mklauber]

Alright, I think we actually tracked this down to being on an old version of openresty that had a deadlock bug in it: openresty/lua-nginx-module#1207 (comment) Closing this

[GUI]

@mklauber: Sorry for the issue and lack of response, but thanks for bringing this up. I've updated the README to better indicate OpenResty 1.15.8.1+ should fix these unfortunate deadlock issues that it seems like some users run into. But I've also tweaked lua-resty-auto-ssl to automatically require the resty.core FFI library, which should hopefully fix this issue in older OpenResty versions in the next lua-resty-auto-ssl release. So thanks for connecting the dots on all this!