The queue-proxy's memory slowly and constantly increases even if there is no request.

[cheimu]

What version of Knative?

0.23.0

Expected Behavior

When there is no request sent to the pod, the memory usage of queue-proxy should be a constant or at least a stable value.

Actual Behavior

When there is no request sent to the pod, the memory usage slowly increases. Specifically, the queue-proxy's memory is increasing. Don't know if it is a memory leak.

Steps to Reproduce the Problem

1. In config-observability, set metrics.backend-destination: prometheus and metrics.request-metrics-backend-destination: prometheus
2. Set annotations of prometheus.io for ksvc you want at ksvc.spec.template.metadata.annotation where port is 9091 and path is /metrics
3. Create ksvc and watch the pod.

[markusthoemmes]

Interesting, we should definitely look into that. Is it converging at ~23megs or is it still rising?

[cheimu]

Interesting, we should definitely look into that. Is it converging at ~23megs or is it still rising?

@markusthoemmes Still rising. I've turned on pprof and am watching the mem diff. Hope to find some details of it.

[skonto]

I can take a look.

[julz]

Just to ask the obvious question here: does step 1 of the reproduction steps imply that the problem does not occur if you don't set prometheus as the metrics backend? Does step 3 imply it only occurs if you actively poll metrics for the pod from prometheus? (Obviously asking because both of those narrow down where the leak might be quite significantly).

[cheimu]

Just to ask the obvious question here: does step 1 of the reproduction steps imply that the problem does not occur if you don't set prometheus as the metrics backend? Does step 3 imply it only occurs if you actively poll metrics for the pod from prometheus? (Obviously asking because both of those narrow down where the leak might be quite significantly).

@julz Hi Julian. Good question. I have no idea about it. The only monitor I have and used is prometheus... There is constant network io and it is from prometheus (because I'm 100% sure there is no request send to the pods created by ksvc)

[markusthoemmes]

FWIW: I totally expect the leak to be in the prometheus realm. I should've spelled that out clearly I guess 😂 .

[cheimu]

FWIW: I totally expect the leak to be in the prometheus realm. I should've spelled that out clearly I guess 😂 .

I got confirmation from my sre team... they said their prometheus will definitely close the connections for each pull... It has been tested for couple years... Huh
I'm using pprof to check the heap usage, and they seem like pretty normal, but there are 17 goroutines got blocked

[markusthoemmes]

I might've phrased myself poorly: I totally expect the "problem" (if it is any) in the realm of our metric collection inside the queue-proxy (which is potentially related to either opencensus and/or prometheus libs).

Not sure the goroutines tell us anything here though. A heap profile would likely be more insightful.

[cheimu]

I have disabled prometheus backend in config-observability, and found the memory of queue proxy is still rising, so probably is not the problem when pulling by prometheus?

The PromQL for it is container_memory_rss_aggregate

The metrics are provided by k8s....

[skonto]

Could you let it run for a few hours and check if it stabilizes at all? For example we have soak tests downstream that run >12h. /cc @maschmid

[cheimu]

@skonto No problem

[cheimu]

@skonto It is still rising

[cheimu]

Is it probably because of golang itself golang/go#42330 ? Upgrade to golang 1.16 may help... It seems like this situation also happened to other components....