Memory leak on 3-node cluster

[spencerkimball]

Run the examples-go/loadgen against a 3-node cluster
The memory consumption of the cluster will increase without bound

I've also seen the memory leak persist after load has stopped, with the cluster remaining mostly quiescent.

[petermattis]

A little memory profiler action while this is happening should at least reveal where the memory is going.

[nvanbenschoten]

See #4367. It looks like this issue was being caused by enormous amounts of time.Timer allocations. This was caused by timer creation during every iteration of a couple of tight loops. #4175 proposed a fix for the worst offender of this issue, and #4367 fixes a few other instances that were showing up during memory profiling.

[JackKrupansky]

Are these timer allocations that the Go GC would eventually have reclaimed
and you just didn't wait long enough for the GC to occur, or was there
something that was preventing Go from reclaiming the old timers or maybe GC
wasn't keeping up with the rate of creation? Just curious, trying to
understand Go better.

-- Jack Krupansky

On Sat, Feb 13, 2016 at 1:08 AM, Nathan VanBenschoten <
notifications@github.com> wrote:

See #4367 #4367. It looks
like this issue was being caused by enormous amounts of time.Timer
allocations. This was caused by timer creation during every iteration of a
couple of tight loops. #4175
#4175 proposed a fix for
the worst offender of this issue, and #4367
#4367 fixes a few other
instances that were showing up during memory profiling.

—
Reply to this email directly or view it on GitHub
#4346 (comment)
.

[nvanbenschoten]

Looking more intro this, and running the example load generator with more users to create higher contention within the key space, I'm seeing a huge amount of memory (roughly 1GB per node) allocated here. Again, this doesn't look like a memory leak, but it does look like an area where a tremendous amount of memory is being allocated, especially during high contention situations.

I'm not exactly sure what the best fix for this would be. One idea would be to pool these pending *WaitGroup slices within the CommandQueue, to avoid continually creating and extending them as contention builds and more cmds are added and processed. I'm going to look into the effect of this approach, as well as look for alternative remedies.