Increased memory usage in version 2.11.18

[aucampia]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

Upgrade from version 2.11.16 to 2.11.18. We did not expect any change in memory usage.

What did you see instead?

It seems Træfik now uses at least 2x as much memory as before, and it may even be leaking memory.

What version of Traefik are you using?

2.11.18

What is your environment & configuration?

N/A

If applicable, please paste the log output in DEBUG level

No response

[nmengin]

Hey @aucampia,

Thanks for reaching out.

Could you provide a minimal reproducible case (for instance, full Kubernetes manifest to reproduce the issue)?

[romainhealth]

Hi,

We are also experiencing this issue with the latest Traefik version 2.11.18.
We are noticing memory usage spikes, which ultimately lead to our Docker Swarm Managers being killed.

On January 8th, we updated to version 2.11.18.

Traefik is running in our Docker Swarm environment. We have 3 managers, 2 workers, and over 20 services.

version: '3.3'

services:
  proxy:
    image: traefik:v2.11.16
    command: 
      - --api=true
      - --providers.docker
      - --providers.docker.swarmMode=true
      - --entryPoints.web-secure.address=:443
      - --entryPoints.web-secure.transport.respondingTimeouts.readTimeout=150
      - --entryPoints.metrics.address=:8082
      - --metrics=true
      - --metrics.prometheus=true
      - --metrics.prometheus.entryPoint=metrics
      - --ping=true
      - --entryPoints.ping.address=:8083
      - --ping.entryPoint=ping
    ports:
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      sample:
      management:
    deploy:
      mode: global
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.traefik.rule=(Host(`${url}`) && PathPrefix(`/api`)) || (Host(`${url}`) && PathPrefix(`/dashboard`))"
        - "traefik.http.services.traefik.loadbalancer.server.port=8080"
        - "traefik.http.routers.traefik.service=api@internal"
        - "traefik.http.routers.traefik.tls=true"
        - "traefik.http.routers.traefik.middlewares=auth"
        - "traefik.http.middlewares.auth.basicauth.users=sample:sample"
        - "traefik.http.routers.traefikping.rule=(Host(`${url}`) && PathPrefix(`/ping`))"
        - "traefik.http.routers.traefikping.service=ping@internal"
        - "traefik.http.routers.traefikping.tls=true"
        - "traefik.http.services.traefikping.loadbalancer.server.port=8083"
      placement:
        constraints:
          - node.role == manager
networks:
  sample:
    external: true
  management:
    external: true

volumes:
  logs:

[johnbizokk]

We also faced this problem

[Bobris]

We have same problem after upgrading from 2.11.16 to 2.11.18, pods starting to be OOM killed.

[aucampia]

Sorry I accidentally closed this, please re-open it.

[emilevauge]

Here are the PRs merged between 2.11.16 and 2.11.18:

• [websocket,server] Disable http2 connect setting for websocket by default (Disable http2 connect setting for websocket by default #11412 by rtribotte)
• [acme] Update go-acme/lego to v4.21.0 (Update go-acme/lego to v4.21.0 #11368 by ldez)
• [middleware] Fix typo in basicauth note (Fix typo in basicauth note #11397 by tieje)
• [service] Configure ErrorLog in httputil.ReverseProxy (Configure ErrorLog in httputil.ReverseProxy #11344 by peacewalker122)
• Bump golang.org/x/net to v0.33.0 (Bump golang.org/x/net to v0.33.0 #11365 by kevinpollet)

The most likely root cause is the bump of the lib golang.org/x/net to v0.33.0. There is no release note associated with v0.33.0. The go team released v0.34.0 on Jan 6, it might be worth a try.

In the meantime, can anyone provide a pprof of Traefik while facing the issue?

[HomoSapiens]

Have the same problem. Memory consumption is too large and increasing.

[mmatur]

Hi,

We’ve built a test version of v2.11.18 with the x/net dependency downgraded to version v0.30.0. You can find the changes here.

A corresponding Docker image is available on Docker Hub.

⚠ Warning:
This is a test image intended solely for debugging purposes, to determine whether the upgrade of x/net to version v0.33.0 could be impacting memory consumption.

Let us know your findings!

[RemiBou]

I saw this also with upgrade from 2.11.16 to 2.11.17

[RemiBou]

a pprof show huge usage of logrus maybe it's #11344

[avivklas]

I think I found something. In #11344, a new logger object is initialized per request.

[emilevauge]

Indeed, this is the issue:
https://github.com/traefik/traefik/pull/11344/files#diff-f7d7f0e8fef165ce3ca78be8f4d887b323d564a29b25d416a6a7d2b0e9ff7df7R86
ErrorLog: stdlog.New(log.WithoutContext().WriterLevel(logrus.DebugLevel), "", 0),
Thanks a lot @RemiBou for helping on this, we will be able to fix it ASAP.

[RemiBou]

I think it's missing a close on the logger and happens mostly hen you have a lot of config changes

[traefiker]

Closed by #11487.