memory leak - high memory usage

[helmut72]

I run dagu with a single instance for about 9 days. Still testing dagu with 3 cronjobs that runs every few minutes. RAM usage starts at around 15MB. After 9 days it use 92MB RAM. In the past it even use over 300MB RAM with a longer uptime.

[yohamta]

Thanks for reporting the issue. Let's try to fix this.

[helmut72]

Thank you. I have started a second instance with example cronjobs. If RAM usage goes up too after some days, I'll share this example.

[helmut72]

My test instance starts at about 10 MB RAM and now need 60 MB RAM. In 3 days.

Command:

docker run -d --name dagu-test --rm -p 8080:8080 -v ./dags:/home/dagu/.dagu/dags -v ./dagu/data:/home/dagu/.dagu/data -v ./logs:/home/dagu/.dagu/logs yohamta/dagu:latest dagu start-all

Using 4 dags from your example:

./dags/hello1.yaml:

name: hello1 

schedule: "*/9 * * * *"

steps:
  - name: get fake json data
    executor: http
    command: GET https://jsonplaceholder.typicode.com/comments
    script: |
      {
        "timeout": 10,
        "headers": {},
        "query": {
          "postId": "1"
        },
        "body": ""
      }

./dags/hello2.yaml:

name: hello2 

schedule: "*/3 * * * *"

steps:
  - name: get fake json data
    executor: http
    command: GET https://jsonplaceholder.typicode.com/comments
    script: |
      {
        "timeout": 10,
        "headers": {},
        "query": {
          "postId": "1"
        },
        "body": ""
      }

./dags/hello3.yaml:

name: hello3 

schedule: "*/5 * * * *"

steps:
  - name: get fake json data
    executor: http
    command: GET https://jsonplaceholder.typicode.com/comments
    script: |
      {
        "timeout": 10,
        "headers": {},
        "query": {
          "postId": "1"
        },
        "body": ""
      }

./dags/hello4.yaml:

name: hello4 

schedule: "*/7 * * * *"

steps:
  - name: get fake json data
    executor: http
    command: GET https://jsonplaceholder.typicode.com/comments
    script: |
      {
        "timeout": 10,
        "headers": {},
        "query": {
          "postId": "1"
        },
        "body": ""
      }

My other instance now runs for 12 days and need 122 MB RAM.

Btw, running of course your latest v1.12.11 release. Thank you.

[yohamta]

Thanks a lot! Will try to pprof using these examples.

[helmut72]

Can you reproduce the memory leak?

[yohamta]

Sorry, not yet because of my bandwidth, but I have removed some unnecessary heap allocations to fix other problems. So it might solve the problem to some extent.

[helmut72]

Ok, will try it after a new release.

[yohamta]

Btw, would you mind clarify the context where you're running Dagu? :) I'm wondering what makes the memory usage so important on your environment.

[helmut72]

Some Raspberry Pi provides only 1GB of RAM or even less (512MB!). It makes you sensible for RAM usage. Any swapping means higher wearing of the SD card and you really don't like a system that is crashed for this reason.

But regardless of this, I think any memory leak is something that should be fixed:
https://en.wikipedia.org/wiki/Memory_leak

[yohamta]

I see, thanks for your clarification.

[helmut72]

Still the same problem with v1.13.0

[yohamta]

Thank you very much for checking it. Let's continue investigation.

[helmut72]

This test instance still runs and use 90MB RAM. It started with 10MB RAM 5 days ago!

To force it, I have changed the timings to */1, */2, */3 and */4 minutes in my example above.

It's not just an HTTP request. I've only used your example because it's simple. Mem usage also goes up with other scripts.

[yohamta]

Thanks for the info! I think we can add code to trigger garbage collection after each DAG completes. Additionally, let's implement some APIs and logging to track the running goroutines, memory, and CPU for better debugging and monitoring. We need to investigate the memory usage in more detail, but needs a bit more time. I hope we can get some help for it.

[arky]

@helmut72 Do you experience memory leaks if you run the dagu on the OS instead via docker?

[helmut72]

Running on the OS is no option. Every process that is needed to fulfill my services, is running in a container (minus SSH). Then no individual configuration can be forgotten. Also this would be the first process that behave this much different.