Huge memory usage (10GB+) during Phoenix debugging on a new project

[fnicastri]

Current behavior

During a debug session thru ElixirLS Debugger there is a huge spike in memory usage during Ecto's query.
Loading the index page (no queries) the memory spike is under a gigabyte.

I've noticed this behavior previously in the last few months.

Steps to Reproduce

• make a new phoenix project (standard Postgres), add a new model and corresponding views with:
• mix phx.gen.html
• start the debugger and add a new record
• reload the model index page
• in activity monitor watch one of the two beam.smp processes consume GBs of memory (10+)

Expected behavior

Not consuming over 10 GB of memory ;)

Environment

• Elixir & Erlang versions (elixir --version):

Erlang/OTP 25 [erts-13.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Elixir 1.14.4 (compiled with Erlang/OTP 25)

• Elixir Language Server version:

Started ElixirLS Debugger v0.17.5
ElixirLS Debugger built with elixir "1.14.4" on OTP "25"
Running on elixir "1.14.4 (compiled with Erlang/OTP 25)" on OTP "25"
Protocols are not consolidated
Starting debugger in directory: /Users/frank/projects/hello
Running with MIX_ENV: dev MIX_TARGET: host

• Operating system: macOS 13.6.1 (22G313)
• Editor or IDE name (e.g. Emacs/VSCode): VSCode
• Editor Plugin/LSP Client name and version: ElixirLS: Elixir support and debugger v0.17.5
• Phoenix: v1.7.9

[lukaszsamson]

Did you try attaching observer to the node to see what is actually taking up the memory? You can set sname/name via ELS_ELIXIR_OPTS
https://github.com/elixir-lsp/elixir-ls#environment-variables in env launch config option https://github.com/elixir-lsp/elixir-ls#debugger-configuration-options

[fnicastri]

@lukaszsamson
I can see eheap_alloc going crazy in Memory allocator tab
and cowboy_stream_h:request_process/3 in Processes

[lukaszsamson]

It's hard to tell what is the cause. It may be a drawback of OTP debugger. Excluding some modules from auto interpreting may help here.

[fnicastri]

@lukaszsamson

It's hard to tell what is the cause. It may be a drawback of OTP debugger. Excluding some modules from auto interpreting may help here.

TBH I don't know this side of Erlang/Elixir, so it is no easy for me to move around in this.
If you can provide me a bit of guidance I could be able to explore more the issue and produce more info.

Anyhow here you can find the repo where I reproduced the issue.

[fnicastri]

@lukaszsamson

I've excluded almost every module (in a real Phoenix project) except from the code from the actual application and the issue persists.

Tried whit OTP26 as well, nothing changes.

In the demo project the situation is better but the queries are small (just a couple of records from a single table)

I think this is very impacting and make the debugger unusable,
sadly I can't try on another platform, do you managed to reproduce the behavior?

Any thoughts?

[fnicastri]

@lukaszsamson
can I at least know if there is no interest in this?
I can understand if this is the case, but would be nice to know...

[lukaszsamson]

@fnicastri There is interest. I'm not closing the issue but currently I'm involved in language server stability fixes

[fnicastri]

@lukaszsamson
thanks for the clarification

[lukaszsamson]

@fnicastri I reproduced easily the problem with your sample app. The memory goes even up to 20GB after a few breakpoints. I'm not sure what is the cause. As you mentioned one of the processes that starts taking up insane amounts of memory is cowboy_stream_h but also ElixirLS.Debugger.Server. It seems the state is not the problem here as it holds only a few kB there. I suspect binaries - if I force GC it goes back to manageable 4GB. I will experiment with Jason settings (the only place I know when one can control how the binaries are handled).

I've excluded almost every module (in a real Phoenix project) except from the code from the actual application and the issue persists.

In the example you did not exclude anything. That will not work as this option needs full module names

"excludeModules": [
                "ecto",
                "cowboy"
]

Instead try this:

"debugAutoInterpretAllModules": false,
"debugInterpretModulesPatterns": ["Hello*"],

This will disable interpreting of everything but modules matching the glob. In case of the example app this completely resolves the issue and debugger stays at ~90MB.

[lukaszsamson]

It's not binaries but heap process heap as you wrote.
when running with "ELS_ERL_OPTS": "+hmax 101483800 +hmaxk false"

[error]      Process:            #PID<0.1679.0> on node :"mynode@MBP"
     Context:            maximum heap size reached
     Max Heap Size:      101483800
     Total Heap Size:    1115915790
     Kill:               false
     Error Logger:       true
     Message Queue Len:  0
     GC Info:            [
  old_heap_block_size: 850246556,
  heap_block_size: 395480352,
  mbuf_size: 0,
  recent_size: 65433676,
  stack_size: 423,
  old_heap_size: 406186569,
  heap_size: 67929058,
  bin_vheap_size: 474246280,
  bin_vheap_block_size: 33514735,
  bin_old_vheap_size: 77068941,
  bin_old_vheap_block_size: 165709447
]

I tried setting ERL_FULLSWEEP_AFTER to some low value but this does not make any difference. Maybe some allocator tuning would help https://www.erlang.org/doc/man/erts_alloc.html
Anyway, I'm afraid it's some strange interaction between cowboy and OTP debugger that we can't do anything about besides disabling interpreting. Please let me know if debugAutoInterpretAllModules and debugInterpretModulesPatterns resolve the issue in your project.

[fnicastri]

@lukaszsamson

In the example you did not exclude anything. That will not work as this option needs full module names

"excludeModules": [
                "ecto",
                "cowboy"
]

Sorry, I never pushed that but I excluded them

[fnicastri]

@lukaszsamson

Anyway, I'm afraid it's some strange interaction between cowboy and OTP debugger that we > can't do anything about besides disabling interpreting. Please let me know if > debugAutoInterpretAllModules and debugInterpretModulesPatterns resolve the issue in your > project.

Tried on a real project, no changes... :(

[lukaszsamson]

Sorry, I never pushed that but I excluded them

Just to be clear in Debugger dumps to console all interpreted modules

Interpreting module Hello.Controller.Some

I wonder which process is taking up memory in your project when you disable interpretting cowboy.

Some other ideas to try:

• do you use phoenix reloader? Debugger is known to be incompatible with it
• try to interpret only the single module you need to debug
• disable interpreting altogether and put dbg in your code when you would want a breakpoint - the debugger will stop the process using elixir Kernel.dbg backend. Note that dbg breakpoints are pretty limited in features - you can examine variables, eval and step through pipes. Step in/out, function breakpoints, logpoints, hit conditions, breakpoint setting and unsetting from UI, breaking in erlang code is not supported

[lukaszsamson]

Closing as this is not addressable in this project. Please reopen an issue to either cowboy or OTP

[lukaszsamson]

@fnicastri can you share your launch.json that I could use with an empty phoenix app? Last time I tried the workarounds from #1017 (comment) fixed the issue. Other people also reported that it works in their case

[fnicastri]

Doing some tests with current releases...

[fnicastri]

Things are better now but still, if you add some db calls or try to debug the Web app, memory goes up.

I've tried many scenarios, some are okayish,
in some others it's just a huge spike (3GB+ or much more) but then memory goes down to around a GB (not always),
others are totally unusable (10GB+ or till saturation).

repo
main is clean
with_data has just a model and a view

LiveView is much worse than a plain view

launch.js

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "mix_task",
            "name": "mix (Default task)",
            "request": "launch",
            "task": "phx.server",
            "projectDir": "${workspaceRoot}",
            "env": {},
            "exitAfterTaskReturns": false,
            "debugAutoInterpretAllModules": false,
            "debugInterpretModulesPatterns": [
                "DebuggerTestWeb.*"
            ],
        }
    ]
}

[fnicastri]

With the Erlang debugger the memory always remains around 100 MB with the same project

[jsermeno]

Some related discussion occurred in this issue as well: #1101. Copying relevant message here:

Referring to that other issue though, I don't think it is Cowboy specific. I've created a minimal reproduction in this repo: https://github.com/jsermeno/elixir_ls_debug_failure. The only dependency I need to recreate this issue in a brand new project is {:stream_data, "~> 1.0", only: [:dev, :test]} and the following code:

  def gen_data() do
    StreamData.list_of(StreamData.fixed_map(%{
      "e" => StreamData.string(:utf8)
    }), min_length: 1, max_length: 1)
  end

  def gen_data_all() do
    gen all fixed_map <- gen_data() do
      fixed_map
    end
  end

  test "greets the world" do
    data = Enum.at(gen_data_all(), 0)
    assert ElixirLsFailure.hello() == :world
  end

I'll open an issue with StreamData as well. However, it seems it's likely not something specific this library since I don't believe Phoenix relies on this library.

[jsermeno]

Minimal reproduction : https://github.com/jsermeno/elixir_ls_debug_failure.