Workbench cannot handle wind energy's debug logging from Taskgraph

[davemfish]

Rob gave us a datastack that includes a large wind_data_path CSV input. It's about 5MB, which is 2-3x larger than our sample data CSV. When Taskgraph logging level is set to DEBUG, a single line that is more than 18 million characters long is logged. This causes the Workbench renderer to consume an unreasonable amount of memory and ultimately crash.

The line begins,

2024-01-04 14:56:17,380 (taskgraph.Task) Task.is_precalculated(1166) DEBUG other_arguments: [[{0: {'LONG': -124.255, 'LATI': 46.9703, 'REF_LAM': 8.42, 'K': 1.92, 

That other_argument is the wind_data argument in,

compute_density_harvested_task = task_graph.add_task(
    func=_compute_density_harvested_fields,
    args=(wind_data, parameters_dict, number_of_turbines,
          wind_data_pickle_path),
    target_path_list=[wind_data_pickle_path],
    task_name='compute_density_harvested_fields')

For some reason, instead of passing the CSV as an argument to _compute_density_harvested_fields, we first read the CSV into a dict and pass that, so wind_data here is a dict. I don't see any reason why we need to do this, so passing the CSV into the function would avoid this logging problem.

Should the workbench more gracefully handle logging lines that are this long? Perhaps.

[davemfish]

Doug and I talked about possibly dis-allowing Taskgraph's DEBUG logging to reach the Workbench at all, only every allowing it to go the file handler. But that might involve some other tough decisions. Some options to accomplish this,

• prevent DEBUG logging for Taskgraph for all invest CLI users (the Workbench being one such user)
• remove the DEBUG menu option from the Workbench UI so a user can never select it. But this menu controls the level for both the StreamHandler and the FileHandler, so these would need to be de-coupled if we still want DEBUG logging available for the file. And I don't much like the idea of adding more UI widgets to control all this, as most Workbench users do not know/care about all these options.

So far I'm inclined to leave the Workbench logging as-is, assuming we can make it perform adequately.

[davemfish]

Here's the ultimate crash from the Workbench. Out-of-memory.

[1] Error sending from webFrameMain:  Error: Render frame was disposed before WebFrameMain could be accessed
[1]     at s.send (node:electron/js2c/browser_init:2:93158)
[1]     at _.send (node:electron/js2c/browser_init:2:78814)
[1]     at C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\electronApi.js:205:23
[1]     at Array.forEach (<anonymous>)
[1]     at sendIpcToRenderer (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\electronApi.js:203:42)
[1]     at Object.sendIpc (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\electronApi.js:185:5)
[1]     at transport (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\transports\ipc.js:40:17)
[1]     at runTransport (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\log.js:44:5)
[1]     at runTransports (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\log.js:27:7)
[1]     at log (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\log.js:21:3)
[1] [10:27:51.012] [undefined] { reason: 'oom', exitCode: -536870904 }
[1] Error sending from webFrameMain:  Error: Render frame was disposed before WebFrameMain could be accessed
[1]     at s.send (node:electron/js2c/browser_init:2:93158)
[1]     at _.send (node:electron/js2c/browser_init:2:78814)
[1]     at C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\electronApi.js:205:23
[1]     at Array.forEach (<anonymous>)
[1]     at sendIpcToRenderer (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\electronApi.js:203:42)
[1]     at Object.sendIpc (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\electronApi.js:185:5)
[1]     at transport (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\transports\ipc.js:40:17)
[1]     at runTransport (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\log.js:44:5)
[1]     at runTransports (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\log.js:27:7)
[1]     at log (C:\Users\dmf\projects\invest\workbench\node_modules\electron-log\src\log.js:21:3)

I saw memory usage get up to 8 or 9 GB which seems strange when the entire logfile itself is only 159MB. So perhaps there is a memory leak here.

[davemfish]

The renderer's memory use is also very high when displaying a pre-existing logfile, such as from a recently completed invest run. For example, a logfile that is 25MB on disk causes the renderer to use ~1GB of memory when displaying the log. Closing the model tab does not immediately free the memory, though it does eventually. And then baseline memory use for the renderer is about 100 MB.

[davemfish]

I'm not certain why this ended up in the 3.15.0 milestone, perhaps only because we wanted to bump it from an earlier milestone. It feels like a bugfix release is more appropriate than a minor release.

[emilyanndavis]

If we see this issue pop up elsewhere, we can look into ways to skip rendering particularly long messages in the Workbench (or block them from being passed to the Workbench in the first place), but since the wind data dictionary is the obvious culprit here, I think it makes most sense to address this in the Wind Energy module.

[emilyanndavis]

Also, for what it's worth, in my initial testing I managed to generate a log file that is ~120MB on disk, and even TextEdit (Apple's basic text file viewer/editor) struggled with it, using nearly 2GB of RAM once I started scrolling through the file. So, while there could certainly be opportunities for performance tuning in the Workbench, I suspect a lot of what we were seeing here is just in the nature of rendering a massive text file in a GUI.

[davemfish]

Also, for what it's worth, in my initial testing I managed to generate a log file that is ~120MB on disk, and even TextEdit (Apple's basic text file viewer/editor) struggled with it, using nearly 2GB of RAM once I started scrolling through the file. So, while there could certainly be opportunities for performance tuning in the Workbench, I suspect a lot of what we were seeing here is just in the nature of rendering a massive text file in a GUI.

That's interesting. Thanks for confirming that!