Improve startup time of workers

[lars-reimann]

During evaluation, we observed delays ranging from 3s to 20s+ between a program message and the first values coming back, regardless of the size of the data. For better usability, this time should be reduced as much as possible, since this

• delays the table view showing up
• and later causes another delay for the profiling.

[lars-reimann]

A few imports in the library constitute most of the delay:

• (Measure-Command { python -c "import seaborn" }).TotalSeconds: 1.21s
• (Measure-Command { python -c "import torch" }).TotalSeconds: 1.24s
• (Measure-Command { python -c "import torchvision" }).TotalSeconds: 1.90s (also loads torch)
• (Measure-Command { python -c "import openpyxl" }).TotalSeconds: 0.26s

We should get major improvements regarding startup by:

1. Lazily loading our own classes in __init__ files (apipkg). Why is this not the default behavior in Python?!
2. Lazily importing external libraries. Why is this not the default behavior in Python?!

Unfortunately, many libraries currently import all their stuff into a global package, so if we use anything from, say seaborn, there will still be the 1.21s delay. This can only be fixed by the libraries themselves.

Regarding the runner, we will need to have a primed process ready that can handle the execution of the next pipeline. Ideally, we would also keep those processes alive and reset them after pipeline execution is done, so we don't have the startup delay again and again.

[WinPlay02]

Ideally, we would also keep those processes alive and reset them after pipeline execution is done, so we don't have the startup delay again and again.

@lars-reimann This is somewhat supported, as metapath (for loading our pipelines) is already reset after a pipeline ends. The process could theoretically be reused, but this behavior would not guarantee correctness in all cases. States from external libraries (like seaborn) however, are not reset.

Regarding the runner, we will need to have a primed process ready that can handle the execution of the next pipeline.

I'll try something out for tomorrow, maybe something like that can be achieved.

[lars-reimann]

The library improvements are already in Safe-DS/Library#624. But that will only get us so far. It would be great if you could take a look.

[lars-reimann]

There's also a new release of the library now, containing your hash functions and your other improvements, as well as the lazy imports.

[lars-reimann]

🎉 This issue has been resolved in version 0.12.0 🎉

The release is available on:

• v0.12.0
• GitHub release

Your semantic-release bot 📦🚀