[Mono]: Add "driver" support to Mono AOT cross compiler.

[lateralusX]

Running Mono AOT cross compiler will keep the process alive while running external tools like opt and llc. When building a large assembly using LLVM the memory footprint of Mono AOT cross compiler can fall in the gigabytes and then running LLVM tools like opt and llc as child processes tends to take up additional memory in the gigabyte space for large assemblies meaning that the total memory needed to compile one large assembly will be max Mono AOT cross compiler memory usage + (opt|llc) memory usage, meaning that it can consume gigabytes of memory. Adding parallel compiles into the mix means machine memory will drain pretty fast.

This PR adds a "driver" mode to AOT compiler. Its integrated into the execute system command infrastructure of the Mono AOT cross compiler, meaning that a Mono AOT compiler process doing the compile work will write the exact commands (opt, llc, clang, ld etc) that it would have executed into a file or stdout and the AOT compiler instance acting as the driver will record them and execute them once the original child process exited (and released all its memory). Doing it like this won't keep memory around and the Mono AOT cross compiler instance acting as the "driver" will only consume a couple of MB's of memory. The benefit of recording the commands into a file/stdout will make sure we are always up to date with the correct set of arguments for each command and it is even possible to write a custom driver and it will be able to run the exact same commands as the AOT cross compiler would have done when not executed as a driver child process.

By default, driver mode in Mono AOT cross compiler is disabled. To enable it add aot=driver argument to the list of all arguments passed to Mono AOT cross compiler. That will run a different instance of the Mono AOT cross compiler with the same set of arguments, but adding aot=driver-logfile=stdout that makes sure child process runs the compile of the assembly, but record selected executed commands into the supplied file, by default that is stdout to only communicate between the pipe setup between parent and child process. Once all commands have been recorded and the child process terminates, the driver instance will execute each command in sequence (after doing validation of the binary used in each command).

Here is an example of how the child process of a driver instance logs into stdout:

Mono Ahead of Time compiler - compiling assembly System.Private.CoreLib.dll
AOTID 41E59EEC-D030-B60E-4C98-5ABE667E0B63
[LLVM_OPT_COMMAND] opt.exe -f -disable-tail-calls -passes="default<O2>,place-safepoints" -spp-all-backedges -o "temp.opt.bc" "temp.bc"
[LLVM_LLC_COMMAND] llc.exe  -march=x86-64 -mcpu=generic -enable-implicit-null-checks -disable-fault-maps -asm-verbose=false -mtriple=x86_64-pc-windows-msvc -disable-gnu-eh-frame -enable-mono-eh-frame -mono-eh-frame-symbol=mono_aot_corlib_eh_frame -disable-tail-calls -no-x86-call-frame-opt -relocation-model=pic -filetype=obj -o "System.Private.CoreLib.dll-llvm.obj.tmp" "temp.opt.bc"
Compiled: 40119/40643
[ASM_COMMAND] "clang.exe" --target=x86_64-pc-windows-msvc -c -x assembler  -o System.Private.CoreLib.dll.obj.tmp temp.s
Output file: System.Private.CoreLib.dll.obj.tmp.
Linking symbol: 'mono_aot_module_System_Private_CoreLib_info'.
JIT time: 11254 ms, Generation time: 8502 ms, Assembly+Link time: 0 ms.

so each of the [XX_COMMAND] are just logged into stdout by child process as it would have been executed, driver parent process reads stdout and extract commands into a command log, the rest is written to its stdout making sure all the output (except the commands) are still visible to the user. It will then execute the list of commands in sequence as recorded by the child process after the child process has terminated and the system has been able to reclaim its memory.

Since the driver currently don't support the unlink, rename and rm commands used by AOT compiler in some scenarios, it currently only works with the outfile || llvm_outfile and temp_path aot options. It is simple to add them in the driver, but in the use case where the complete driver option was implemented didn't have the need since things where always executed with outfile and temp_path. The same applies to assemblies compiled through the MonoAOTCompiler msbuild task.

[vargaz]

Typo 'executin'.

[vargaz]

Wouldn't it be possible to just replace this with a generic 'exec_command' facility ?

[lateralusX]

Wanted to have a more structural way to describe supported commands making it simpler for clients to quickly identify/validate the commands that are requested to be executed. Also, since several tools have different names on different platforms, having a way to type it also makes it simpler for the clients to identify what things its supposed to be executed and take decisions based on that. The pace of adding new commands to AOT compiler is also very slow, so don't anticipate to much change here either.

[vargaz]

Wouldn't it be better to have this as a python/c#/msbuild script/task ?

[lateralusX]

Wouldn't it be better to have this as a python/c#/msbuild script/task ?

You mean the driver portion of the PR, running the second instance of AOT compiler + executing the commands returned over stdout? With the current implementation it is possible to implement a driver in any language as a complement to the driver, just run AOT compiler with driver-logfile=stdout from a different driver implementation and then take care of the commands returned in the stream. I also added the driver capabilities to AOT compiler since it will make it simple to use the feature for existing build system by just adding aot=driver to the command line. That made the integration into an existing downstream build infrastructure very very simple, so I think the solution in this PR as well as the potential to do custom drivers together with changes in this PR adds a powerful mechanism to the overall AOT compiler chain and in the LLVM case, saves a lot of memory.

[ghost]

Draft Pull Request was automatically closed for 30 days of inactivity. Please let us know if you'd like to reopen it.

[lateralusX]

@vargaz what do you think on getting this in? I've been running it downstream for quite some time and it has big impact on the parallelization on the machines since we consume half RAM per compiled assembly when running LLVM. Its also disabled by default and needs to be explicitly enabled and it opens up for alternative driver implementations in other front ends as well.

[vargaz]

I'm trying to make a version which doesn't use pipes, but executes the 'compile code' phase in a child.

[lateralusX]

I'm trying to make a version which doesn't use pipes, but executes the 'compile code' phase in a child.

That implementation can use a file instead of pipes as well, just pass in a path to a file and that will be used for the commands. That was the original implementation, I then extended with possibility to use stdout as an alternative.

[dotnet-policy-service]

Draft Pull Request was automatically closed for 30 days of inactivity. Please let us know if you'd like to reopen it.