Support early print buffer flushing in debug mode

[zibaiwan]

Added a flow to flush printf() buffer before the Kernel ends. This will be helpful for debugging a hang Kernel. The Runtime periodically polls from the device and process the printf buffer from the last processed point.

If ACL_HAL_DEBUG = 1 and the Kernel is Hang, Kernel status (existing feature) and printf buffer will be flushed to stdout every 10 seconds (approximately). The thread will run if there are spare cycles.

If ACL_HAL_DEBUG = 3, Kernel status and printf buffer will flushed every 10 seconds, even the Kernel isn't hung.

The behavior of printf() is not altered in non-debug mode.

Sample output

  Kernel  0 Status: 0x00048000 running:: Accelerator 0 printf buffer size is 160.
:: Calling acl_process_printf_buffer_fn with activation_id=1 and printf_size=160.


Previously processed buffer size is 0 
Another test string 14
This is a test
Another test string 11
Another test string 12
Another test string 13
  Kernel  0 Status: 0x00048000 running:: Accelerator 0 printf buffer size is 160.
:: Calling acl_process_printf_buffer_fn with activation_id=1 and printf_size=160.


Previously processed buffer size is 160 
All Printf() buffer has already been dumped

ToDo

• Fix Klocwork if applicable
• Run Clang format
• manual testing
• Clear commit messages and Rebase
• Run existing regtest to ensure existing feature still passes
• Create a new regtest to test this specific feature

[pcolberg]

Thanks @zibaiwan! Could you include a commit message that summarizes the change and also motivates why it was made?

Have you taken a look at the unit tests to see if anything could be inserted to test this new functionality, as far as is feasible in a unit test? For instance, checking that the new state variables added here correspond to the expected state.

[zibaiwan]

@pcolberg , thank you very much for your review. I have updated the PR based on your suggestions.

[pcolberg]

Thanks @zibaiwan for the additional tests, much appreciated 🙂

[sherry-yuan]

A small question about a (maybe) corner case: What happens if the kernel stalls before the next 10 sec printf buffer dumps?

[zibaiwan]

A small question about a (maybe) corner case: What happens if the kernel stalls before the next 10 sec printf buffer dumps?

@sherry-yuan Good question. If a kernel is stalled because the printf buffer is full (runtime doesn't seem to stall for other reason, even it does, it wouldn't matter here), it will print the rest of the printf buffer which hasn't been dumped by the debug dumps and clear the buffer.