Move logging to a ring buffer approach

[ripienaar]

Proposed change

Today we see cases where if the logging output device blocks the entire server blocks since we are working on the assumption that logging always works and so we do it in the server hot path.

We should create a ring buffer where logs go to first and then the buffer is logged to the device in a background process.

While doing this we have some opportunities to fix related issues/requests:

• The ring buffer should support a set of fields for each log entry, this should indicate account, connection, stream, consumer etc that produce the logs, we can then log these as structured logs if configured
• The ring buffer should be queryable over an API with filters for above fields
• We should also evaluate if this buffer can be leveraged to support the ability to place only a certain account/client/stream/gateway/route into debug/trace level at run time via an API

Use case

To improve server stability as well as increased visibility into behaviours with a smaller impact vs running the entire server at debug/trace

Contribution

No response