fix: improve error handling, and more!

[MicaiahReid]

This PR introduces a few fixes in an effort to improve reliability and debugging problems when running Chainhook as a service:

• Revisits log levels throughout the tool (fixes [service] Revisit log levels #498, fixes Classify & format panic logs correctly #521). The general approach for the logs were:
  • crit - fatal errors that will crash mission critical component of Chainhook. In these cases, Chainhook should automatically kill all main threads (not individual scanning threads, which is tracked by add the ability to abort an initial scan if a predicate is deregistered #404) to crash the service.
  • erro - something went wrong the could lead to a critical error, or that could impact all users
  • warn - something went wrong that could impact an end user (usually due to user error)
  • info - control flow logging and updates on the state of all registered predicates
  • debg - updates on the state of a predicate
• Crash the service if a mission critical thread fails (see Identify mission critical threads, and force process termination when they crash #517 (comment) for a list of these threads). Previously, if one of these threads failed, the remaining services would keep running. For example, if the event observer handler crashed, the event observer API would keep running. This means that the stacks node is successfully emitting blocks that Chainhook is acknowledging but not ingesting. This causes gaps in our database Fixes Identify mission critical threads, and force process termination when they crash #517
• Removes an infinite loop with bitcoin ingestion, crashing the service instead: Fixes Potential infinite loop in bitcoin block ingestion #506
• Fixes how we delete predicates from our db when one is deregistered. This should reduce the number of logs we have on startup. Fixes [service] deleting an interrupted predicate does not actually remove it from redis db #510
• Warns on all reorgs. Fixes Emit warnings for re-orgs (Bitcoin + Stacks) #519

[csgui]

Changes are looking good.

Only some changes requested due to the usage of the format! macro.

[csgui]

LGTM! Thanks for addressing the change requests.

[CharlieC3]

erro - something went wrong the could lead to a critical error, or that could impact all users

@MicaiahReid Just briefly reading this, but is this a typo? erro isn't a term recognized by Grafana, so it won't be properly formatted in our logs by default. I mentioned this in the referenced issue here, but I may have not called it out enough.

[csgui]

@MicaiahReid @CharlieC3 Just chiming in!
Where are you reading that erro? I saw (in someplace that I can't recall right now) that a logger was truncating the error word to erro.

[CharlieC3]

Where are you reading that erro?

It's in the PR description. Admittedly I haven't gone very deep in the code to verify, but though it may be best just to verify directly you or Micaiah :)

[MicaiahReid]

@CharlieC3
The logging library (slog that Chainhook (as well as the devnet API and I believe a few of our other rust-based tools) uses truncates the log type to four characters (ERROR => ERRO, DEBUG => DEBG).

From what I've seen, Grafana hasn't ever had a problem recognizing these as errors. The panic message referenced in #521 wasn't using this library, so Grafana wasn't picking it up. Now all logs are done through slog.

I was curious if the CRIT log level would be a problem - slog doesn't have a FATAL option.

[CharlieC3]

Grafana hasn't ever had a problem recognizing these as errors

haha that is because of a quick fix I put up in our log collection 😄 If there's no way around this I'm happy to keep it in, but if it's something easy to configure, I'd rather the logging be compatible with Grafana by default without needing extra configuration at the infra level.

[csgui]

Quick investigation on that:

Here is where it defines the log level constants: https://github.com/slog-rs/slog/blob/master/src/lib.rs#L2112

And it seems that the short naming is the default: https://github.com/slog-rs/slog/blob/master/src/lib.rs#L2291. I've never used slog, but maybe/probably, there is a formatter where that info can be defined.

[CharlieC3]

Seems like it's possible to configure slog to use Grafana-compatible logging levels using slog.Leveler like so: https://betterstack.com/community/guides/logging/logging-in-go/#creating-custom-log-levels

Again, I'm somewhat flexible on this if it's too difficult at a service level, but I think aligning ourselves to what Grafana categorizes by default is an easy way to get compatibility across all our services and fall under a new standard for Hiro without the need for one-offs for individual services at an infra level.

[MicaiahReid]

Cool @CharlieC3, I didn't realize we had custom patches that were required to make our logging work. I've created this PR in the logging library that is used by a few of our tools - this will just print the full log level prefix so that you won't need to have these patches. (granted, it will require users of this lib to upgrade + use the new feature, but Chainhook will be the first on that front!)

[CharlieC3]

Nice! Thanks so much, seems like a perfect way to handle this.