Investigate the osquery watchdog (Due 2023-12)

[directionless]

A long long time ago (#95, #103) we disabled the osquery watchdog. In the intervening years, both osquery and launcher have improved, and it's likely the original issue is long resolved. We should test enabling it.

If we want the watchdog to also be aware of launcher, so to take launcher's resource utilization into account, we may need to patch osquery. I think if you dig into osquery, there's an option enable_extensions_watchdog which is used to control whether extensions are monitored. But this might only cover the extensions osquery starts. We might need to add some kind of option in here.

We might also need an option to monitor, but not kill, launcher. This would allow the watchdog to understand the utilization and try to kill running queries.

There's a lot of research to be done in here

slack conversation https://kolide.slack.com/archives/CGFJY1SP2/p1687964775228619

[directionless]

We need to ship this by the end of 2023

[RebeccaMahany]

Initial testing looks promising, but further refinement might be needed. SELECT * FROM hash WHERE path LIKE '/Users/%/%%'; ran for close to an hour on my Windows machine before being denylisted.

[RebeccaMahany]

Closed by #1460 -- this will be available in the next release after v1.2.1.

[RebeccaMahany]

We are seeing one person experience the osquery watchdog shut osquery down for reasons that appear to be unrelated to distributed/scheduled queries -- reopening this issue until we understand the issue and have addressed it.

[directionless]

Launcher 1.3 brings in support for the watchdog. It's off by default, but can be enabled by us server side. I think that's pretty solid, and we'll keep iterating.