feat: Add Zookeeper Plugin

[dmikolay]

Proposed Change

• Added a logging plugin for apache zookeeper
• Tested using the file located here: https://github.com/observIQ/log-library/blob/master/library/hbase/zookeeper.log
• Example Log:

LogRecord #32
ObservedTimestamp: 2022-06-16 19:53:47.538867 +0000 UTC
Timestamp: 1970-01-01 00:00:00 +0000 UTC
Severity: 
Body: 2020-12-09 20:38:42,979 INFO  [main] server.ZooKeeperServer: Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
Attributes:
     -> log.file.name: STRING(zookeeper.txt)
Trace ID: 
Span ID: 
Flags: 0

Checklist

• Changes are tested
• CI has passed

[BinaryFissionGames]

That example parse doesn't look like it's working right; I'd expect to see the severity + timestamp parsed, and also a few attributes on the LogRecord like message and source.

Maybe we need to take some of the changes that were being made in observIQ/stanza-plugins#399? Seems like there is a regex there for messages w/ no ID, which seems to be the problem with this data.

I also wonder if the log format ends up being different for "standalone" zookeeper, as opposed to when it's run in conjunction with hbase/kafka. Seems strange that we didn't account for this in the original plugin.

[cpheps]

@BinaryFissionGames and @dmikolay

That example parse doesn't look like it's working right; I'd expect to see the severity + timestamp parsed, and also a few attributes on the LogRecord like message and source.

I ran into this when working on ops agent. If the node is part of a cluster it'll have the myid:[0-9]+ before the severity. If it's not that field will be missing. We'll likely need a router unless we want to get extra clever with our regexes.

Here's what I did in ops-agent https://github.com/GoogleCloudPlatform/ops-agent/blob/master/apps/zookeeper.go#L76-L120

[dmikolay]

After adding a router and making regex changes, here's a new example of the plugin at work

This line in the test file:

2022-01-31 17:51:45,451 [myid:1] - INFO  [NIOWorkerThread-3:NIOServerCnxn@514] - Processing mntr command from /0:0:0:0:0:0:0:1:50284

Produced this example log:

LogRecord #0
ObservedTimestamp: 2022-06-20 19:23:17.086837 +0000 UTC
Timestamp: 2022-01-31 22:51:45.451 +0000 UTC
Severity: Info
Body: 2022-01-31 17:51:45,451 [myid:1] - INFO  [NIOWorkerThread-3:NIOServerCnxn@514] - Processing mntr command from /0:0:0:0:0:0:0:1:50284
Attributes:
     -> log.file.name: STRING(zookeeper.txt)
     -> thread: STRING(NIOWorkerThread-3)
     -> message: STRING(Processing mntr command from /0:0:0:0:0:0:0:1:50284)
     -> zookeeper_severity: STRING(INFO)
     -> source: STRING(NIOServerCnxn)
     -> line: STRING(514)
     -> timestamp: STRING(2022-01-31 17:51:45,451)
     -> myid: STRING(1)
     -> log_type: STRING(zookeeper)
Trace ID: 
Span ID: 
Flags: 0