Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow DBID to be empty & Correct case matching #331

Merged
merged 6 commits into from
Sep 21, 2021
Merged

Conversation

Dylan-M
Copy link
Contributor

@Dylan-M Dylan-M commented Sep 15, 2021

The DBID field is able to be empty on some versions of Oracle DB
The multiline regex was looking for Audit File, but logs have Audit file

The DBID field is able to be empty on some versions of Oracle DB
The multiline regex was looking for `Audit File`, but logs have `Audit file`
@Dylan-M Dylan-M marked this pull request as draft September 15, 2021 18:14
@jsirianni jsirianni self-requested a review September 15, 2021 18:32
@Dylan-M Dylan-M marked this pull request as ready for review September 17, 2021 13:27
@Dylan-M Dylan-M self-assigned this Sep 17, 2021
@Dylan-M Dylan-M added the bug Something isn't working label Sep 17, 2021
@jsirianni
Copy link
Member

The following config does not work for me

pipeline:
- type: oracledb
  enable_audit_log: true
  enable_truncate_audit_action: false
  enable_alert_log: false
  enable_listener_log: true

  audit_log_path: "/home/jsirianni/git/log-library/library/oracledb/audit*"
  alert_log_path: "/home/jsirianni/git/log-library/library/oracledb/alert*"
  listener_log_path: /home/jsirianni/git/log-library/library/oracledb/list*

  start_at: beginning
  
- type: file_output
  path: ./out
{"level":"error","timestamp":"2021-09-20T11:01:37.051-0400","message":"Failed to build agent","error":{"description":"operator '$.oracledb.[$.file_output]' does not exist","details":{"operator_id":"$.oracledb.audit_regex_parser"}}}

If I do enable_truncate_audit_action: true, it starts fine.

Copy link
Member

@jsirianni jsirianni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get the following:

➜  stanza-plugins git:(dbid-empty) ✗ ./stanza_linux_amd64 --config ./config.yaml | jq .
{"level":"info","timestamp":"2021-09-20T11:04:10.630-0400","message":"Starting stanza agent"}
{"level":"info","timestamp":"2021-09-20T11:04:10.630-0400","message":"Stanza agent started"}
{"level":"info","timestamp":"2021-09-20T11:04:10.830-0400","message":"Started watching file","operator_id":"$.oracledb.listener_reader","operator_type":"file_input","path":"/home/jsirianni/git/log-library/library/oracledb/listener.log"}
{"level":"info","timestamp":"2021-09-20T11:04:10.831-0400","message":"Started watching file","operator_id":"$.oracledb.listener_reader","operator_type":"file_input","path":"/home/jsirianni/git/log-library/library/oracledb/listener.xml"}
{"level":"info","timestamp":"2021-09-20T11:04:10.831-0400","message":"Started watching file","operator_id":"$.oracledb.listener_reader","operator_type":"file_input","path":"/home/jsirianni/git/log-library/library/oracledb/listener2.log"}
{"level":"info","timestamp":"2021-09-20T11:04:10.831-0400","message":"Started watching file","operator_id":"$.oracledb.listener_reader","operator_type":"file_input","path":"/home/jsirianni/git/log-library/library/oracledb/listener_DB12C_log.xml"}
{"level":"info","timestamp":"2021-09-20T11:04:10.831-0400","message":"Started watching file","operator_id":"$.oracledb.listener_reader","operator_type":"file_input","path":"/home/jsirianni/git/log-library/library/oracledb/listener_DB19C_log.xml"}
{"level":"info","timestamp":"2021-09-20T11:04:10.831-0400","message":"Started watching file","operator_id":"$.oracledb.audit_reader","operator_type":"file_input","path":"/home/jsirianni/git/log-library/library/oracledb/audit.log"}
{"level":"info","timestamp":"2021-09-20T11:04:10.831-0400","message":"Started watching file","operator_id":"$.oracledb.audit_reader","operator_type":"file_input","path":"/home/jsirianni/git/log-library/library/oracledb/audit2.log"}
{"level":"error","timestamp":"2021-09-20T11:04:10.833-0400","message":"Failed to process entry","operator_id":"$.oracledb.audit_regex_parser","operator_type":"regex_parser","error":"regex pattern does not match","action":"send","entry":{"timestamp":"2021-09-20T11:04:10.833194324-04:00","severity":0,"labels":{"file_name":"audit.log","log_type":"oracledb.audit","plugin_id":"oracledb"},"record":"Sat Jun 27 16:07:30 2020 -04:00\nLENGTH : '5988'\nACTION :[5732] 'DECLARE\n  reloading_registry BOOLEAN := FALSE;\n  current_type NUMBER;\n  table_name_122 VARCHAR2(30);\n  constraint_name_122 VARCHAR2(30);\n\n"}}
{"level":"error","timestamp":"2021-09-20T11:04:10.833-0400","message":"Failed to process entry","operator_id":"$.oracledb.audit_action_restructurer","operator_type":"restructure","error":"evaluate value_expr: invalid operation: int(string) (1:7)\n | len($.action) > 150000 ? $.action[0:150000] + \"... Action has \" + $.length + \" characters which is over the 150000 characters limit, truncating action\" : $.action\n | ......^","action":"send","entry":{"timestamp":"2021-09-20T11:04:10.833194324-04:00","severity":0,"labels":{"file_name":"audit.log","log_type":"oracledb.audit","plugin_id":"oracledb"},"record":"Sat Jun 27 16:07:30 2020 -04:00\nLENGTH : '5988'\nACTION :[5732] 'DECLARE\n  reloading_registry BOOLEAN := FALSE;\n  current_type NUMBER;\n  table_name_122 VARCHAR2(30);\n  constraint_name_122 VARCHAR2(30);\n\n"}}
{"level":"error","timestamp":"2021-09-20T11:04:10.833-0400","message":"error while writing entry: evaluate value_expr: invalid operation: int(string) (1:7)\n | len($.action) > 150000 ? $.action[0:150000] + \"... Action has \" + $.length + \" characters which is over the 150000 characters limit, truncating action\" : $.action\n | ......^","operator_id":"$.oracledb.audit_regex_parser","operator_type":"regex_parser"}
{"level":"error","timestamp":"2021-09-20T11:04:10.834-0400","message":"Failed to process entry","operator_id":"$.oracledb.audit_regex_parser","operator_type":"regex_parser","error":"regex pattern does not match","action":"send","entry":{"timestamp":"2021-09-20T11:04:10.834316275-04:00","severity":0,"labels":{"file_name":"audit.log","log_type":"oracledb.audit","plugin_id":"oracledb"},"record":"Sat Jun 27 16:07:56 2020 -04:00\nLENGTH : '9090'\nACTION :[8834] 'CREATE OR REPLACE PACKAGE dbms_registry AS\n\n"}}
{"level":"error","timestamp":"2021-09-20T11:04:10.834-0400","message":"Failed to process entry","operator_id":"$.oracledb.audit_action_restructurer","operator_type":"restructure","error":"evaluate value_expr: invalid operation: int(string) (1:7)\n | len($.action) > 150000 ? $.action[0:150000] + \"... Action has \" + $.length + \" characters which is over the 150000 characters limit, truncating action\" : $.action\n | ......^","action":"send","entry":{"timestamp":"2021-09-20T11:04:10.834316275-04:00","severity":0,"labels":{"file_name":"audit.log","log_type":"oracledb.audit","plugin_id":"oracledb"},"record":"Sat Jun 27 16:07:56 2020 -04:00\nLENGTH : '9090'\nACTION :[8834] 'CREATE OR REPLACE PACKAGE dbms_registry AS\n\n"}}
{"level":"error","timestamp":"2021-09-20T11:04:10.834-0400","message":"error while writing entry: evaluate value_expr: invalid operation: int(string) (1:7)\n | len($.action) > 150000 ? $.action[0:150000] + \"... Action has \" + $.length + \" characters which is over the 150000 characters limit, truncating action\" : $.action\n | ......^","operator_id":"$.oracledb.audit_regex_parser","operator_type":"regex_parser"}
{"level":"error","timestamp":"2021-09-20T11:04:10.835-0400","message":"Failed to process entry","operator_id":"$.oracledb.audit_regex_parser","operator_type":"regex_parser","error":"regex pattern does not match","action":"send","entry":{"timestamp":"2021-09-20T11:04:10.835840606-04:00","severity":0,"labels":{"file_name":"audit.log","log_type":"oracledb.audit","plugin_id":"oracledb"},"record":"Sat Jun 27 16:09:23 2020 -04:00\nLENGTH : '42061'\nACTION :[41804] 'create or replace package drixmd authid definer is\n\n"}}
{"level":"error","timestamp":"2021-09-20T11:04:10.835-0400","message":"Failed to process entry","operator_id":"$.oracledb.audit_action_restructurer","operator_type":"restructure","error":"evaluate value_expr: invalid operation: int(string) (1:7)\n | len($.action) > 150000 ? $.action[0:150000] + \"... Action has \" + $.length + \" characters which is over the 150000 characters limit, truncating action\" : $.action\n | ......^","action":"send","entry":{"timestamp":"2021-09-20T11:04:10.835840606-04:00","severity":0,"labels":{"file_name":"audit.log","log_type":"oracledb.audit","plugin_id":"oracledb"},"record":"Sat Jun 27 16:09:23 2020 -04:00\nLENGTH : '42061'\nACTION :[41804] 'create or replace package drixmd authid definer is\n\n"}}
{"level":"error","timestamp":"2021-09-20T11:04:10.835-0400","message":"error while writing entry: evaluate value_expr: invalid operation: int(string) (1:7)\n | len($.action) > 150000 ? $.action[0:150000] + \"... Action has \" + $.length + \" characters which is over the 150000 characters limit, truncating action\" : $.action\n | ......^","operator_id":"$.oracledb.audit_regex_parser","operator_type":"regex_parser"}

@jsirianni jsirianni self-requested a review September 21, 2021 19:29
@jsirianni jsirianni merged commit b3a18b7 into master Sep 21, 2021
@jsirianni jsirianni deleted the dbid-empty branch September 21, 2021 19:53
schmikei added a commit that referenced this pull request Sep 28, 2021
* Update regex to parse IPv6 (#334)

Update default listener log path

* Add HAProxy Plugin (#335)

* Add haproxy plugin

* Add supported platforms and min stanza version

* PR Feedback fixes

* Rename frontend_name to frontend_name_transport in regex

* for all move operations, check if field is nil before moving. "set log type to haproxy and haproxy.error (not .http / .tcp)"

* typoe: nill --> nil

* typo, log_format: http --> default

Co-authored-by: jsirianni <joe.sirianni@bluemedora.com>

* Allow DBID to be empty & Correct case matching (#331)

* Allow DBID to be empty & Correct case matching

The DBID field is able to be empty on some versions of Oracle DB
The multiline regex was looking for `Audit File`, but logs have `Audit file`

* Switch to line end for multiline with double newline pattern

* Fix plugin failure when using inline truncate check

* Switch back to a regex parse for record splitting

Co-authored-by: jsirianni <joe.sirianni@bluemedora.com>

* Release 0.0.79 (#336)

* 0.0.79 changelog

* dbid oracle pr

* fix release date

* move frontend port to resources (#338)

* Add more checks to reduce errors (#337)

* Add more checks to reduce errors

* Add ac_lite_ap_parser change to changelog for ubiquiti

* 0.0.80 changelog

Co-authored-by: jsirianni <joe.sirianni@observiq.com>

* rebase the stanza-plugins changes

* fix haproxy

* fix ubiquiti

* fix labels rather than attributes on operator field

* oracledb attributes

* fix haproxy

* update regex to handle {} brackets before http request info (#342)

* update regex to handle {} brackets before http request info

* haproxy http default log format fix

* make change backwards compatible

* Adjust parsing further based on more detailed oracle db audit logs (#343)

* release 0.0.82

Co-authored-by: Dylan Myers <Dylan-M@users.noreply.github.com>
Co-authored-by: EricWHolt <39141134+ericwholt@users.noreply.github.com>
Co-authored-by: jsirianni <joe.sirianni@bluemedora.com>
Co-authored-by: jsirianni <joe.sirianni@observiq.com>
jsirianni pushed a commit that referenced this pull request Oct 5, 2021
* Update regex to parse IPv6 (#334)

Update default listener log path

* Add HAProxy Plugin (#335)

* Add haproxy plugin

* Add supported platforms and min stanza version

* PR Feedback fixes

* Rename frontend_name to frontend_name_transport in regex

* for all move operations, check if field is nil before moving. "set log type to haproxy and haproxy.error (not .http / .tcp)"

* typoe: nill --> nil

* typo, log_format: http --> default

Co-authored-by: jsirianni <joe.sirianni@bluemedora.com>

* Allow DBID to be empty & Correct case matching (#331)

* Allow DBID to be empty & Correct case matching

The DBID field is able to be empty on some versions of Oracle DB
The multiline regex was looking for `Audit File`, but logs have `Audit file`

* Switch to line end for multiline with double newline pattern

* Fix plugin failure when using inline truncate check

* Switch back to a regex parse for record splitting

Co-authored-by: jsirianni <joe.sirianni@bluemedora.com>

* Release 0.0.79 (#336)

* 0.0.79 changelog

* dbid oracle pr

* fix release date

* move frontend port to resources (#338)

* Add more checks to reduce errors (#337)

* Add more checks to reduce errors

* Add ac_lite_ap_parser change to changelog for ubiquiti

* 0.0.80 changelog

Co-authored-by: jsirianni <joe.sirianni@observiq.com>

* update regex to handle {} brackets before http request info (#342)

* update regex to handle {} brackets before http request info

* haproxy http default log format fix

* make change backwards compatible

* Adjust parsing further based on more detailed oracle db audit logs (#343)

* release 0.0.82

* CI Testing: End to End Tests (#345)

* end to end nginx testing

* fix format

* fix format

* use sudo to compare against files from container mount

* sleep so stanza can parse, kill stanza when done

* try cloning log library

* use token to clone lob lib

* fix repo name

* fix expect and output paths

* handle both nginx formats

* add apache_http workflow

* dump container log

* dump container log

* 10 second sleep

* use jq with diff to prevent formatting issues

* sudo

* cannot use sudo with redirection to diff, so just format with jq before using diff

* Switch back to diff, something else is going on..

* pause and cat raw output before comparing

* sudo

* fix paths

* sort before compare

* redirect output

* sort and cat

* use jtool for comparing json files

* use jtool for comparing json files

* chmod it

* haproxy workflow

* add oracledb workflow

* single test case for oracledb

* mount plugins dir

* stop and then get stanza logs

* sleep 20 seconds instead of 10, sometimes 10 is not enough

* fix log dirg

* install jtool in its own step

* fix mount

* split oracle up. start with alert logs

* oracle audit log

* upgrade jtool and use skip timestamp for haproxy and oracle

* upgrade jtoo

* upgrade jtool

* pause, stop, logs

* listener log, oracle

* Handle second {} in http log entry if present (#346)

Co-authored-by: jsirianni <joe.sirianni@observiq.com>

* Add tcpudp plugin (#341)

* Add tcpudp plugin

* Add tcpudp schema and tests

* Split into two plugins udp and tcp

* Add schema files for tests

* Update plugins/tcp.yaml

* Update plugins/tcp.yaml

* Update plugins/udp.yaml

* Update plugins/udp.yaml

* Update test/configs/tcp/invalid/invalid_listen_port.yaml

* Update plugins/udp.yaml

Co-authored-by: jsirianni <joe.sirianni@observiq.com>
Co-authored-by: Joseph Sirianni <joe.sirianni@bluemedora.com>

* tcp / udp: move  to message field (#347)

* move  to message field

* \n

* Add common event format plugin (#328)

* Add common event format plugin

* use key value parser for parsing extensions field

* Promote fields to labels and resources

* Promote fields to labels and resources

* Update changelog

* Remove key value parser

* Add promote device_vendor and device_version to resources

Co-authored-by: jsirianni <joe.sirianni@bluemedora.com>
Co-authored-by: jsirianni <joe.sirianni@observiq.com>

* release 0.0.83

* Add http plugin (#352)

* Add http plugin

* Update log_type label to http

* remove duplicate param

* tcp --> http

* typo

* token_header --> auth_header

* small refactor

* upgrade stanza 1.2.9

Co-authored-by: jsirianni <joe.sirianni@bluemedora.com>

* Update Titles and uwsgi field name (#350)

Co-authored-by: Joseph Sirianni <joe.sirianni@bluemedora.com>

* Update cisco_meraki plugin to use key_value_parser (#349)

* Update cisco_meraki plugin to use key_value_parser instead of custom regex

* use stanza 1.2.9

Co-authored-by: jsirianni <joe.sirianni@bluemedora.com>

* Create Sonicwall log parser plugin (#340)

* Create Sonicwall log parser plugin

* Add pri field severity_parser

* Rename msg field to message

* Add parameter location to support setting timezone

* Update to use udp_input and add extra tests

* Use stanza 1.2.7 for tests

* update stanza and get go.sum

* Update plugins/sonicwall.yaml

Co-authored-by: Joseph Sirianni <joe.sirianni@bluemedora.com>

* Fix using wrong parameter if you defined listen_port

Co-authored-by: jsirianni <joe.sirianni@observiq.com>
Co-authored-by: Joseph Sirianni <joe.sirianni@bluemedora.com>

* release-0.0.84

* fix ci link

* remove start_at reference

* remove start_at test for sonicwall, not needed

* enable new operators

* fix start_at for w3c tests due to delete_at_end being added

* Add cisco_catalyst plugin (#351)

* Add cisco_catalyst plugin

* Add severity field group to regex. Update parse from field for severity and regex.

* Remove parse_to message in udp_input and parse_from message in regex_parser

* fix ci link

* remove start_at reference

* remove start_at test for sonicwall, not needed

* enable new operators

* fix start_at for w3c tests due to delete_at_end being added

Co-authored-by: Joseph Sirianni <joe.sirianni@bluemedora.com>

* remove tests for now, not compatable with otel branch

* remove tests for now, not compatable with otel branch

* port cisco catalyst to otel

* fix cef

* fix haproxy

* port http

* port sonicwall

* fix haproxy

* Update plugins/cisco_catalyst.yaml

Co-authored-by: Keith Schmitt <32067685+schmikei@users.noreply.github.com>

* rebase oracledb

* try and fix severities that were missed

Co-authored-by: Dylan Myers <Dylan-M@users.noreply.github.com>
Co-authored-by: EricWHolt <39141134+ericwholt@users.noreply.github.com>
Co-authored-by: Keith Schmitt <32067685+schmikei@users.noreply.github.com>
Co-authored-by: schmikei <keith.schmitt@bluemedora.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants