Investigate way to prove events ingested with & without shipper are functionally the same

[leehinman]

We need an automated way to verify that events ingested with or without the shipper are functionally equal. Currently we are missing @timestamp when using the shipper but other interesting areas are timestamp formatting, floating point numbers, numbers at 32 & 64 bit max, etc.

[pierrehilbert]

To split in two steps: one to handle in sprint 8 (manual testing) and one in sprint 9

[leehinman]

manual inspection when using the "Custom Log Package" shows that only the following fields are different between events that are ingested with or without the shipper

• _source.@timestamp
• _source.agent.ephemeral_id
• _source.agent.id
• _source.elastic_agent.id

We would expect these fields to be different because the events were ingested at different times and by different installations of elastic-agent which would result in a different id.

Also POC of code to check this in an automated fashion exists at https://github.com/leehinman/docdiff and running that utility on a variety of documents show the same thing.

[leehinman]

added #290 so we can add this as an e2e test when those tests are available. And closing for now since we have a way to check until the e2e tests exist.