Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

elk7: 7.11.1 -> 7.16.1, 6.8.3 -> 6.8.21 + add filebeat module and tests #150879

Merged
merged 7 commits into from
Dec 16, 2021

Conversation

talyz
Copy link
Contributor

@talyz talyz commented Dec 15, 2021

Motivation for this change
  • Update ELK stack versions 6 and 7 to their respective latest releases, which include fixes for CVE-2021-44228, see also CVE-2021-44228 (Apache log4j) tracking issue #150288.
  • Remove the journalbeat7 package, since journalbeat has been removed from upstream.
  • Add a filebeat module to replace journalbeat's functionality. I'm currently not interested in using it myself, but I've tested basic functionality and added it to the test.
  • Fix various issues in the tests, hopefully making them more robust. With elasticsearch7, lots of checks that shouldn't have passed did, because the wrong object was queried with jq.
  • Fix the parsedmarc tests.
Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 22.05 Release Notes (or backporting 21.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

@github-actions github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Dec 15, 2021
@github-actions github-actions bot added 8.has: changelog 8.has: documentation This PR adds or changes documentation labels Dec 15, 2021
@happysalada
Copy link
Contributor

happysalada commented Dec 15, 2021

I don't use es at the moment, but it looks good to me.
Thank you for doing this!

- Use comparisons in jq instead of grepping
- Match for `.hits.total.value` if version >= 7, otherwise it always
  passes
- Make curl fail if requests fails
Filebeat is an open source file harvester, mostly used to fetch logs
files and feed them into logstash.

This module can be used instead of journalbeat if used with
`filebeat7` and configured with the `journald` input.
- Don't use the deprecated elasticsearch7-oss package
- Improve jq query robustness and add tracing
@talyz
Copy link
Contributor Author

talyz commented Dec 16, 2021

The elasticsearch plugins that had to be marked broken have now released compatible versions, so I've updated them and unmarked them.

@talyz talyz added the 1.severity: security Issues which raise a security issue, or PRs that fix one label Dec 16, 2021
@happysalada happysalada mentioned this pull request Dec 16, 2021
12 tasks
@happysalada
Copy link
Contributor

I'm personally happy to merge anytime. If you want to be conservative, waiting until Monday for feedback makes sense. (I'm suggesting this because I don't think this PR will get a lot of attention, but I might be wrong).

The latest version includes a fix for CVE-2021-44228.
Avoid unnecssary back-off when elasticsearch is running on the same
host.
@talyz talyz changed the title elk7: 7.11.1 -> 7.16.1 + add filebeat module and tests elk7: 7.11.1 -> 7.16.1, 6.8.3 -> 6.8.21 + add filebeat module and tests Dec 16, 2021
@talyz
Copy link
Contributor Author

talyz commented Dec 16, 2021

Updated ELK 6 to the latest release, too.

@happysalada Sure, merging ASAP sounds good to me, since it's a critical security fix. I'll backport it to 21.11 when it's merged.

@happysalada happysalada merged commit ebaa226 into NixOS:master Dec 16, 2021
@talyz talyz deleted the elk-7.16.1 branch December 16, 2021 15:22
@talyz talyz restored the elk-7.16.1 branch December 16, 2021 15:29
talyz added a commit to talyz/nixpkgs that referenced this pull request Dec 16, 2021
… and tests (NixOS#150879)"

This reverts commit ebaa226 which was
a squash of multiple commits that shouldn't have been squashed.
@talyz talyz deleted the elk-7.16.1 branch December 16, 2021 15:39
@talyz
Copy link
Contributor Author

talyz commented Dec 16, 2021

@happysalada Did you do a "squash and merge"? All the commits were squashed, PGP signatures removed and author replaced. This messes up the history, makes backporting harder and, in this case, combined unrelated changes into one commit. I fixed it in #150979, so no worries, but if you want stuff squashed in other PRs, just tell the author :)

@talyz talyz mentioned this pull request Dec 16, 2021
5 tasks
@happysalada
Copy link
Contributor

ah my bad, by default it's a squash and merge and I didn't pay attention. Not intended, sorry for the trouble!

@talyz
Copy link
Contributor Author

talyz commented Dec 16, 2021

Ah, no worries. That's a pretty strange default action, but github does weird stuff sometimes 🤷

@happysalada
Copy link
Contributor

if you choose squash and merge for someone, github will remember and set it as your default for the next one.
I won't forget that this happens!

@talyz
Copy link
Contributor Author

talyz commented Dec 16, 2021

Oh, I see. Good to know!

jsoo1 pushed a commit to awakesecurity/nixpkgs that referenced this pull request Jun 1, 2023
… and tests (NixOS#150879)"

This reverts commit ebaa226 which was
a squash of multiple commits that shouldn't have been squashed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1.severity: security Issues which raise a security issue, or PRs that fix one 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: changelog 8.has: clean-up 8.has: documentation This PR adds or changes documentation 8.has: module (update) This PR changes an existing module in `nixos/` 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants