Merge in OSSEC repo #1668
Merge in OSSEC repo #1668
Conversation
build generic ossec-server and ossec-agent deb packages.
OSSEC updated their download repository and now they do support HTTPS on the download page. They still do not provide a signed download just the MD5 and SHA1 checksums. The download url and download names also changed now that OSSEC is hosted on github. * Updated default values needed in the ansible playbook for OSSEC version 2.8.2 * Updated ansible tasks to unpack the download to reflect the new naming convention. * Updated README with notes about determining the `download_checksum_sha256` and other build processes. * Updated var names to more accurately describe what they are used for.
Updated OSSEC readme and defaults
Updated readme adding directions for updating some files not managed by ansible. Updated the files not managed by ansible with new version numbers. TODO: update readme with process for updating changelog. TODO: update the changelog with info from upstream release notes.
Updated readme and more version numbers
Ensuring that the version can only be supplied in one location. Removing unused hash.
Ensures that the build process doesn't mask failures by reporting success back to calling commands, e.g. the shell. Note that regardless of exit code for the failed subtask (`vagrant up` most likely), the fail() call will always exit 1.
Hardcodes the version at "2.8.2" to avoid surprises. Supports adding in newer versions, but haven't tested that yet.
The `get_url` module supports SHA256 checksums, but the OSSEC maintainers provide only MD5 and SHA1 (?!). Since we don't have a SHA256 sum to check against, we'll download the file, and verify against both MD5 and SHA1 checksums, failing loudly if there's a mismatch with either.
The `version` variable was ambiguous, so replacing with a namespaced `ossec_version` var instead.
The Rakefile implementation for looking up checksums was helpful, but bundling a custom library for performing the lookups automatically when the role is run is even better. Updates README with new provisioning steps. The info in the "Examples" section should be updated when the vars are renamed throughout the role.
Fetches OSSEC URL and checksum info via Ansible
If not running playbook under local vagrant box, copy over current repo root folder to the remote server.
Useful in situations when not running playbook under local vagrant environment. Only kicks off when `local_build_path` path var is overwritten.
Loop variables need to utilize jinja syntax.
Over in the SecureDrop repository, we updated the base boxes for all machines (build, dev, staging, etc.) to the Bento Trusty boxes [0]. Matching the setup here, so that we're building on a consistent platform, which closely matches the expected OS on production instances. [0] #1378
Most FPFers have forsaken virtualbox for libvirt/qemu, and now everything is roses. This change still supports virtualbox, and in fact preserves it as default provider, but also permits building with libvirt support now.
Tweaks needed to build packages on remote server
These play-level vars should have been role defaults from the beginning, but they're no longer used, so removing them from the repo.
The role requires these vars, but they're not available within the context of the role. Let's move the vars into defaults, so they can be sanely overridden at the site level. Doing so will keep the playbook clean and neat in preparation for merging into the SecureDrop repo.
Reuse the apt cache for up to an hour, then refresh after that point. Don't refresh the apt cache on every build run, since it's highly unlikely that packages have changed. Additionally, use state=present for packages, rather than state=latest, since the latter forces an apt cache update every time.
Ansible is already smart enough to skip redownloading files via `get_url` if the destination file exists. One must declare the fullpath to the dest file, but we're already doing that. Unfortunately Ansible is *not* smart enough to update dest files in place if the checksum doesn't match, but that's why we have a hard fail on a subsequent task, to handle the checksum verification.
Keeping all the build logic encapsulated within the targeted role allows us to reuse the build playbook.
|
Cleaned up some of the OSSEC build logic in freedomofpress/ossec#11, so I'll pull that in here. Will also shove the included roles into the existing |
Every use of the `ossec_build_dir` var was prefixed by `/tmp/` within the Ansible tasks, so let's just move that prefix into the var value itself. Doing so will enable overrides to customize the build directory being used, not just the subdirectory within /tmp.
Now building all packages under /tmp/build/, which is consistent with the targeting for other Debian packages created by the `build` VM. Also switching to synchronize strategy by default, rather than using the local Vagrant directory.
Previously we were using the /vagrant shared directory by default, and trying conditionally to use the sychronize module in CI. Instead let's just use synchronize and fetch always, to simplify the build logic and consistently support a variety of build environments, including CI and local development in VMs.
Matches the location of all other Debian packages maintained in this repository. Previously we had the OSSEC source packages in `install_files/ossec-packages`, mostly as an artifact of subtreeing in the old freedomofpress/ossec repository. Now that the OSSEC source build logic is integrated with the SD Build VM, we it makes sense to reorganize the directory structure so all package source trees are in a single location.
Extremely minimal; just checking for the newly installed apt dependencies. We can reuse most of the functional checks in the other deb package logic. Mostly reusing the existing logic in the config test suite for the "build" VM. All those checks pass without modification, save one: the "Homepage" control field check. Added branching logic to confirm a different value specifically for the OSSEC source packages: * ossec-agent * ossec-server The other packages remain with the securedrop.org homepage tested.
conorsch
force-pushed
the
bring-in-OSSEC-repo
branch
from
6c0a3a4 to
e0eace7
Compare
|
Pulled in the external repo entire, including commit history, via Simplified the logic for fetching back the built deb packages: now the role always uses the |
| # Ansible v1 does not support the `use_ssh_args`, which was added in 2.0. | ||
| # When resolving #1146, to upgrade Ansible, this option should be | ||
| # activated to permit builds in CI. | ||
| # use_ssh_args: yes |
We were previously lazily requiring developers to build the OSSEC source deb packages (not the metapackages, which have always been maintained in this repo) in a separate working directory, then copy them into the SecureDrop repo. That manual workflow was required in order to use the staging environment, which is hardly a desirable state of affairs. Now that we build even the OSSEC source packages inside the securedrop repo, we no longer need to warn developers about this shortcoming. Removed the pleasant failure message from the `install-local-packages` role, and removed the tipbox from the documentation.
|
Hmm, I got a failure while provisioning the build VM on |
|
Drat, I'm guessing you have an older version of rsync than I do—I'm using to the |
|
Yep I am using rsync version 2.6.9. Your suggestion of using Ansible 2.2.1 worked - I was able to successfully provision a build VM. 🤘 |
|
Ah good, your rsync option also worked. So that's probably a good solution for now in order to avoid what the pros call "mission creep". |
|
Good to know both options (upgrading Ansible and explicitly setting Confirmed the |
|
Confirming that Ansible 2.2.1 and the |
|
Confirmed the |
The `synchronize` module shipped with Ansible v1 does not intelligently handle privilege escalation with rsync 2.6.9. In order to support both Ansible v1 and v2 (pending resolution of #1146), and both rsync v2.6.9 (macOS) and rsync v3.1.2 (Debian Stretch), let's explictly declare a strategy for setting the sudo privileges on the rsync task. Full discussion of this problem can be found in comments for #1668.
The flow of building the ossec package is tightly integrated with ossec
deployment and testing. We debated utilizing submoduling it in, but decided
it would make more sense to just bring it in and mange in this repo from here on.
Status
Ready for review
Description of Changes
Fixes #1468 and sprinkles in changes from freedomofpress/ossec#10 .
Changes proposed in this pull request:
Well the big change is that the ossec repo files are brought in directly under the securedrop repo. Second, there have been some changes to assist in building and fetching back ossec package on a remote server. This was needed for CI work.
Testing
Ehh - really needs to be tested as part of #1666 but I guess you could confirm local vagrant up is still working.Though honestly I havent messed with that part yet :|
Deployment
Any special considerations for deployment? Consider both: