Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Log Group Cleanup to Clean up Action #1575

Merged
merged 19 commits into from
Mar 4, 2025
Merged

Added Log Group Cleanup to Clean up Action #1575

merged 19 commits into from
Mar 4, 2025

Conversation

okankoAMZ
Copy link
Contributor

@okankoAMZ okankoAMZ commented Feb 27, 2025

Description of Changes

This pull request introduces a new job to the GitHub Actions workflow for cleaning up old AWS CloudWatch log groups. The key changes include:

Workflow Changes

  • File: .github/workflows/clean-aws-resources.yml
    • Added a new job clean-log-groups to clean old CloudWatch log groups.

New Feature

  • File: tool/clean/clean_log_group/clean_log_group.go
    • Implemented a new Go script to clean old CloudWatch log groups.
    • The script deletes log groups that are older than a specified threshold and have been inactive for a defined period. By default is will delete any log group created 3 days ago which have not received a log in a day.
    • It also has a exception list that if a string there is found in a log group name it will ignore it.
    • Configurable via command-line flags for threshold days, inactive days, and dry-run mode.

Dependency Updates

  • File: tool/clean/go.mod
    • Updated Go version to 1.22 and added a toolchain for Go 1.23.6.
    • Upgraded dependencies including github.com/aws/aws-sdk-go-v2 and added new dependencies for cloudwatchlogs.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

  • Verified workflows on GitHub Actions: Latest Run details
  • Added unit tests and ran: go test ./clean_log_group -v
\=== RUN   TestGetLastLogEventTime
    clean_log_group_test.go:84: PASS:   DescribeLogStreams(string,*cloudwatchlogs.DescribeLogStreamsInput,string)
    clean_log_group_test.go:84: PASS:   DescribeLogStreams(string,*cloudwatchlogs.DescribeLogStreamsInput,string)
--- PASS: TestGetLastLogEventTime (0.00s)
=== RUN   TestHandleLogGroup
=== RUN   TestHandleLogGroup/expired-test-log-group
2025/03/03 12:15:05 🚨 Worker: 1| Old & Inactive Log Group: expired-test-log-group (Created: 57121-08-18 15:37:59 -0400 EDT, Last Event: 57121-08-18 15:37:59 -0400 EDT)
2025/03/03 12:15:05 🛑 Dry-Run: Would delete log group: expired-test-log-group
2025/03/03 12:15:05 🔍 Processed 1 log groups so far
=== RUN   TestHandleLogGroup/fresh-test-log-group
=== RUN   TestHandleLogGroup/old-test-log-group
--- PASS: TestHandleLogGroup (3.00s)
    --- PASS: TestHandleLogGroup/expired-test-log-group (1.00s)
    --- PASS: TestHandleLogGroup/fresh-test-log-group (1.00s)
    --- PASS: TestHandleLogGroup/old-test-log-group (1.00s)
=== RUN   TestDeleteLogGroups
=== RUN   TestDeleteLogGroups/expired-test-log-group
2025/03/03 12:15:08 👷 Creating 15 workers
2025/03/03 12:15:08 🔍 Described 0 times | Found 1 log groups
2025/03/03 12:15:08 🚨 Worker: 1| Old & Inactive Log Group: expired-test-log-group (Created: 57121-08-18 16:28:04 -0400 EDT, Last Event: 57121-08-18 16:28:04 -0400 EDT)
2025/03/03 12:15:08 🛑 Dry-Run: Would delete log group: expired-test-log-group
2025/03/03 12:15:08 🔍 Processed 1 log groups so far
=== RUN   TestDeleteLogGroups/fresh-test-log-group
2025/03/03 12:15:08 👷 Creating 15 workers
2025/03/03 12:15:08 🔍 Described 0 times | Found 1 log groups
=== RUN   TestDeleteLogGroups/old-test-log-group
2025/03/03 12:15:08 👷 Creating 15 workers
2025/03/03 12:15:08 🔍 Described 0 times | Found 1 log groups
=== RUN   TestDeleteLogGroups/exceptional-test-log-group
2025/03/03 12:15:08 👷 Creating 15 workers
2025/03/03 12:15:08 🔍 Described 0 times | Found 1 log groups
--- PASS: TestDeleteLogGroups (0.00s)
    --- PASS: TestDeleteLogGroups/expired-test-log-group (0.00s)
    --- PASS: TestDeleteLogGroups/fresh-test-log-group (0.00s)
    --- PASS: TestDeleteLogGroups/old-test-log-group (0.00s)
    --- PASS: TestDeleteLogGroups/exceptional-test-log-group (0.00s)
PASS
ok      github.com/aws/amazon-cloudwatch-agent/tool/clean/clean_log_group       3.211s

Requirements

Before committing the code, please complete the following steps:

  1. Run make fmt and make fmt-sh.
  2. Run make lint.

@okankoAMZ okankoAMZ marked this pull request as ready for review February 27, 2025 17:54
@okankoAMZ okankoAMZ requested a review from a team as a code owner February 27, 2025 17:54
Copy link
Contributor

@the-mann the-mann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is huge for cost savings! would be nice to see some unit tests here, but other than that look sgood

@okankoAMZ okankoAMZ added the github_actions Pull requests that update GitHub Actions code label Mar 3, 2025
@okankoAMZ okankoAMZ force-pushed the clean-log-group branch 2 times, most recently from 0b86f64 to 9bdc496 Compare March 3, 2025 17:29
jefchien
jefchien previously approved these changes Mar 3, 2025
jefchien
jefchien previously approved these changes Mar 3, 2025
Copy link
Contributor

@Paramadon Paramadon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@okankoAMZ okankoAMZ merged commit ef35ee7 into main Mar 4, 2025
7 checks passed
@okankoAMZ okankoAMZ deleted the clean-log-group branch March 4, 2025 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
github_actions Pull requests that update GitHub Actions code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants