Add scheduled task to clean up old files from workspace #16247
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jdpgrailsdev
reviewed
Sep 2, 2022
| @@ -2,7 +2,7 @@ ARG JDK_VERSION=19-slim-bullseye | |||
| ARG JDK_IMAGE=openjdk:${JDK_VERSION} | |||
| FROM ${JDK_IMAGE} AS cron | |||
|
|
|||
| ARG VERSION=0.40.0-alpha | |||
| ARG VERSION=0.40.3 | |||
There was a problem hiding this comment.
Just curious if we know why this didn't get bumped by @evantahler's PR. Do we need to add airbyte-cron to a script somewhere to ensure it gets updated?
There was a problem hiding this comment.
@gosusnp found the problem! .bumpversion.cfg is what the publish command uses to know which files to bump. This wasn't in there.
There was a problem hiding this comment.
Yep, that should be the issue.
jdpgrailsdev
reviewed
Sep 2, 2022
| .map(Path::toFile) | ||
| .filter(f -> new AgeFileFilter(oldestAllowed).accept(f)) | ||
| .forEach(file -> { | ||
| log.info("Deleting file: " + file.toString()); |
There was a problem hiding this comment.
Nit: this should probably be lowered to debug, as I suspect it will cause log file spam when we actually delete stuff. Perhaps replace it with a counter and a log message to indicate how many files were cleaned at the end.
letiescanciano
added a commit
that referenced
this pull request
Sep 5, 2022
* master: (47 commits) Add email to identify users analytics call (#16327) 🎉 Source Amazon Ads: improve `config.start_date` validation (#16191) Add comments about intermediate state emission (#16262) MySQL Source : Standardize spec.json for DB connectors that support log-based CDC replication (#16216) MSSQL Source : Standardize spec.json for DB connectors that support log-based CDC replication (#16215) Hide a bunch more destination with potential unsecure API access (#16320) Skip unit tests when run-tests is false (#16267) Hide Destination connections from UI (#16310) Add scheduled task to clean up old files from workspace (#16247) Source Google Analytics v4: Re-name google analytics connector (#16306) 🐛 Source Facebook Marketing: remove "end_date" from config if empty value (re-implement #16096) (#16222) Fix github action syntax (#16277) Re-name google analytics cionnectors (#16287) Bump Airbyte version from 0.40.3 to 0.40.4 (#16275) Hide ES and Redis destination connectors from Cloud (#16276) 15700 add tests for PokeAPI (#15701) Add ProtocolVersion to StandardDefs (#16237) 🪟 🔧 🧹 Migrate attempt `bytesSynced` to `totalStats.bytesEmitted` and cleanup `AttemptDetails` component (#16126) Improve behavior of password input field (#16011) Improve airbyte-metrics support in the Helm chart (#16166) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
We need a job to clean up old files from the workspace to keep it size under control.
Closes #15567
How
Add a scheduled micronaut task that cleans up old data.
We are currently using
TEMPORAL_HISTORY_RETENTION_IN_DAYSas configuration for older files.This should be improved once we have the temporal history clean up in place.
Tests
Local validation from a running docker instance.
All files older than 30days have been removed, workspace went from 450Mb to 45Mb.