Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

File CDK: add start_date config option #28136

Closed
clnoll opened this issue Jul 11, 2023 · 1 comment · Fixed by #28845
Closed

File CDK: add start_date config option #28136

clnoll opened this issue Jul 11, 2023 · 1 comment · Fixed by #28845
Assignees
@brianjlai
Labels
area/file-cdk team/extensibility

Comments

@clnoll
Copy link
Contributor

clnoll commented Jul 11, 2023 •
edited
Loading

The new file-based CDK does not offer the start_date configuration option, but it seems particularly useful and will also give us backwards compatibility with the existing S3 connector.

When start_date is set, we should only sync files whose last_modified is equal to or greater than that date.
@clnoll clnoll mentioned this issue Jul 11, 2023
Closed
@brianjlai
Copy link
Contributor

brianjlai commented Jul 11, 2023 •
edited
Loading

Additional notes: this should be pretty straightforward to add in and the InMemoryFilesSource has a sample Pydantic field defined that I was using for testing. It should hopefully be as simple as adding a new field to the AbstractFileBasedSpec which is part of this PR: https://github.com/airbytehq/airbyte/pull/28139/files#diff-350520aab49a744e977546bc24f69c609e1564e3078839395d0394d86d6d0d47R15-R27

@brianjlai brianjlai self-assigned this Jul 25, 2023
@brianjlai brianjlai mentioned this issue Jul 30, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brianjlai brianjlai

Labels
area/file-cdk team/extensibility
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add start_date to all file-based configs airbytehq/airbyte
2 participants
@clnoll @brianjlai