Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partition json table by _ingestion_id #1446

Merged
merged 10 commits into from
Aug 22, 2022
Merged

Conversation

bmitchinson
Copy link
Member

@bmitchinson bmitchinson commented Aug 15, 2022

Ticket Link #792

Description

  • Ensure that _ingestion_id is after _extract_start_time in schema to allow for partitioning
  • Partitions the json table by _ingestion_id
    • Updates s3Writer to support partition_value and partition_key options
  • Move tests from unit/integration to unit to avoid confusion

Reminders:

  • If changing elixir code in an "app", did you update the relevant version
    in mix.exs?
  • If altering an API endpoint, was the relevant postman collection updated?
    • If a new version of smart_city is being used (new fields on a struct), were the relevant postman collections updated?

@bmitchinson bmitchinson marked this pull request as ready for review August 15, 2022 21:15
@bmitchinson bmitchinson changed the title WIP: Partition json table by _ingestion_id Partition json table by _ingestion_id Aug 22, 2022
@bmitchinson bmitchinson merged commit b64e4a5 into master Aug 22, 2022
@bmitchinson bmitchinson deleted the 792-partition-json-table branch August 22, 2022 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants