Skip to content
This repository has been archived by the owner on Apr 2, 2024. It is now read-only.

Ingest metric data with COPY to get better performance #1462

Merged
merged 1 commit into from
Jul 8, 2022
Merged

Ingest metric data with COPY to get better performance #1462

merged 1 commit into from
Jul 8, 2022

Conversation

niksajakovljevic
Copy link
Contributor

@niksajakovljevic niksajakovljevic commented Jun 29, 2022 •
edited
Loading

We try COPY into metric table. If there is a conflict situation
(eg. duplicate key violation) we catch it on client and retry
by COPYing to temporary table and doing ON CONFLICT insert from there.

@niksajakovljevic niksajakovljevic requested a review from a team as a code owner June 29, 2022 15:07
@niksajakovljevic niksajakovljevic self-assigned this Jun 29, 2022
@niksajakovljevic niksajakovljevic changed the title Ingest metric data with COPY to get better performance [WIP] Ingest metric data with COPY to get better performance Jun 30, 2022
@niksajakovljevic niksajakovljevic force-pushed the niksa/ingest-with-copy branch 3 times, most recently from f75178c to a42238a Compare July 4, 2022 12:39
@niksajakovljevic niksajakovljevic changed the title [WIP] Ingest metric data with COPY to get better performance Ingest metric data with COPY to get better performance Jul 4, 2022
@niksajakovljevic niksajakovljevic force-pushed the niksa/ingest-with-copy branch from a42238a to 7fe0b84 Compare July 6, 2022 09:49
@niksajakovljevic
Copy link
Contributor Author

niksajakovljevic commented Jul 6, 2022 •
edited
Loading

Due to recent changes in the extension I needed to temporarily tweak EXTENSION to be able to run e2e tests. I believe long term we want to trigger e2e tests on the new branch from the extension side.
Also we might want to pin EXTENSION to master so we can test against most recent extension code.

@niksajakovljevic niksajakovljevic force-pushed the niksa/ingest-with-copy branch 14 times, most recently from e486765 to 47f919b Compare July 8, 2022 08:01
Harkishen-Singh
Harkishen-Singh suggested changes Jul 8, 2022
View reviewed changes
Copy link
Member

@Harkishen-Singh Harkishen-Singh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation looks good. We should have some end-to-end test, that can stimulate normal COPY ingest and with conflict.

@niksajakovljevic niksajakovljevic requested a review from a team as a code owner July 8, 2022 09:37
@niksajakovljevic
Copy link
Contributor Author

@Harkishen-Singh There are existing e2e tests for testing various duplicate scenarios (check create_test.go).

@niksajakovljevic niksajakovljevic force-pushed the niksa/ingest-with-copy branch from 8665cc0 to ce0c08a Compare July 8, 2022 10:56
@JamesGuthrie JamesGuthrie mentioned this pull request Jul 8, 2022
Merged
antekresic
antekresic approved these changes Jul 8, 2022
View reviewed changes
Copy link
Contributor

@antekresic antekresic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nit, looks good otherwise 🎉

@niksajakovljevic niksajakovljevic force-pushed the niksa/ingest-with-copy branch 2 times, most recently from 7e93d98 to 9cb1b3f Compare July 8, 2022 13:54
@niksajakovljevic
Ingest metric data with COPY to get better performance
38747c0 
We try COPY into metric table. If there is a conflict situation
(eg. duplicate key violation) we catch it on client and retry
by COPYing to temporary table and doing ON CONFLICT insert from there.

Prevent using old extension branch as docker image - I will revert this before merge
@niksajakovljevic niksajakovljevic force-pushed the niksa/ingest-with-copy branch from 9cb1b3f to 38747c0 Compare July 8, 2022 14:27
@niksajakovljevic niksajakovljevic merged commit 82a573d into master Jul 8, 2022
@niksajakovljevic niksajakovljevic deleted the niksa/ingest-with-copy branch July 8, 2022 15:05
@niksajakovljevic niksajakovljevic mentioned this pull request Jul 8, 2022
Closed
@peppercoffee peppercoffee added the IIP-1 Improve Ingestion Performance (part 1) label Jul 19, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@JamesGuthrie JamesGuthrie JamesGuthrie left review comments

@antekresic antekresic antekresic approved these changes

@Harkishen-Singh Harkishen-Singh Harkishen-Singh approved these changes

@arajkumar arajkumar Awaiting requested review from arajkumar

@alejandrodnm alejandrodnm Awaiting requested review from alejandrodnm

Assignees

@niksajakovljevic niksajakovljevic

Labels
IIP-1 Improve Ingestion Performance (part 1)
Projects
None yet
Milestone
0.13.0
Development

Successfully merging this pull request may close these issues.
5 participants
@niksajakovljevic @JamesGuthrie @antekresic @Harkishen-Singh @peppercoffee