perf(stream): optimize native pg sink #19688
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This stack of pull requests is managed by Graphite. Learn more about stacking. |
kwannoel
changed the title
kwannoel
force-pushed
the
12-04-add_prepared_sql_to_error_logs_add_slt_bench
branch
from
47a0411 to
1d408c5
Compare
kwannoel
changed the base branch from
main
to
12-06-feat_common_support_switching_from_pg_jdbc_to_pg_native_sinks
9 tasks
kwannoel
force-pushed
the
12-06-feat_common_support_switching_from_pg_jdbc_to_pg_native_sinks
branch
from
e01f6d2 to
0062ecd
Compare
kwannoel
force-pushed
the
12-04-add_prepared_sql_to_error_logs_add_slt_bench
branch
2 times, most recently
from
a195d55 to
068fee0
Compare
kwannoel
changed the base branch from
12-06-feat_common_support_switching_from_pg_jdbc_to_pg_native_sinks
to
graphite-base/19688
kwannoel
force-pushed
the
12-04-add_prepared_sql_to_error_logs_add_slt_bench
branch
from
068fee0 to
bfa8d33
Compare
kwannoel
force-pushed
the
graphite-base/19688
branch
from
e78d259 to
2e302e9
Compare
|
Bump |
chenzl25
approved these changes
Dec 23, 2024
src/connector/src/sink/postgres.rs
Outdated
Comment on lines
556
to
570
| let row_parameters: String = pk_indices | ||
| .iter() | ||
| .map(|j| { | ||
| format!( | ||
| "{} = ${}", | ||
| schema.fields()[*j].name, | ||
| i * number_of_pk + j + 1 | ||
| ) | ||
| }) | ||
| .collect_vec() | ||
| .join(" AND "); | ||
| format!("({row_parameters})") | ||
| }) | ||
| .collect_vec() | ||
| .join(" AND "); | ||
| .join("OR"); |
There was a problem hiding this comment.
Maybe we can use PK In (...) instead of PK = xxx or PK = yyy which is more compacted and easy to be recognized by the external system.
There was a problem hiding this comment.
Improve later. Merge to let user try it. #19913
kwannoel
force-pushed
the
graphite-base/19688
branch
from
2e302e9 to
81f651f
Compare
kwannoel
force-pushed
the
12-04-add_prepared_sql_to_error_logs_add_slt_bench
branch
from
bfa8d33 to
4c34456
Compare
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
kwannoel
force-pushed
the
12-04-add_prepared_sql_to_error_logs_add_slt_bench
branch
from
59a512d to
c58d906
Compare
kwannoel
deleted the
12-04-add_prepared_sql_to_error_logs_add_slt_bench
branch
kwannoel
added a commit
that referenced
this pull request
Jan 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Just measure the throughput for both via grafana.
The performance improvement of unoptimized to optimized native pg sink went from 30r/s to 10K r/s.
The performance is around the same as JDBC. LHS is for native pg sink. RHS is for jdbc sink. Toggle the screenshot below.
screenshot
Implementation details:
updateInserthandling path, since compaction will normalizzeupdateInserttoInsert.LogSinkerinstead ofSinkWriterto avoid redundant flushes to the log store.Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.