Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Load CDK BulkLoad interface + MSSQL V2 Usage #55671

Draft
wants to merge 1 commit into
base: jschmidt/s3v2/s3-uses-new-interface
Choose a base branch
from

Conversation

johnny-schmidt
Copy link
Contributor

What

Migrates MSSQL to the new LoadPipeline CDK interface.

Practical implication: you can now do the upload-to-azure part in parallel while still limiting the load-into-mssql step to 1 thread per stream.

  • creates a BulkLoad LoadStrategy to support it
  • some other modifications to make it actually work
  • BulkLoad ITs are all passing
  • injection is hacked in -- some Bulk Load stuff will need to be behind a custom Condition that checks the config (so StandardInsert is probably broken until that)
  • I haven't done any performance testing/tuning yet

Copy link

vercel bot commented Mar 10, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 10, 2025 0:32am

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues CDK Connector Development Kit connectors/destination/mssql-v2
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants