Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

don't split lines on LSEP unicode characters when reading lines in destinations #3327

Merged
merged 9 commits into from
May 10, 2021
Merged

don't split lines on LSEP unicode characters when reading lines in destinations #3327

merged 9 commits into from
May 10, 2021

Conversation

jrhizor
Copy link
Contributor

@jrhizor jrhizor commented May 10, 2021

See comments on #2832 and the specific instance of this problem: https://airbytehq.slack.com/archives/C01MFR03D5W/p1620314953027000

Also, here's a concrete example of the problem: #3281

This PR switches the Java destinations to use the strict definition JSONL, which only uses \n new lines.

@jrhizor jrhizor mentioned this pull request May 10, 2021
Closed
@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/828315695
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/828315695

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-csv

🕑 connectors/destination-csv https://github.com/airbytehq/airbyte/actions/runs/828316231
✅ connectors/destination-csv https://github.com/airbytehq/airbyte/actions/runs/828316231

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-local-json

🕑 connectors/destination-local-json https://github.com/airbytehq/airbyte/actions/runs/828318421
✅ connectors/destination-local-json https://github.com/airbytehq/airbyte/actions/runs/828318421

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-meilisearch

🕑 connectors/destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/828316926
✅ connectors/destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/828316926

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-postgres

🕑 connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/828317319
✅ connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/828317319

@jrhizor jrhizor requested a review from cgardens May 10, 2021 15:46
@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-local-json

🕑 connectors/destination-local-json https://github.com/airbytehq/airbyte/actions/runs/828740736
✅ connectors/destination-local-json https://github.com/airbytehq/airbyte/actions/runs/828740736

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-csv

🕑 connectors/destination-csv https://github.com/airbytehq/airbyte/actions/runs/828741008
❌ connectors/destination-csv https://github.com/airbytehq/airbyte/actions/runs/828741008

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-postgres

🕑 connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/828741219
❌ connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/828741219

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/828741492
❌ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/828741492

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/828741939
❌ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/828741939

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/828742011
❌ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/828742011

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-meilisearch

🕑 connectors/destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/828742106
❌ connectors/destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/828742106

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-mysql

🕑 connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/828742289
❌ connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/828742289

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited
Loading

Re-bumping versions since #3289 was merged

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-local-json

🕑 connectors/destination-local-json https://github.com/airbytehq/airbyte/actions/runs/828873446
❌ connectors/destination-local-json https://github.com/airbytehq/airbyte/actions/runs/828873446

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-csv

🕑 connectors/destination-csv https://github.com/airbytehq/airbyte/actions/runs/828873522
❌ connectors/destination-csv https://github.com/airbytehq/airbyte/actions/runs/828873522

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-postgres

🕑 connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/828873930
❌ connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/828873930

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/828874166
❌ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/828874166

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/828874497
❌ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/828874497

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-meilisearch

🕑 connectors/destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/828874583
❌ connectors/destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/828874583

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-mysql

🕑 connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/828874672
❌ connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/828874672

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-local-json

🕑 connectors/destination-local-json https://github.com/airbytehq/airbyte/actions/runs/828902712
✅ connectors/destination-local-json https://github.com/airbytehq/airbyte/actions/runs/828902712

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-csv

🕑 connectors/destination-csv https://github.com/airbytehq/airbyte/actions/runs/828903115
✅ connectors/destination-csv https://github.com/airbytehq/airbyte/actions/runs/828903115

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-postgres

🕑 connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/828903600
✅ connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/828903600

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/828903849
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/828903849

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/828904067
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/828904067

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021

/publish connector=connectors/destination-redshift

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-meilisearch

🕑 connectors/destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/828905183
✅ connectors/destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/828905183

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-mysql

🕑 connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/828905368
❌ connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/828905368

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/829186862
❌ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/829186862

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-mysql

🕑 connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/829258490
✅ connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/829258490

@jrhizor
Copy link
Contributor Author

jrhizor commented May 10, 2021 •
edited
Loading

Publishing redshift locally without test.

EDIT: done

@jrhizor jrhizor merged commit 6fd8e00 into master May 10, 2021
@jrhizor jrhizor deleted the jrhizor/skip-lsep-with-scanner branch May 10, 2021 19:57
@marcosmarxm marcosmarxm mentioned this pull request May 13, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ChristopheDuong ChristopheDuong ChristopheDuong approved these changes

@cgardens cgardens cgardens approved these changes

@masonwheeler masonwheeler Awaiting requested review from masonwheeler

Assignees
No one assigned
Labels
connectors/destination/bigquery connectors/destination/local-csv connectors/destination/local-json connectors/destination/meilisearch connectors/destination/snowflake connectors/destinations-api connectors/destinations-files connectors/destinations-warehouse
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jrhizor @ChristopheDuong @cgardens @karinakuz