Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Source S3: work-around for format.delimiter change '\\t' -> '\t' #9163

Merged
merged 19 commits into from
Jan 6, 2022
Merged

🐛 Source S3: work-around for format.delimiter change '\\t' -> '\t' #9163

Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
ca2755e
work-around for format.delimiter '\\t' -> '\t'
grubberr Dec 28, 2021
96283fd
Merge branch 'master' into grubberr/8789-source-s3
grubberr Jan 4, 2022
8177d41
check disk space
grubberr Jan 4, 2022
6769362
Revert "check disk space"
grubberr Jan 4, 2022
c49a5e2
add some debug info
grubberr Jan 4, 2022
8de2b19
build airbyteDocker
grubberr Jan 4, 2022
67cb8c5
Merge branch 'master' into grubberr/8789-source-s3
grubberr Jan 5, 2022
03d0b82
remove debug
grubberr Jan 5, 2022
9b84368
Merge branch 'master' into grubberr/8789-source-s3
grubberr Jan 6, 2022
b53f7a5
test_transform_backslash_t_to_tab added
grubberr Jan 6, 2022
af80ed6
bump version 0.1.8 -> 0.1.9
grubberr Jan 6, 2022
30a0924
s3.md updated
grubberr Jan 6, 2022
145ccf3
fix 2021 -> 2022
grubberr Jan 6, 2022
3fe044e
s3.md fixed
grubberr Jan 6, 2022
d76302d
s3.md updated 2-nd time
grubberr Jan 6, 2022
fa3bfd6
fix SAT tests
grubberr Jan 6, 2022
48397eb
Merge branch 'master' into grubberr/8789-source-s3
grubberr Jan 6, 2022
897a0b1
Merge branch 'master' into grubberr/8789-source-s3
grubberr Jan 6, 2022
a1b546b
source_specs.yaml updated
grubberr Jan 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion airbyte-integrations/connectors/source-s3/source_s3/source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
#


from typing import Optional
from typing import Any, Mapping, Optional

from pydantic import BaseModel, Field

Expand Down Expand Up @@ -47,3 +47,9 @@ class SourceS3(SourceFilesAbstract):
stream_class = IncrementalFileStreamS3
spec_class = SourceS3Spec
documentation_url = "https://docs.airbyte.io/integrations/sources/s3"

def read_config(self, config_path: str) -> Mapping[str, Any]:
config = super().read_config(config_path)
if config.get("format", {}).get("delimiter") == r"\t":
config["format"]["delimiter"] = "\t"
return config