Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(source-google-sheets): add sheet id encoding #52671

Merged
merged 79 commits into from
Jan 31, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
c9faf8b
source-google-sheets: adding dymamic streams and dynamic schemas for …
aldogonzalez8 Dec 12, 2024
b4f8deb
source-google-sheets: add custom retriever with custom slicer for ranges
aldogonzalez8 Dec 14, 2024
86f4cc9
source-google-sheets: fix nested records in value field with custom e…
aldogonzalez8 Dec 16, 2024
777daa5
source-google-sheets:" -m " - Add transformations to record selector…
aldogonzalez8 Dec 18, 2024
9435c86
source-google-sheets: remove teste file
aldogonzalez8 Dec 18, 2024
624d671
source-google-sheets:
aldogonzalez8 Dec 18, 2024
4d87723
source-google-sheets: initial changes for integrations tests passing.
aldogonzalez8 Dec 19, 2024
5b61aed
source-google-sheets: - all check operations (excelpt service auth fl…
aldogonzalez8 Dec 21, 2024
e1f6a75
source-google-sheets: all discovery operations passing
aldogonzalez8 Dec 23, 2024
30b6673
source-google-sheets: all original read integration tests passing
aldogonzalez8 Dec 24, 2024
510e074
source-google-sheets: test for read with conversion ready and also ch…
aldogonzalez8 Dec 24, 2024
76def28
source-google-sheets: add support for empty cells
aldogonzalez8 Dec 24, 2024
f9f658b
source-google-sheets: add tests for empty columns
aldogonzalez8 Dec 24, 2024
4b6636b
source-google-sheets: complete all read protocol tests
aldogonzalez8 Dec 31, 2024
6f24fac
source-google-sheets: merge from master
aldogonzalez8 Jan 1, 2025
6c53e9e
source-google-sheets: reduce unnecesary response filters
aldogonzalez8 Jan 1, 2025
041280a
source-google-sheets: fix styles
aldogonzalez8 Jan 1, 2025
00edfb4
source-google-sheets: split tests in different files per type
aldogonzalez8 Jan 1, 2025
5dd67d0
source-google-sheets: ruff-format
aldogonzalez8 Jan 1, 2025
6021253
source-google-sheets: update release information
aldogonzalez8 Jan 2, 2025
5876dc9
chore: auto-fix lint and format issues
octavia-squidington-iii Jan 2, 2025
874c191
source-google-sheets: add tests for components and urls
aldogonzalez8 Jan 3, 2025
c0e9e6f
source-google-sheets: ruff fix format
aldogonzalez8 Jan 3, 2025
ba3620b
source-google-sheets: add tests for parsing spreadsheet_id in config …
aldogonzalez8 Jan 3, 2025
a35f07f
source-google-sheets: ruff-format
aldogonzalez8 Jan 3, 2025
7ed534a
source-google-sheets: add manager for to increase batch size when 429…
aldogonzalez8 Jan 6, 2025
6ff3244
source-google-sheets: ruff-format
aldogonzalez8 Jan 6, 2025
e625be6
source-google-sheets: minor changes to batch size manager to handle t…
aldogonzalez8 Jan 6, 2025
6b487ea
source-google-sheets: ruff-format
aldogonzalez8 Jan 6, 2025
fc59b0d
source-google-sheets: add sheet id to range logging and remove old files
aldogonzalez8 Jan 6, 2025
1803ba8
source-google-sheets: ruff-format
aldogonzalez8 Jan 6, 2025
1dc50a5
source-google-sheets: Update lock file with latest CDK
aldogonzalez8 Jan 9, 2025
2b2f970
source-google-sheets: initial changes for jwt_requester
aldogonzalez8 Jan 10, 2025
0eef551
source-google-sheets: ruff format
aldogonzalez8 Jan 10, 2025
75ec92b
Merge branch 'master' into aldogonzalez8/source-google-sheets/migrate…
lazebnyi Jan 21, 2025
9dace30
Update authentication, add config migration, and fix backoff
lazebnyi Jan 23, 2025
91f4ff6
Merge branch 'aldogonzalez8/source-google-sheets/migrate-low-code' of…
lazebnyi Jan 23, 2025
7f052d9
chore: auto-fix lint and format issues
octavia-squidington-iii Jan 23, 2025
65b09ce
Fix some checks
lazebnyi Jan 23, 2025
02a140a
Merge branch 'aldogonzalez8/source-google-sheets/migrate-low-code' of…
lazebnyi Jan 23, 2025
1a19868
chore: auto-fix lint and format issues
octavia-squidington-iii Jan 23, 2025
8838242
Bump CDK version
lazebnyi Jan 23, 2025
18be3a4
Merge branch 'aldogonzalez8/source-google-sheets/migrate-low-code' of…
lazebnyi Jan 23, 2025
fee240e
Merge branch 'master' into aldogonzalez8/source-google-sheets/migrate…
lazebnyi Jan 24, 2025
4e2c3fd
Removed google libs
lazebnyi Jan 24, 2025
d5fad2d
Merge branch 'master' into aldogonzalez8/source-google-sheets/migrate…
lazebnyi Jan 29, 2025
78daf19
Merge branch 'aldogonzalez8/source-google-sheets/migrate-low-code' of…
lazebnyi Jan 29, 2025
fe14f92
Fix unit tests
lazebnyi Jan 29, 2025
8014fca
chore: auto-fix lint and format issues
octavia-squidington-iii Jan 29, 2025
3331a9b
Update configs name
lazebnyi Jan 29, 2025
9419696
Merge branch 'aldogonzalez8/source-google-sheets/migrate-low-code' of…
lazebnyi Jan 29, 2025
bf396e0
Merge branch 'master' into aldogonzalez8/source-google-sheets/migrate…
lazebnyi Jan 29, 2025
90eb46d
Update test secrets names
lazebnyi Jan 29, 2025
d3cb431
Fix coverage
lazebnyi Jan 29, 2025
3d724ba
chore: auto-fix lint and format issues
octavia-squidington-iii Jan 29, 2025
3f2b754
Fix unit tests
lazebnyi Jan 29, 2025
58d2686
Fix unit tests
lazebnyi Jan 29, 2025
be9b645
chore: auto-fix lint and format issues
octavia-squidington-iii Jan 29, 2025
2d76e00
Fix backoff policy
lazebnyi Jan 29, 2025
d9ef054
Merge branch 'aldogonzalez8/source-google-sheets/migrate-low-code' of…
lazebnyi Jan 29, 2025
17d7781
Speedup conenctor
lazebnyi Jan 29, 2025
23c494d
chore: auto-fix lint and format issues
octavia-squidington-iii Jan 29, 2025
37724b8
Remove unnecessary imports
lazebnyi Jan 29, 2025
46981a1
Add coverage ignore file
lazebnyi Jan 29, 2025
caafa61
disable http proxy
lazebnyi Jan 30, 2025
8996ce3
rollback http proxy
lazebnyi Jan 30, 2025
0b684e3
Update docs and add rc version
lazebnyi Jan 30, 2025
de3c0aa
Add enableProgressiveRollout to metadata
lazebnyi Jan 30, 2025
5e2ecd2
Update version in pyproject
lazebnyi Jan 30, 2025
a009326
Bump CDK version to latest
lazebnyi Jan 30, 2025
dfa5dae
Merge branch 'master' into aldogonzalez8/source-google-sheets/migrate…
lazebnyi Jan 30, 2025
bcec7e9
Remove config migration
lazebnyi Jan 30, 2025
48ea89d
Merge branch 'aldogonzalez8/source-google-sheets/migrate-low-code' of…
lazebnyi Jan 30, 2025
bf1ecac
Move to CDK last relese version
lazebnyi Jan 30, 2025
b656c05
Fix jinja macros
lazebnyi Jan 30, 2025
a65f723
Fix encoding sheet id
lazebnyi Jan 30, 2025
8675595
Merge branch 'master' into lazebnyi/fix-sheed-id-encode-for-google-sh…
lazebnyi Jan 31, 2025
1b9c98d
Update expected records
lazebnyi Jan 31, 2025
efc835c
Merge branch 'lazebnyi/fix-sheed-id-encode-for-google-sheets' of gith…
lazebnyi Jan 31, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"streams": [
{
"stream": {
"name": "Sheet1-one-col",
"name": "Sheet1 one col&special name%?",
"json_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,26 +1,26 @@
{"stream":"Sheet1-one-col","data":{"ID":"aa"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"bb"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"1"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"2"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"3"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"4"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"5"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"6"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"7"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"8"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"9"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"10"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"11"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"12"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"13"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"14"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"15"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"16"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"17"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"18"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"19"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"20"},"emitted_at":1673989565000}
{"stream":"Sheet1-one-col","data":{"ID":"21"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"aa"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"bb"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"1"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"2"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"3"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"4"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"5"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"6"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"7"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"8"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"9"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"10"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"11"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"12"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"13"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"14"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"15"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"16"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"17"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"18"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"19"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"20"},"emitted_at":1673989565000}
{"stream":"Sheet1 one col&special name%?","data":{"ID":"21"},"emitted_at":1673989565000}
{"stream":"Sheet2-two-cols","data":{"ID":"a","Name":"a"},"emitted_at":1673989565000}
{"stream":"Sheet2-two-cols","data":{"ID":"b","Name":"b"},"emitted_at":1673989565000}
{"stream":"Sheet2-two-cols","data":{"ID":"c","Name":"c"},"emitted_at":1673989565000}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ data:
connectorSubtype: file
connectorType: source
definitionId: 71607ba1-c0ac-4799-8049-7f4b90dd50f7
dockerImageTag: 0.9.0-rc.1
dockerImageTag: 0.9.0-rc.2
dockerRepository: airbyte/source-google-sheets
documentationUrl: https://docs.airbyte.com/integrations/sources/google-sheets
githubIssueLabel: source-google-sheets
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ requires = [ "poetry-core>=1.0.0",]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
version = "0.9.0-rc.1"
version = "0.9.0-rc.2"
name = "source-google-sheets"
description = "Source implementation for Google Sheets."
authors = [ "Airbyte <contact@airbyte.io>",]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ dynamic_streams:
name: ""
http_method: GET
path: >-
{% if config["spreadsheet_id"] | regex_search("^(https://.*)") %}{{ config["spreadsheet_id"] | regex_search("/([-\\w]{20,})([/]?)") }}{% else %}{{ config["spreadsheet_id"] }}{% endif %}/values:batchGet?ranges={{parameters["sheet_id"]}}!{{stream_partition.start_range}}:{{stream_partition.end_range}}&majorDimension=ROWS&alt=json
{% if config["spreadsheet_id"] | regex_search("^(https://.*)") %}{{ config["spreadsheet_id"] | regex_search("/([-\\w]{20,})([/]?)") }}{% else %}{{ config["spreadsheet_id"] }}{% endif %}/values:batchGet?ranges={{parameters["sheet_id"] | urlencode}}!{{stream_partition.start_range}}:{{stream_partition.end_range}}&majorDimension=ROWS&alt=json
error_handler:
type: DefaultErrorHandler
backoff_strategies:
Expand Down Expand Up @@ -85,7 +85,7 @@ dynamic_streams:
sheet_id: ""
http_method: GET
path: >-
{% if config["spreadsheet_id"] | regex_search("^(https://.*)") %}{{ config["spreadsheet_id"] | regex_search("/([-\\w]{20,})([/]?)") }}{% else %}{{ config["spreadsheet_id"] }}{% endif %}?includeGridData=true&ranges={{parameters["sheet_id"]}}!1:1&alt=json
{% if config["spreadsheet_id"] | regex_search("^(https://.*)") %}{{ config["spreadsheet_id"] | regex_search("/([-\\w]{20,})([/]?)") }}{% else %}{{ config["spreadsheet_id"] }}{% endif %}?includeGridData=true&ranges={{parameters["sheet_id"] | urlencode}}!1:1&alt=json
error_handler:
type: CompositeErrorHandler
error_handlers:
Expand Down Expand Up @@ -221,7 +221,7 @@ definitions:
description: spreadsheet_id can be either the full url to spreadsheet or the spreadsheet id.
http_method: GET
path: >-
{% if config["spreadsheet_id"] | regex_search("^(https://.*)") %}{{ config["spreadsheet_id"] | regex_search("/([-\\w]{20,})([/]?)") }}{% else %}{{ config["spreadsheet_id"] }}{% endif %}?includeGridData=true&ranges={{stream_partition.sheet_id}}!1:1&alt=json
{% if config["spreadsheet_id"] | regex_search("^(https://.*)") %}{{ config["spreadsheet_id"] | regex_search("/([-\\w]{20,})([/]?)") }}{% else %}{{ config["spreadsheet_id"] }}{% endif %}?includeGridData=true&ranges={{stream_partition.sheet_id | urlencode}}!1:1&alt=json
error_handler:
type: CompositeErrorHandler
error_handlers:
Expand Down
3 changes: 2 additions & 1 deletion docs/integrations/sources/google-sheets.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,8 @@ Airbyte batches requests to the API in order to efficiently pull data and respec

| Version | Date | Pull Request | Subject |
|------------|------------|----------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0.9.0-rc.1 | 2025-02-03 | [50843](https://github.com/airbytehq/airbyte/pull/50843) | Migrate to low-code |
| 0.9.0-rc.2 | 2025-01-31 | [52671](https://github.com/airbytehq/airbyte/pull/52671) | Fix sheet id encoding |
| 0.9.0-rc.1 | 2025-01-30 | [50843](https://github.com/airbytehq/airbyte/pull/50843) | Migrate to low-code |
| 0.8.5 | 2025-01-11 | [44270](https://github.com/airbytehq/airbyte/pull/44270) | Starting with this version, the Docker image is now rootless. Please note that this and future versions will not be compatible with Airbyte versions earlier than 0.64 |
| 0.8.4 | 2024-12-09 | [48835](https://github.com/airbytehq/airbyte/pull/48835) | Implementing integration tests |
| 0.7.4 | 2024-09-09 | [45108](https://github.com/airbytehq/airbyte/pull/45108) | Google Sheets API errors now cause syncs to fail |
Expand Down
Loading