Source Sendgrid: Update manifest for adapting changes with AsyncRetriever

[btkcodedev]

What

Closes https://github.com/airbytehq/airbyte-internal-issues/issues/11787

How

Definition of Done:

The source-sendgrid works with the updated manifest.yaml considering the changes made here
In specific:

upgrade the CDK version to at least 6.36.3

rename the urls_extractor to download_target_extractor, here

move off the stream_slice interpolation context to use the {{creation_response['id']}}, here

move off the stream_slice interpolation context to use the {{download_target}}, here

make sure the CAT passes, as before

make sure the Regression Tests pass, as before

no source behaviour changes are expected

Follow the airbytehq/airbyte-python-cdk#368 (User Impact section) for more information about the changes.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 10, 2025 6:23pm

[btkcodedev]

/bump-version type="minor" changelog="Update manifest for adapting changes with AsyncRetriever"

Bump Version job started... Check job output.

✅ Changes applied successfully. (b799c36)

[btkcodedev]

This is ready
CC @bazarnov @natikgadzhi @DanyloGL

[bazarnov]

This is ready CC @bazarnov @natikgadzhi @DanyloGL

Hey @btkcodedev we've prepared another small update for CDK > Decoders to be able to re-use them instead of the ResponseToFileExtractor, which support is terminated currently.

So, after this one is merged and released, we would need to update this PR as follows:

1. Update the version of the CDK to the latest one possible

2. Replace the:

download_extractor:
  type: ResponseToFileExtractor

With the following:

download_decoder:
  type: GzipDecoder
  decoder:
    type: CsvDecoder

Once the tests are green we should be good to merge it.

[btkcodedev]

Ok Thanks, Will do after airbytehq/airbyte-python-cdk#378 is merged!! 🙇

[bazarnov]

@btkcodedev The CDK update mentioned earlier was merged, please refer to v6.38.2 when updating the CDK version. Thanks!

[bazarnov]

Minor Change:

• Replace path with url_base in the download_requester.

After this modification, the code will be LGTM.

Thank you, @btkcodedev

[btkcodedev]

@bazarnov, the bounces stream was fine before the upgrade, its failing now, any idea why?

[btkcodedev]

the stream looks untouched for me though!

[bazarnov]

The bounces stream has no records apparently:

{"type":"LOG","log":{"level":"INFO","message":"Starting syncing"}}
{"type":"LOG","log":{"level":"INFO","message":"Marking stream bounces as STARTED"}}
{"type":"LOG","log":{"level":"INFO","message":"Syncing stream: bounces "}}
{"type":"TRACE","trace":{"type":"STREAM_STATUS","emitted_at":1741625706627.451,"stream_status":{"stream_descriptor":{"name":"bounces"},"status":"STARTED"}}}
{"type":"LOG","log":{"level":"INFO","message":"Read 0 records from bounces stream"}}
{"type":"LOG","log":{"level":"INFO","message":"Marking stream bounces as STOPPED"}}

Looks like we should update them @DanyloGL Could you please help with this?

Alternatively, we can skip checking this stream in the acceptance-test-config.yaml, here by adding the bounces stream to the list of empty_streams.

[bazarnov]

@btkcodedev I confirm that with the old CDK 5.13.0 there are no Records for the Bounces stream. Not sure why and what has happened, let's add this stream to the list of the empty_streams for now.

[btkcodedev]

Added! This should work now

[bazarnov]

Feel free to merge, once the CI is green again.