🎉 Source Orb: enrich credits ledger entries with cost basis data, description, and update expiration date fields #11528

anushree-agrawal · 2022-03-29T23:09:58Z

What

This PR enriches the Orb source connector's credits ledger entries with newly added cost basis data and description fields.
We also pull the block_expiry_date from a serialized credit_block object now, instead of pulling it off the credit_ledger_entry object itself.
We also filter to only return ledger entries that have a status of committed.

How

We've added the new cost basis and description fields to the underlying Orb API, and we now parse those fields in the connector. The Orb source API has also been updated to return expiry_date as part of a nested credit_block object, which we now unwrap and map to block_expiry_date in the connector.

The Orb source API has also been updated to filter ledger entries based on status.

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Changes are all additive, so do not expect any user impact.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Tests

Unit

Acceptance

CLAassistant · 2022-03-29T23:10:08Z

All committers have signed the CLA.

anushree-agrawal · 2022-03-29T23:19:52Z

This PR is still a work in progress, I'll comment here when it's ready for review.

marcosmarxm · 2022-03-30T00:00:20Z

@anushree-agrawal ping me when the PR is ready to review. Thanks for your contribution!

anushree-agrawal · 2022-04-21T16:54:23Z

@marcosmarxm this is ready for review now! Thanks for taking a look.

marcosmarxm

A few comments

marcosmarxm · 2022-04-25T19:29:50Z

airbyte-integrations/connectors/source-orb/source_orb/source.py

+        ledger_entry_record["block_expiry_date"] = nested_expiry_date
+        ledger_entry_record["per_unit_cost_basis"] = nested_per_unit_cost_basis


Suggested change

ledger_entry_record["block_expiry_date"] = nested_expiry_date

ledger_entry_record["per_unit_cost_basis"] = nested_per_unit_cost_basis

ledger_entry_record["credit_block_expiry_date"] = nested_expiry_date

ledger_entry_record["credit_block_per_unit_cost_basis"] = nested_per_unit_cost_basis

@marcosmarxm Thanks for the suggestion! I believe renaming block_expiry_date to credit_block_expiry_date might cause some issues since we already expose this field on the ledger_entry_record and we're just changing where in the response shape we're pulling the data from.

I can rename per_unit_cost_basis safely though, since that's a net-new field.

marcosmarxm · 2022-04-25T19:30:30Z

airbyte-integrations/connectors/source-orb/source_orb/source.py

@@ -239,7 +239,9 @@ def request_params(self, stream_state: Mapping[str, Any], stream_slice: Mapping[
        is of the same format as the stream_state of a regular incremental stream.
        """
        current_customer_state = stream_state.get(stream_slice["customer_id"], {})
-        return super().request_params(current_customer_state, **kwargs)
+        params = super().request_params(current_customer_state, **kwargs)
+        params["entry_status"] = "committed"


Can you add to docstring of the class or the function why you're getting only committed records?

marcosmarxm · 2022-04-25T20:29:39Z

@anushree-agrawal can you run ./gradlew format and see if there is any change in your code?

marcosmarxm · 2022-04-25T20:34:57Z

airbyte-integrations/connectors/source-orb/source_orb/source.py

+        del ledger_entry_record["credit_block"]
+        ledger_entry_record["block_expiry_date"] = nested_expiry_date
+        ledger_entry_record["credit_block_per_unit_cost_basis"] = nested_per_unit_cost_basis
+
        return ledger_entry_record


@anushree-agrawal would be nice to have a unit test to validate the data transformation here. wydt?

Sure, I'll add one!

anushree-agrawal · 2022-04-25T21:06:03Z

Thanks for the review!

I ran ./gradlew format and committed the changes that were for the Orb connector
I added a unit test to validate the data transformation code

marcosmarxm · 2022-04-25T22:27:01Z

/test connector=connectors/source-orb

🕑 connectors/source-orb https://github.com/airbytehq/airbyte/actions/runs/2223236332
✅ connectors/source-orb https://github.com/airbytehq/airbyte/actions/runs/2223236332
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              275    106    61%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  876    259    70%
Name                     Stmts   Miss  Cover
--------------------------------------------
source_orb/__init__.py       2      0   100%
source_orb/source.py       192     21    89%
--------------------------------------------
TOTAL                      194     21    89%

marcosmarxm

@anushree-agrawal can you bump the version of the connector inside the airbyte-config/init/.../resources/seed in source_definitions.yml and source_specs.yml files?

marcosmarxm · 2022-04-25T23:19:59Z

/publish connector=connectors/source-orb auto-bump-version=false

🕑 connectors/source-orb https://github.com/airbytehq/airbyte/actions/runs/2223431832
🚀 Successfully published connectors/source-orb
✅ connectors/source-orb https://github.com/airbytehq/airbyte/actions/runs/2223431832

anushree-agrawal · 2022-04-26T16:28:17Z

Thanks @marcosmarxm, just bumped the version

…cription, and update expiration date fields (#11528) * Add cost basis data to connector and remap block_expiry_date field * Update dockerfile and orb.md * Update orb.md and comments * Add entry_status=committed as filter for getting CreditLedgerEntries * Update version information * Move entry_status filter to the correct endpoint * PR feedback: rename field and add docstring for committed entries * Fix unit tests to include entry_status param * Add a unit test to validate transform behavior * Format using gradlew format * Bump connector veresion in spec and definitions

anushree-agrawal added 2 commits March 29, 2022 15:47

Add cost basis data to connector and remap block_expiry_date field

e31b477

Update dockerfile and orb.md

967bb72

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Mar 29, 2022

octavia-squidington-iii added the community label Mar 29, 2022

Update orb.md and comments

64213f9

anushree-agrawal marked this pull request as draft March 29, 2022 23:19

marcosmarxm added the on-hold label Apr 4, 2022

marcosmarxm self-assigned this Apr 4, 2022

anushree-agrawal added 3 commits April 20, 2022 19:52

Add entry_status=committed as filter for getting CreditLedgerEntries

5cbc2a5

Update version information

44c83f9

Move entry_status filter to the correct endpoint

558b853

anushree-agrawal marked this pull request as ready for review April 21, 2022 16:53

marcosmarxm requested changes Apr 25, 2022

View reviewed changes

marcosmarxm removed the on-hold label Apr 25, 2022

PR feedback: rename field and add docstring for committed entries

a30682f

anushree-agrawal requested a review from marcosmarxm April 25, 2022 20:28

Fix unit tests to include entry_status param

9bde1bd

marcosmarxm requested changes Apr 25, 2022

View reviewed changes

anushree-agrawal added 2 commits April 25, 2022 14:02

Add a unit test to validate transform behavior

c8887df

Format using gradlew format

1698709

anushree-agrawal requested a review from marcosmarxm April 25, 2022 21:06

marcosmarxm approved these changes Apr 25, 2022

View reviewed changes

Bump connector veresion in spec and definitions

a45226c

marcosmarxm merged commit 03a6b6b into airbytehq:master Apr 26, 2022

octavia-squidington-iii mentioned this pull request Apr 27, 2022

Bump Airbyte version from 0.36.3-alpha to 0.36.4-alpha #12415

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Orb: enrich credits ledger entries with cost basis data, description, and update expiration date fields #11528

🎉 Source Orb: enrich credits ledger entries with cost basis data, description, and update expiration date fields #11528

anushree-agrawal commented Mar 29, 2022 •

edited

Loading

CLAassistant commented Mar 29, 2022 •

edited

Loading

anushree-agrawal commented Mar 29, 2022

marcosmarxm commented Mar 30, 2022

anushree-agrawal commented Apr 21, 2022

marcosmarxm left a comment

marcosmarxm Apr 25, 2022

anushree-agrawal Apr 25, 2022

marcosmarxm Apr 25, 2022

marcosmarxm commented Apr 25, 2022

marcosmarxm Apr 25, 2022

anushree-agrawal Apr 25, 2022

anushree-agrawal commented Apr 25, 2022 •

edited

Loading

marcosmarxm commented Apr 25, 2022 •

edited by github-actions bot

Loading

marcosmarxm left a comment

marcosmarxm commented Apr 25, 2022 •

edited by github-actions bot

Loading

anushree-agrawal commented Apr 26, 2022

		ledger_entry_record["block_expiry_date"] = nested_expiry_date
		ledger_entry_record["per_unit_cost_basis"] = nested_per_unit_cost_basis

🎉 Source Orb: enrich credits ledger entries with cost basis data, description, and update expiration date fields #11528

🎉 Source Orb: enrich credits ledger entries with cost basis data, description, and update expiration date fields #11528

Conversation

anushree-agrawal commented Mar 29, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Mar 29, 2022 • edited Loading

anushree-agrawal commented Mar 29, 2022

marcosmarxm commented Mar 30, 2022

anushree-agrawal commented Apr 21, 2022

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm Apr 25, 2022

Choose a reason for hiding this comment

anushree-agrawal Apr 25, 2022

Choose a reason for hiding this comment

marcosmarxm Apr 25, 2022

Choose a reason for hiding this comment

marcosmarxm commented Apr 25, 2022

marcosmarxm Apr 25, 2022

Choose a reason for hiding this comment

anushree-agrawal Apr 25, 2022

Choose a reason for hiding this comment

anushree-agrawal commented Apr 25, 2022 • edited Loading

marcosmarxm commented Apr 25, 2022 • edited by github-actions bot Loading

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm commented Apr 25, 2022 • edited by github-actions bot Loading

anushree-agrawal commented Apr 26, 2022

anushree-agrawal commented Mar 29, 2022 •

edited

Loading

CLAassistant commented Mar 29, 2022 •

edited

Loading

anushree-agrawal commented Apr 25, 2022 •

edited

Loading

marcosmarxm commented Apr 25, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Apr 25, 2022 •

edited by github-actions bot

Loading