🎉New Source: Postmarkapp [low-code cdk] #18820

jhammarstedt · 2022-11-01T22:06:52Z

What

Added Postmarkapp Source using the low code CDK

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

jhammarstedt · 2022-11-01T22:11:05Z

I've implemented the main streams but I wanted to add a few more and had some questions:

There is a server-api key that I use now for some streams, but they also have a account-api-key that can be used for some other streams. Is this possible with the low-code cdk somehow?
Some streams, like Messages want their 'pagination' parameters count and offset, but my offset strategy returns: raise ReadException(error_message)\nairbyte_cdk.sources.declarative.exceptions.ReadException: Request <PreparedRequest [GET]> failed with response <Response [422]>\n", "failure_type": "system_error"}}}. Am I doing something wrong in the yaml configuration?

marcosmarxm · 2022-11-02T13:23:08Z

I've implemented the main streams but I wanted to add a few more and had some questions:

There is a server-api key that I use now for some streams, but they also have a account-api-key that can be used for some other streams. Is this possible with the low-code cdk somehow?

Some streams, like Messages want their 'pagination' parameters count and offset, but my offset strategy returns: raise ReadException(error_message)\nairbyte_cdk.sources.declarative.exceptions.ReadException: Request <PreparedRequest [GET]> failed with response <Response [422]>\n", "failure_type": "system_error"}}}. Am I doing something wrong in the yaml configuration?

You can create another requester with a different auth
Are you running with the --debug flag? Probably will give more info about the error.

marcosmarxm

Hello @jhammarstedt, Marcos from Airbyte here 👋 . We received more than 25 new contributions along the weekend. One is yours 🎉 thank so much for! Our team is limited and maybe the review process can take longer than expected. As described in the Airbyte's Hacktoberfest your contribution was submitted before November 2nd and it is eligible to win the prize. The review process will validate other requirements. I ask to you patience until someone from the team review it.

Because I reviewed some contributions for Hacktoberfest so far I saw some common patterns you can check in advance:

Make sure you have added connector documentation to /docs/integrations/
Remove the file catalog from /integration_tests
Edit the sample_config.json inside /integration_tests
For the configured_catalog you can use only json_schema: {}
Add title to all properties in the spec.yaml
Make sure the documentationUrl in the spec.yaml redirect to Airbyte's future connector page, eg: connector Airtable the documentationUrl: https://docs.airbyte.com/integrations/sources/airtable
Review now new line at EOF (end-of-file) for all files.

If possible send to me a DM in Slack with the tests credentials, this process will make easier to us run integration tests and publish your connector. If you only has production keys, make sure to create a bootstrap.md explaining how to get the keys.

jhammarstedt · 2022-11-02T20:07:05Z

Hi @girarda,
I just spoke with @marcosmarxm on the paginator issue (mentioned above). We found some unexpected behaviour as it was not behaving as we expected. We therefore thought that it might qualify for the "Bug / Issue surfaced on the Low-Code CDK: $100".
Running python main.py check --config secrets/config.json --debug with the following yaml :

version: "0.1.0"

definitions:
  selector:
    extractor:
      field_pointer: []
  requester:
    url_base: "https://api.postmarkapp.com/"
    http_method: "GET"
    request_options_provider:
      request_body_json:
          Accept: "application/json"
    authenticator:
      type: "ApiKeyAuthenticator"
      header: "X-Postmark-Server-Token"
      api_token: "{{ config['X-Postmark-Server-Token'] }}"
  requester_account:
    url_base: "https://api.postmarkapp.com/"
    http_method: "GET"
    request_options_provider:
      request_body_json:
          Accept: "application/json"
    authenticator:
      type: "ApiKeyAuthenticator"
      header: "X-Postmark-Account-Token"
      api_token: "{{ config['X-Postmark-Account-Token'] }}"
  retriever_account:
    record_selector:
      $ref: "*ref(definitions.selector)"
    paginator:
      type: DefaultPaginator
      $options:
        url_base: "*ref(definitions.requester.url_base)"
      pagination_strategy:
        type: "OffsetIncrement"
        page_size: 500
      page_token_option:
        inject_into: "request_parameter"
        field_name: "offset"
      page_size_option:
        inject_into: "request_parameter"
        field_name: "count"
    requester:
      $ref: "*ref(definitions.requester_account)"
  retriever:
    record_selector:
      $ref: "*ref(definitions.selector)"
    paginator:
      type: DefaultPaginator
      $options:
        url_base: "*ref(definitions.requester.url_base)"
      pagination_strategy:
        type: "OffsetIncrement"
        page_size: 500
      page_token_option:
        inject_into: "request_parameter"
        field_name: "offset"
      page_size_option:
        inject_into: "request_parameter"
        field_name: "count"
    requester:
      $ref: "*ref(definitions.requester)"
  base_stream:
    retriever:
      $ref: "*ref(definitions.retriever)"
  base_stream_account:
    retriever:
      $ref: "*ref(definitions.retriever_account)"
  deliverystats:
    $ref: "*ref(definitions.base_stream)"
    $options:
      name: "deliverystats"
      primary_key: "Type"
      path: "/deliverystats"
  message-streams:
    $ref: "*ref(definitions.base_stream)"
    $options:
      name: "message-streams"
      primary_key: "ID"
      path: "/message-streams"
  bounces:
    $ref: "*ref(definitions.base_stream)"
    $options:
      name: "bounces"
      primary_key: "ID"
      path: "/bounces"
  server:
    $ref: "*ref(definitions.base_stream_account)"
    $options:
      name: "server"
      primary_key: "ID"
      path: "/server"
  servers:
    $ref: "*ref(definitions.base_stream_account)"
    $options:
      name: "servers"
      primary_key: "ID"
      path: "/servers"
  messages:
    $ref: "*ref(definitions.base_stream)"
    $options:
      name: "messages"
      primary_key: "MessageID"
      path: "/messages/outbound"

streams:
  - "*ref(definitions.deliverystats)"
  - "*ref(definitions.message-streams)"
  - "*ref(definitions.server)"
  #- "*ref(definitions.messages)"
  #- "*ref(definitions.outbound-stats)"
  #- "*ref(definitions.bounces)"
  - "*ref(definitions.servers)"


check:
  stream_names:
    - "servers"

Gives:

{"type": "DEBUG", "message": "Receiving response", "data": {"headers": "{'cache-control': 'private', 'content-type': 'application/json; charset=utf-8', 'x-postmark-account': '252847', 'x-postmark-server': '-', 'x-pm-apierrorcode': '600', 'server': 'unicorns-double-rainbow', 'x-powered-by': 'ASP.NET', 'date': 'Wed, 02 Nov 2022 19:49:45 GMT', 'content-length': '81', 'strict-transport-security': 'max-age=16000000;'}", "status": "422", "body": "{\"ErrorCode\":600,\"Message\":\"Parameter 'count' is required but has been left out\"}"}}

girarda · 2022-11-02T20:13:23Z

@jhammarstedt can you also share the request that produced that response?

marcosmarxm · 2022-11-04T08:56:13Z

Hello! I'm going to be out of the office this Friday and won't be able to review your contribution again today, I return next Monday. So far, most contributions look solid and are almost done to be approved. As said in Chris' comment all contributions made before 2-November are eligible to receive the prize and have 2 weeks to merge the contributions. But I ensure next week we're going to have your contribution merged. If you have questions about the implementation you can send them in #hacktoberfest-2022 in Airbyte's Slack.

Sorry the inconvenience and see you again next week, thank you so much for your contribution!

marcosmarxm · 2022-11-08T00:03:02Z

@jhammarstedt can you create a Github issue related to the pagination issue?

jhammarstedt · 2022-11-09T16:43:35Z

main.py check --config secrets/config.json --debug
@marcosmarxm

Here! #19232

Although the version I have now is a workaround it still works, so it should still be ok to merge for the hackathon right?

jhammarstedt · 2022-11-09T16:56:09Z

Output from test:

jhammarstedt · 2022-11-09T16:57:15Z

@marcosmarxm tests working now

marcosmarxm · 2022-11-09T20:18:51Z

/test connector=connectors/source-postmarkapp

🕑 connectors/source-postmarkapp https://github.com/airbytehq/airbyte/actions/runs/3431539380
✅ connectors/source-postmarkapp https://github.com/airbytehq/airbyte/actions/runs/3431539380
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              345    110    68%   53, 64-72, 77-84, 88-89, 93-94, 178, 216-233, 242-250, 254-259, 265, 298-303, 341-348, 391-393, 396, 461-469, 481-484, 489, 545-546, 552, 555, 591-601, 614-639
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1508    348    77%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:65: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:243: The previous connector image could not be retrieved.
======================== 24 passed, 3 skipped in 24.82s ========================

marcosmarxm · 2022-11-09T20:32:16Z

/publish connector=connectors/source-postmarkapp

🕑 Publishing the following connectors:
connectors/source-postmarkapp
https://github.com/airbytehq/airbyte/actions/runs/3431623771

Connector	Did it publish?	Were definitions generated?
connectors/source-postmarkapp	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

marcosmarxm

Thanks @jhammarstedt

* added WIP connector Postmarkapp * pagination fix * fixing tests * fixing records * fixing tests * correct source def * correct build.md * run format * update doc * auto-bump connector version Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Nov 1, 2022

octavia-squidington-iv added community connectors/source/delighted connectors/source/postmarkapp labels Nov 1, 2022

jhammarstedt mentioned this pull request Nov 1, 2022

🎉New Source: Postmarkapp airbytehq/connector-contest#251

Closed

marcosmarxm added the hacktober label Nov 2, 2022

marcosmarxm requested changes Nov 2, 2022

View reviewed changes

jhammarstedt marked this pull request as ready for review November 3, 2022 00:00

octavia-squidington-iv added the bounty label Nov 3, 2022

jhammarstedt mentioned this pull request Nov 9, 2022

Paginator does not work as expected on low-code CDK #19232

Closed

jhammarstedt and others added 7 commits November 9, 2022 17:16

added WIP connector Postmarkapp

029916b

pagination fix

f5d1bcb

fixing tests

1ddbe19

fixing records

b006547

fixing tests

3927309

correct source def

29ef88b

correct build.md

ec13068

marcosmarxm force-pushed the source/postmarkapp branch from 3a9793b to ec13068 Compare November 9, 2022 20:17

run format

5652c0b

update doc

6888076

octavia-squidington-iii and others added 2 commits November 9, 2022 20:50

auto-bump connector version

83060a7

Merge branch 'master' into source/postmarkapp

fcdf584

marcosmarxm approved these changes Nov 9, 2022

View reviewed changes

marcosmarxm merged commit f050abc into airbytehq:master Nov 9, 2022

RealChrisSean added the low-code label Nov 15, 2022

github-actions bot mentioned this pull request Nov 15, 2022

Bump helm chart version reference to 0.40.43 #19453

Closed

octavia-squidington-iii mentioned this pull request Nov 18, 2022

Bump Airbyte version from 0.40.18 to 0.40.19 #19579

Merged

marcosmarxm changed the title ~~🎉New Source: Postmarkapp~~ 🎉New Source: Postmarkapp [low-code cdk] Nov 18, 2022

sajarin added internal and removed bounty labels Dec 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉New Source: Postmarkapp [low-code cdk] #18820

🎉New Source: Postmarkapp [low-code cdk] #18820

jhammarstedt commented Nov 1, 2022 •

edited

Loading

jhammarstedt commented Nov 1, 2022

marcosmarxm commented Nov 2, 2022

marcosmarxm left a comment

jhammarstedt commented Nov 2, 2022

girarda commented Nov 2, 2022

marcosmarxm commented Nov 4, 2022

marcosmarxm commented Nov 8, 2022

jhammarstedt commented Nov 9, 2022

jhammarstedt commented Nov 9, 2022

jhammarstedt commented Nov 9, 2022

marcosmarxm commented Nov 9, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 9, 2022 •

edited by github-actions bot

Loading

marcosmarxm left a comment

🎉New Source: Postmarkapp [low-code cdk] #18820

🎉New Source: Postmarkapp [low-code cdk] #18820

Conversation

jhammarstedt commented Nov 1, 2022 • edited Loading

What

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Tests

jhammarstedt commented Nov 1, 2022

marcosmarxm commented Nov 2, 2022

marcosmarxm left a comment

Choose a reason for hiding this comment

jhammarstedt commented Nov 2, 2022

girarda commented Nov 2, 2022

marcosmarxm commented Nov 4, 2022

marcosmarxm commented Nov 8, 2022

jhammarstedt commented Nov 9, 2022

jhammarstedt commented Nov 9, 2022

jhammarstedt commented Nov 9, 2022

marcosmarxm commented Nov 9, 2022 • edited by github-actions bot Loading

Build Passed

marcosmarxm commented Nov 9, 2022 • edited by github-actions bot Loading

marcosmarxm left a comment

Choose a reason for hiding this comment

jhammarstedt commented Nov 1, 2022 •

edited

Loading

marcosmarxm commented Nov 9, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 9, 2022 •

edited by github-actions bot

Loading