Improved schema_generator tool #13518

dougpm · 2022-06-07T00:50:04Z

What

Improves the developer experience when using the schema generator tool.

How

Created a package for the tool.
Added a command for generating a Configured Catalog.
Added documentation on how the tool works and how to use it.

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

CLAassistant · 2022-06-07T00:50:10Z

All committers have signed the CLA.

vincentkoc

Good stuff, i think you need to add unit tests and re-instate the licence. I only did a quick eye-ball on the logic

vincentkoc · 2022-06-07T11:53:03Z

tools/schema_generator/schema_generator/infer_schemas.py

@@ -1,27 +1,3 @@
-#
-# MIT License


I dont think you can drop the licence

vincentkoc · 2022-06-07T11:53:58Z

tools/schema_generator/schema_generator/main.py

+from .configure_catalog import configure_catalog
+
+
+def main():


Any unit tests for the tool?

alafanechere · 2022-06-07T17:13:09Z

Thank you @dougpm for this great contribution! Our team will review it soon.

alafanechere · 2022-06-10T09:19:40Z

Hi @dougpm let me know if I'm wrong:

Your tool is generating schemas files from running a read command
The read command requires a catalog to run, hence the underlying stream schema

So I'm under the impression that this tool can't be used to initially create schema files while starting a new connector project. Am I right?

dougpm · 2022-06-14T17:34:42Z

Hey @alafanechere

You're right, to generate the catalog we need the schema files. These files can be empty json files, they just need to be there for the discover and read commands to succeed.
I could add instructions for the creation of empty schema files for each stream before the connector container is built.
Should we go with that or are there any other options?

alafanechere · 2022-06-14T18:08:40Z

Hey @dougpm I pinged our Developer Experience team to get their opinion about this addition 😄

alafanechere · 2022-06-15T18:31:22Z

@dougpm it would help if you can add a bit more details in your documentation about the context in which this tool can be used in the connector development cycle.

dougpm · 2022-06-19T18:20:46Z

@alafanechere Added a Prerequisites section in the tool README file.

sherifnada

Generally looks good, thanks for adding docs! would also support having unit tests here but otherwise LGTM

alafanechere

@dougpm how do you feel about adding unit tests?

dougpm · 2022-06-22T11:30:12Z

@dougpm how do you feel about adding unit tests?

@alafanechere Sure I'll do that!

alafanechere · 2022-07-04T14:16:08Z

Hey @dougpm do you have the bandwidth to implement the unit tests soon?

dougpm · 2022-07-05T14:10:03Z

Hey @alafanechere yes, I'll do it this week, sorry for the delay.

dougpm · 2022-07-12T01:38:37Z

@alafanechere tests added!

alafanechere

Thank you for adding the tests. I played with the schema generator locally, I had to make minor changes to the installation process to make it work.

dougpm added 2 commits June 6, 2022 21:38

create package for schema_generator

c67acea

add docs

caa4939

github-actions bot added the area/documentation Improvements or additions to documentation label Jun 7, 2022

octavia-squidington-iii added the community label Jun 7, 2022

dougpm mentioned this pull request Jun 7, 2022

Improve schema_generator.py developer experience #11674

Closed

vincentkoc suggested changes Jun 7, 2022

View reviewed changes

marcosmarxm added the CDK Connector Development Kit label Jun 7, 2022

re-instate licence

d1954a4

github-actions bot removed the CDK Connector Development Kit label Jun 7, 2022

alafanechere added CDK Connector Development Kit team/extensibility labels Jun 7, 2022

alafanechere linked an issue Jun 7, 2022 that may be closed by this pull request

Improve schema_generator.py developer experience #11674

Closed

alafanechere self-assigned this Jun 7, 2022

fix docs

4a7e99b

github-actions bot removed the CDK Connector Development Kit label Jun 7, 2022

alafanechere requested a review from bleonard June 9, 2022 07:37

alafanechere added the CDK Connector Development Kit label Jun 14, 2022

add prerequisites

9aa2843

github-actions bot removed the CDK Connector Development Kit label Jun 19, 2022

alafanechere requested a review from sherifnada June 20, 2022 14:16

sherifnada approved these changes Jun 21, 2022

View reviewed changes

alafanechere suggested changes Jun 22, 2022

View reviewed changes

add tests

b5eec8b

fix install

df2ef0e

alafanechere approved these changes Jul 13, 2022

View reviewed changes

alafanechere mentioned this pull request Jul 13, 2022

Test #13518 #14665

Closed

alafanechere temporarily deployed to more-secrets July 13, 2022 09:45 Inactive

Merge branch 'master' into douglas/issue-11674

fc59cd5

alafanechere merged commit bfc2646 into airbytehq:master Jul 13, 2022

octavia-squidington-iii mentioned this pull request Jul 13, 2022

Bump Airbyte version from 0.39.35-alpha to 0.39.36-alpha #14690

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved schema_generator tool #13518

Improved schema_generator tool #13518

dougpm commented Jun 7, 2022

CLAassistant commented Jun 7, 2022 •

edited

Loading

vincentkoc left a comment

vincentkoc Jun 7, 2022

vincentkoc Jun 7, 2022

alafanechere commented Jun 7, 2022

alafanechere commented Jun 10, 2022

dougpm commented Jun 14, 2022

alafanechere commented Jun 14, 2022

alafanechere commented Jun 15, 2022

dougpm commented Jun 19, 2022

sherifnada left a comment

alafanechere left a comment

dougpm commented Jun 22, 2022

alafanechere commented Jul 4, 2022

dougpm commented Jul 5, 2022

dougpm commented Jul 12, 2022

alafanechere left a comment

Improved schema_generator tool #13518

Improved schema_generator tool #13518

Conversation

dougpm commented Jun 7, 2022

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Jun 7, 2022 • edited Loading

vincentkoc left a comment

Choose a reason for hiding this comment

vincentkoc Jun 7, 2022

Choose a reason for hiding this comment

vincentkoc Jun 7, 2022

Choose a reason for hiding this comment

alafanechere commented Jun 7, 2022

alafanechere commented Jun 10, 2022

dougpm commented Jun 14, 2022

alafanechere commented Jun 14, 2022

alafanechere commented Jun 15, 2022

dougpm commented Jun 19, 2022

sherifnada left a comment

Choose a reason for hiding this comment

alafanechere left a comment

Choose a reason for hiding this comment

dougpm commented Jun 22, 2022

alafanechere commented Jul 4, 2022

dougpm commented Jul 5, 2022

dougpm commented Jul 12, 2022

alafanechere left a comment

Choose a reason for hiding this comment

CLAassistant commented Jun 7, 2022 •

edited

Loading