Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make config-based docs accessible through direct link #15662

Merged
merged 5 commits into from
Aug 15, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/connector-development/config-based/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
:warning: This framework is in alpha stage. Support is not in production and is available only to select users. :warning:

The goal of this document is to give enough technical specifics to understand how config-based connectors work.
When you're ready to start building a connector, you can start with [the tutorial](../../../config-based/tutorial/0-getting-started.md) or dive into the [reference documentation](https://airbyte-cdk.readthedocs.io/en/latest/api/airbyte_cdk.sources.declarative.html)
When you're ready to start building a connector, you can start with [the tutorial](./tutorial/0-getting-started.md) or dive into the [reference documentation](https://airbyte-cdk.readthedocs.io/en/latest/api/airbyte_cdk.sources.declarative.html)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix link


## Overview

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ The connector will now always read data for the start date, which is not exactly
Instead, we would like to iterate over all the dates between the start_date and today and read data for each day.

We can do this by adding a `DatetimeStreamSlicer` to the connector definition, and update the `path` to point to the stream_slice's `start_date`:
More details on the stream slicers can be found [here](./link-to-stream-slicers.md) <FIXME: need to fix links>
More details on the stream slicers can be found [here](../stream-slicers.md) <FIXME: need to fix links>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix link


Let's first define a stream slicer at the top level of the connector definition:

Expand Down Expand Up @@ -298,4 +298,4 @@ Next, we'll run the [Source Acceptance Tests suite to ensure the connector invar

- [Incremental reads](../../cdk-python/incremental-stream.md)
- [Stream slicers](../stream-slicers.md)
- [Stream slices](../cdk-python/stream-slices.md)
- [Stream slices](../../cdk-python/stream-slices.md)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix link

Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ $ python -m pytest integration_tests -p integration_tests.acceptance

Next, we'll add the connector to the [Airbyte platform](https://docs.airbyte.com/connector-development/tutorials/cdk-tutorial-python-http/use-connector-in-airbyte).

See your [Contributiong guide]() on how to get started releasing your connector.
See your [Contributiong guide](../../../contributing-to-airbyte/README.md) on how to get started releasing your connector.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix link


## Read more:

Expand Down
2 changes: 1 addition & 1 deletion docs/connector-development/config-based/yaml-structure.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ In this example, outer.inner.k2 will evaluate to "MyKey is MyValue"
Strings can contain references to previously defined values.
The parser will dereference these values to produce a complete ConnectionDefinition

References can be defined using a *ref(<arg>) string.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think docusaurus was interpreting the <> as an html directive. the build step stopped complaining after changing to {arg}

References can be defined using a "*ref({arg})" string.

```yaml
key: 1234
Expand Down
16 changes: 9 additions & 7 deletions docs/contributing-to-airbyte/gradle-cheatsheet.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ We have 3 ways of slicing our builds:
In our CI we run **Build Platform** and **Build Connectors Base**. Then separately, on a regular cadence, we build each connector and run its integration tests.

We split Build Platform and Build Connectors Base from each other for a few reasons:

1. The tech stacks are very different. The Platform is almost entirely Java. Because of differing needs around separating environments, the Platform build can be optimized separately from the Connectors one.
2. We want to the iteration cycles of people working on connectors or the platform faster _and_ independent. e.g. Before this change someone working on a Platform feature needs to run formatting on the entire codebase \(including connectors\). This led to a lot of cosmetic build failures that obfuscated actually problems. Ideally a failure on the connectors side should not block progress on the platform side.
3. The lifecycles are different. One can safely release the Platform even if parts of Connectors Base is failing \(and vice versa\).
Expand Down Expand Up @@ -101,18 +102,18 @@ We split Acceptance Tests into 2 different test suites:
* Platform Acceptance Tests: These tests are a coarse test to sanity check that each major feature in the platform. They are run with the following command: `SUB_BUILD=PLATFORM ./gradlew :airbyte-tests:acceptanceTests`. These tests expect to find a local version of Airbyte running. For testing the docker version start Airbyte locally. For an example, see the [acceptance_test script](../../tools/bin/acceptance_test.sh) that is used by the CI. For Kubernetes, see the [accetance_test_kube script](../../tools/bin/acceptance_test_kube.sh) that is used by the CI.
* Migration Acceptance Tests: These tests make sure the end-to-end process of migrating from one version of Airbyte to the next works. These tests are run with the following command: `SUB_BUILD=PLATFORM ./gradlew :airbyte-tests:automaticMigrationAcceptanceTest --scan`. These tests do not expect there to be a separate deployment of Airbyte running.

These tests currently all live in [airbyte-tests](../.././airbyte-tests)
These tests currently all live in [airbyte-tests](https://github.com/airbytehq/airbyte/airbyte-tests)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

link to github because airbyte-tests isn't included in the docs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ryankfu FYI


**Frontend Acceptance Tests**

These are acceptance tests for the frontend. They are run with
These are acceptance tests for the frontend. They are run with
```shell
SUB_BUILD=PLATFORM ./gradlew --no-daemon :airbyte-webapp-e2e-tests:e2etest
```
Like the Platform Acceptance Tests, they expect Airbyte to be running locally. See the [script](https://github.com/airbytehq/airbyte/blob/master/tools/bin/e2e_test.sh) that is used by the CI.

Like the Platform Acceptance Tests, they expect Airbyte to be running locally. See the [script](https://github.com/airbytehq/airbyte/blob/master/tools/bin/e2e_test.sh) that is used by the CI.

These tests currently all live in [airbyte-webapp-e2e-tests](../.././airbyte-webapp-e2e-tests)
These tests currently all live in [airbyte-webapp-e2e-tests](https://github.com/airbytehq/airbyte/airbyte-webapp-e2e-tests)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

link to github because airbyte-webapp-e2e-tests isn't part of the docs


**Future Work**

Expand All @@ -124,11 +125,12 @@ Our story around "integration testing" or "E2E testing" is a little ambiguous. O

All connectors, regardless of implementation language, implement the following interface to allow uniformity in the build system when run from CI:

**Build connector, run unit tests, and build Docker image**:
**Build connector, run unit tests, and build Docker image**:
```shell
./gradlew :airbyte-integrations:connectors:<connector_name>:build
```
**Run integration tests**:

**Run integration tests**:
```shell
./gradlew :airbyte-integrations:connectors:<connector_name>:integrationTest
```
Expand All @@ -139,7 +141,7 @@ The ideal end state for a Python connector developer is that they shouldn't have

We're almost there, but today there is only one Gradle command that's needed when developing in Python, used for formatting code.

**Formatting python module**:
**Formatting python module**:
```shell
./gradlew :airbyte-integrations:connectors:<connector_name>:airbytePythonFormat
```
Expand Down
1 change: 0 additions & 1 deletion docusaurus/docusaurus.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,6 @@ const config = {
sidebarPath: require.resolve('./sidebars.js'),
editUrl: 'https://github.com/airbytehq/airbyte/blob/master/docs',
path: '../docs',
exclude: ['**/connector-development/config-based/**']
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

include /connector-development/config-based/ to the build, but not to the sidebar

},
blog: false,
theme: {
Expand Down