Skip to content

Commit

Permalink
Merge branch 'master' of https://github.com/airbytehq/airbyte into ai…
Browse files Browse the repository at this point in the history
…rbytehq-master

* 'master' of https://github.com/airbytehq/airbyte: (1141 commits)
  pass USE_STREAM_CAPABLE_STATE env var to containers/deployments (airbytehq#14737)
  Bump mqtt connector (airbytehq#14648)
  Add error code to ManualOperationResult (airbytehq#14657)
  Bump elasticsearch version (airbytehq#14640)
  Ryan/sync oracle version number (airbytehq#14736)
  Fixed linter issue with add_fields.py comments (airbytehq#14742)
  🎉Redshift, Databricks, Snowflake, S3 Destinations: Make S3 output filename configurable (airbytehq#14494)
  🐛Source-mssql: aligned regular and cdc syncs and its datatype tests (airbytehq#14379)
  🎉 Source Amazon Seller Partner: Add new streams (airbytehq#13604)
  bump source-file-secure (airbytehq#14704)
  🎉 New source: Timely airbytehq#13292 (airbytehq#14335)
  🪟🔧 Refactor feature service (airbytehq#14559)
  [low code cdk] add a transformation for adding fields into an outgoing record (airbytehq#14638)
  Bump destination-postgres to 0.3.21 (airbytehq#14479)
  Remove `additionalProperties: false` from JDBC destination connectors (airbytehq#14618)
  🎉 Source Notion: add OAuth authorization for source-notion connector (airbytehq#14706)
  Use the configuration diff calculation in the update endpoint (airbytehq#14626)
  🪟 🐛 Fix input validation on blur and cleanup signup error handling (airbytehq#14724)
  lower sleep after wait for successful job (airbytehq#14725)
  Add configuration diff (airbytehq#14603)
  ...
  • Loading branch information
jsrcodes committed Jul 15, 2022
2 parents 7b9ea2b + 805a155 commit 755e0d7
Show file tree
Hide file tree
Showing 6,097 changed files with 228,925 additions and 97,855 deletions.
The diff you're trying to view is too large. We only load the first 3000 changed files.
8 changes: 5 additions & 3 deletions .bumpversion.cfg
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 0.35.67-alpha
current_version = 0.39.37-alpha
commit = False
tag = False
parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\-[a-z]+)?
Expand All @@ -16,8 +16,6 @@ serialize =

[bumpversion:file:airbyte-metrics/reporter/Dockerfile]

[bumpversion:file:airbyte-scheduler/app/Dockerfile]

[bumpversion:file:airbyte-server/Dockerfile]

[bumpversion:file:airbyte-webapp/package.json]
Expand Down Expand Up @@ -47,3 +45,7 @@ serialize =
[bumpversion:file:octavia-cli/README.md]

[bumpversion:file:octavia-cli/Dockerfile]

[bumpversion:file:octavia-cli/setup.py]
serialize =
{major}.{minor}.{patch}
26 changes: 17 additions & 9 deletions .env
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@


### SHARED ###
VERSION=0.35.67-alpha
VERSION=0.39.37-alpha

# When using the airbyte-db via default docker image
CONFIG_ROOT=/data
Expand Down Expand Up @@ -40,7 +40,7 @@ DATABASE_PASSWORD=docker
DATABASE_HOST=db
DATABASE_PORT=5432
DATABASE_DB=airbyte
# translate manually DATABASE_URL=jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT/${DATABASE_DB} (do not include the username or password here)
# translate manually DATABASE_URL=jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_DB} (do not include the username or password here)
DATABASE_URL=jdbc:postgresql://db:5432/airbyte
JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.29.15.001

Expand Down Expand Up @@ -70,26 +70,34 @@ JOB_MAIN_CONTAINER_MEMORY_LIMIT=

### LOGGING/MONITORING/TRACKING ###
TRACKING_STRATEGY=segment
JOB_ERROR_REPORTING_STRATEGY=logging
# Although not present as an env var, expected by Log4J configuration.
LOG_LEVEL=INFO
# Although not present as an env var, helps Airbyte track job healthiness.
SENTRY_DSN="https://d4b03de0c4574c78999b8d58e55243dc@o1009025.ingest.sentry.io/6102835"


### APPLICATIONS ###
# Scheduler #
# Relevant to scaling.
SUBMITTER_NUM_THREADS=10

# Worker #
# Relevant to scaling.
MAX_SYNC_WORKERS=5
MAX_SPEC_WORKERS=5
MAX_CHECK_WORKERS=5
MAX_DISCOVER_WORKERS=5
# Temporal Activity configuration
ACTIVITY_MAX_ATTEMPT=
ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS=
ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS=
WORKFLOW_FAILURE_RESTART_DELAY_SECONDS=


### FEATURE FLAGS ###
NEW_SCHEDULER=false
AUTO_DISABLE_FAILING_CONNECTIONS=false
EXPOSE_SECRETS_IN_EXPORT=false
FORCE_MIGRATE_SECRET_STORE=false

### MONITORING FLAGS ###
# Accepted values are datadog and otel (open telemetry)
METRIC_CLIENT=
# Useful only when metric client is set to be otel. Must start with http:// or https://.
OTEL_COLLECTOR_ENDPOINT="http://host.docker.internal:4317"

USE_STREAM_CAPABLE_STATE=false
75 changes: 0 additions & 75 deletions .gitbook.yaml

This file was deleted.

40 changes: 40 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,43 @@
## Exclude the package(-lock).json from code ownership to prevent version bump PRs from triggering codeowners review
/airbyte-webapp/package.json
/airbyte-webapp/package-lock.json

# CDK and SAT
/airbyte-cdk/ @airbytehq/api-connectors-dx
/airbyte-integrations/bases/source-acceptance-tests/ @airbytehq/api-connectors-dx
/airbyte-integrations/connector-templates/ @airbytehq/api-connectors-dx

# Protocol related items
/airbyte-protocol/ @airbytehq/protocol-reviewers
/docs/understanding-airbyte/airbyte-protocol.md @airbytehq/protocol-reviewers

# Normalization
/airbyte-integrations/bases/base-normalization/ @airbytehq/normalization

# JDBC-based connectors
/airbyte-integrations/bases/base-java/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/source-jdbc/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/source-bigquery/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/source-clickhouse/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/source-cockroachdb/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/source-db2/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/source-mssql/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/source-mysql/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/source-oracle/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/source-postgres/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/source-redshift/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/source-snowflake/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/source-tidb/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/destination-jdbc/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/destination-azure-blob-storage/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/destination-clickhouse/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/destination-databricks/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/destination-gcs/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/destination-mariadb-columnstore/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/destination-mysql/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/destination-mssql/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/destination-oracle/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/destination-postgres/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/destination-redshift/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/destination-rockset/ @airbytehq/jdbc-connectors
/airbyte-integrations/connectors/source-snowflake/ @airbytehq/jdbc-connectors
24 changes: 8 additions & 16 deletions .github/ISSUE_TEMPLATE/bug-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ assignees: ''
---
<!--
Welcome to Airbyte!
We're really appreciate your report and know that this will help us build an
amazing tool. If you want to contribute yourself, you can find a good place
to start by searching for the good-first-issues label or maybe... by trying
Expand All @@ -28,7 +29,6 @@ You can remove the examples bellow and fill out with your information.
- **Deployment**: example are Docker or Kubernetes deploy env
- **Source Connector and version**: (if applicable example Salesforce 0.2.3) <!-- Found in the admin page in the UI in the Source tab. -->
- **Destination Connector and version**: (if applicable example Postgres 0.3.3) <!-- Found in the admin page in the UI in the Destination tab. -->
- **Severity**: Very Low / Low / Medium / High / Critical
- **Step where error happened**: Deploy / Sync job / Setup new connection / Update connector / Upgrade Airbyte

## Current Behavior
Expand All @@ -38,22 +38,14 @@ You can remove the examples bellow and fill out with your information.
*Tell us what should happen.*

## Logs
*If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.*

<details>
<summary>LOG</summary>

```
<!--
Please upload the complete log files from the failing operation.
replace this with
your long log
output here
```

</details>
We ask for complete log files because these files capture essential metadata about your environment.
The team will read the full stack trace of errors and try to find the root cause of the error.
For a sync failure, you can download the log in the sync page, and for other errors please download the server and scheduler logs in the Settings Page -> Configuration.
You can remove this after uploading the file.
-->

## Steps to Reproduce
1.
Expand Down
48 changes: 9 additions & 39 deletions .github/actions/build-and-push-branch/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,60 +4,30 @@ inputs:
branch_version_tag:
description: 'Used to tag jars and docker images with a branch-specific version (should use the form "dev-<commit_hash>" to pass AirbyteVersion validation)'
required: false
dockerhub_username:
description: "Used to log in to dockerhub for pushing images"
required: true
dockerhub_token:
description: "Used to log in to dockerhub for pushing images"
required: true
runs:
using: "composite"
steps:
- name: "Parse Input"
id: parse-input
shell: bash
run: |-
# if the *branch_version_tag* input param is not specified, then generate it as 'dev-<commit_hash>`
#
[[ "${{ inputs.branch_version_tag }}" != '' ]] && echo "::set-output name=branch_version_tag::${{ inputs.branch_version_tag }}" \
|| { short_hash=$(git rev-parse --short HEAD); echo "::set-output name=branch_version_tag::dev-$short_hash"; }
- uses: actions/setup-java@v1
with:
java-version: "17"

- uses: actions/setup-node@v1
with:
node-version: "16.13.0"

- name: Set up CI Gradle Properties
run: |
mkdir -p ~/.gradle/
cat > ~/.gradle/gradle.properties <<EOF
org.gradle.jvmargs=-Xmx8g -Xss4m --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
org.gradle.workers.max=8
org.gradle.vfs.watch=false
EOF
shell: bash

- name: Build
run: VERSION=${{ steps.parse-input.outputs.branch_version_tag }} SUB_BUILD=PLATFORM ./gradlew build --scan
shell: bash

- name: Publish to Maven Local
run: VERSION=${{ steps.parse-input.outputs.branch_version_tag }} SUB_BUILD=PLATFORM ./gradlew publishToMavenLocal
shell: bash
id: build
uses: ./.github/actions/build-branch
with:
branch_version_tag: ${{ inputs.branch_version_tag }}

- name: Login to Docker (on Master)
uses: docker/login-action@v1
with:
username: airbytebot
username: ${{ inputs.dockerhub_username }}
password: ${{ inputs.dockerhub_token }}

- name: Push Docker Images
run: |
GIT_REVISION=$(git rev-parse HEAD)
[ [ -z "$GIT_REVISION" ] ] && echo "Couldn't get the git revision..." && exit 1
VERSION=${{ steps.parse-input.outputs.branch_version_tag }} GIT_REVISION=$GIT_REVISION docker-compose -f docker-compose-cloud.build.yaml push
VERSION=${{ steps.build.outputs.branch_version_tag }} GIT_REVISION=$GIT_REVISION docker-compose -f docker-compose-cloud.build.yaml push
shell: bash
52 changes: 52 additions & 0 deletions .github/actions/build-branch/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: "Build OSS Branch"
description: "Build jars and docker images tagged for a particular branch. Primarily used for running OSS branch code in Cloud."
inputs:
branch_version_tag:
description: 'Used to tag jars and docker images with a branch-specific version (should use the form "dev-<commit_hash>" to pass AirbyteVersion validation)'
required: false
outputs:
branch_version_tag:
description: 'Tag used for jars and docker images. Either user specified or auto generated as `dev-<commit_hash>`'
value: ${{ steps.parse-input.outputs.branch_version_tag }}
runs:
using: "composite"
steps:
- name: "Parse Input"
id: parse-input
shell: bash
run: |-
# if the *branch_version_tag* input param is not specified, then generate it as 'dev-<commit_hash>`
#
[[ "${{ inputs.branch_version_tag }}" != '' ]] && echo "::set-output name=branch_version_tag::${{ inputs.branch_version_tag }}" \
|| { short_hash=$(git rev-parse --short HEAD); echo "::set-output name=branch_version_tag::dev-$short_hash"; }
- uses: actions/setup-java@v1
with:
java-version: "17"

- uses: actions/setup-node@v2
with:
node-version: "lts/gallium"

- name: Set up CI Gradle Properties
run: |
mkdir -p ~/.gradle/
cat > ~/.gradle/gradle.properties <<EOF
org.gradle.jvmargs=-Xmx8g -Xss4m --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
org.gradle.workers.max=8
org.gradle.vfs.watch=false
EOF
shell: bash

- name: Build
run: VERSION=${{ steps.parse-input.outputs.branch_version_tag }} SUB_BUILD=PLATFORM ./gradlew build --scan
shell: bash

- name: Publish to Maven Local
run: VERSION=${{ steps.parse-input.outputs.branch_version_tag }} SUB_BUILD=PLATFORM ./gradlew publishToMavenLocal
shell: bash

Loading

0 comments on commit 755e0d7

Please sign in to comment.