-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Postgres source: add integration with data dog #21533
Postgres source: add integration with data dog #21533
Conversation
airbyte-commons-worker/src/main/java/io/airbyte/workers/process/DockerProcessFactory.java
Outdated
Show resolved
Hide resolved
airbyte-commons-worker/src/main/java/io/airbyte/workers/process/DockerProcessFactory.java
Outdated
Show resolved
Hide resolved
...rs/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java
Outdated
Show resolved
Hide resolved
...-workers/src/main/java/io/airbyte/workers/config/ContainerOrchestratorConfigBeanFactory.java
Outdated
Show resolved
Hide resolved
/publish connector=connectors/source-postgres
if you have connectors that successfully published but failed definition generation, follow step 4 here |
…tgres-source' into omneshcheret/20423-add-dd-to-postgres-source
/publish connector=connectors/source-postgres-strict-encrypt
if you have connectors that successfully published but failed definition generation, follow step 4 here |
…add-dd-to-postgres-source # Conflicts: # airbyte-config/init/src/main/resources/seed/source_definitions.yaml # airbyte-config/init/src/main/resources/seed/source_specs.yaml # airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile # airbyte-integrations/connectors/source-postgres/Dockerfile
Affected Connector ReportNOTE
|
Connector | Version | Changelog | Publish |
---|---|---|---|
source-alloydb |
2.0.3 |
✅ | ✅ |
source-alloydb-strict-encrypt |
2.0.3 |
🔵 (ignored) |
🔵 (ignored) |
source-bigquery |
0.2.3 |
✅ | ✅ |
source-clickhouse |
0.1.16 |
✅ | ✅ |
source-clickhouse-strict-encrypt |
0.1.16 |
🔵 (ignored) |
🔵 (ignored) |
source-cockroachdb |
0.1.21 |
✅ | ✅ |
source-cockroachdb-strict-encrypt |
0.1.21 |
🔵 (ignored) |
🔵 (ignored) |
source-db2 |
0.1.18 |
✅ | ✅ |
source-db2-strict-encrypt |
0.1.18 |
🔵 (ignored) |
🔵 (ignored) |
source-dynamodb |
0.1.2 |
✅ | ✅ |
source-e2e-test |
2.1.4 |
✅ | ✅ |
source-e2e-test-cloud |
2.1.4 |
🔵 (ignored) |
🔵 (ignored) |
source-elasticsearch |
0.1.1 |
✅ | ✅ |
source-jdbc |
0.3.5 |
🔵 (ignored) |
🔵 (ignored) |
source-kafka |
0.2.3 |
✅ | ✅ |
source-mongodb-strict-encrypt |
0.1.19 |
🔵 (ignored) |
🔵 (ignored) |
source-mongodb-v2 |
0.1.19 |
✅ | ✅ |
source-mssql |
1.0.4 |
✅ | ✅ |
source-mssql-strict-encrypt |
1.0.4 |
🔵 (ignored) |
🔵 (ignored) |
source-mysql |
2.0.5 |
✅ | ✅ |
source-mysql-strict-encrypt |
2.0.5 |
🔵 (ignored) |
🔵 (ignored) |
source-oracle |
0.3.23 |
✅ | ✅ |
source-oracle-strict-encrypt |
0.3.23 |
🔵 (ignored) |
🔵 (ignored) |
source-postgres |
2.0.5 |
✅ | ✅ |
source-postgres-strict-encrypt |
2.0.5 |
🔵 (ignored) |
🔵 (ignored) |
source-redshift |
0.3.16 |
✅ | ✅ |
source-relational-db |
0.3.1 |
🔵 (ignored) |
🔵 (ignored) |
source-scaffold-java-jdbc |
0.1.0 |
🔵 (ignored) |
🔵 (ignored) |
source-sftp |
0.1.2 |
✅ | ✅ |
source-snowflake |
0.1.31 |
✅ | ✅ |
source-tidb |
0.2.3 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
❌ Destinations (48)
Connector | Version | Changelog | Publish |
---|---|---|---|
destination-azure-blob-storage |
0.2.0 |
✅ | ✅ |
destination-bigquery |
1.2.17 |
✅ | ✅ |
destination-bigquery-denormalized |
1.2.17 |
✅ | ✅ |
destination-cassandra |
0.1.4 |
✅ | ✅ |
destination-clickhouse |
0.2.2 |
✅ | ✅ |
destination-clickhouse-strict-encrypt |
0.2.2 |
🔵 (ignored) |
🔵 (ignored) |
destination-csv |
1.0.0 |
✅ | ✅ |
destination-databricks |
1.0.0 |
✅ | ✅ |
destination-dev-null |
0.2.7 |
🔵 (ignored) |
🔵 (ignored) |
destination-doris |
0.1.0 |
✅ | ✅ |
destination-dynamodb |
0.1.7 |
✅ | ✅ |
destination-e2e-test |
0.2.4 |
✅ | ✅ |
destination-elasticsearch |
0.1.6 |
✅ | ✅ |
destination-elasticsearch-strict-encrypt |
0.1.6 |
🔵 (ignored) |
🔵 (ignored) |
destination-exasol |
0.1.1 |
✅ | ✅ |
destination-gcs |
0.2.16 |
✅ | ✅ |
destination-iceberg |
0.1.0 |
✅ | ✅ |
destination-jdbc |
0.3.14 |
🔵 (ignored) |
🔵 (ignored) |
destination-kafka |
0.1.10 |
✅ | ✅ |
destination-keen |
0.2.4 |
✅ | ✅ |
destination-kinesis |
0.1.5 |
✅ | ✅ |
destination-local-json |
0.2.11 |
✅ | ✅ |
destination-mariadb-columnstore |
0.1.7 |
✅ | ✅ |
destination-mongodb |
0.1.9 |
✅ | ✅ |
destination-mongodb-strict-encrypt |
0.1.9 |
🔵 (ignored) |
🔵 (ignored) |
destination-mqtt |
0.1.3 |
✅ | ✅ |
destination-mssql |
0.1.22 |
✅ | ✅ |
destination-mssql-strict-encrypt |
0.1.22 |
🔵 (ignored) |
🔵 (ignored) |
destination-mysql |
0.1.20 |
✅ | ✅ |
destination-mysql-strict-encrypt |
❌ 0.1.21 (mismatch: 0.1.20 ) |
🔵 (ignored) |
🔵 (ignored) |
destination-oracle |
0.1.19 |
✅ | ✅ |
destination-oracle-strict-encrypt |
0.1.19 |
🔵 (ignored) |
🔵 (ignored) |
destination-postgres |
0.3.26 |
✅ | ✅ |
destination-postgres-strict-encrypt |
0.3.26 |
🔵 (ignored) |
🔵 (ignored) |
destination-pubsub |
0.2.0 |
✅ | ✅ |
destination-pulsar |
0.1.3 |
✅ | ✅ |
destination-r2 |
0.1.0 |
✅ | ✅ |
destination-redis |
0.1.4 |
✅ | ✅ |
destination-redpanda |
0.1.0 |
✅ | ✅ |
destination-redshift |
0.4.3 |
✅ | ✅ |
destination-rockset |
0.1.4 |
✅ | ✅ |
destination-s3 |
0.3.22 |
✅ | ✅ |
destination-s3-glue |
0.1.2 |
✅ | ✅ |
destination-scylla |
0.1.3 |
✅ | ✅ |
destination-snowflake |
0.4.54 |
✅ | ✅ |
destination-teradata |
0.1.1 |
✅ | ✅ |
destination-tidb |
0.1.0 |
✅ | ✅ |
destination-yugabytedb |
0.1.1 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Other Modules (0)
Actionable Items
(click to expand)
Category | Status | Actionable Item |
---|---|---|
Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
Publish | ⚠ not in seed |
The connector is not in the seed file (e.g. source_definitions.yaml ), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug. |
❌ diff seed version |
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version. |
…add-dd-to-postgres-source
/publish connector=connectors/source-postgres-strict-encrypt
if you have connectors that successfully published but failed definition generation, follow step 4 here |
/publish connector=connectors/source-postgres
if you have connectors that successfully published but failed definition generation, follow step 4 here |
/publish connector=connectors/source-postgres
if you have connectors that successfully published but failed definition generation, follow step 4 here |
…add-dd-to-postgres-source
…tgres-source' into omneshcheret/20423-add-dd-to-postgres-source
/test connector=connectors/source-postgres
Build FailedTest summary info:
|
/test connector=connectors/source-postgres
Build PassedTest summary info:
|
/publish connector=connectors/source-postgres
if you have connectors that successfully published but failed definition generation, follow step 4 here |
…add-dd-to-postgres-source # Conflicts: # docs/integrations/sources/postgres.md
…tgres-source' into omneshcheret/20423-add-dd-to-postgres-source
/publish connector=connectors/source-postgres if you have connectors that successfully published but failed definition generation, follow step 4 here |
/publish connector=connectors/source-postgres-strict-encrypt
if you have connectors that successfully published but failed definition generation, follow step 4 here |
/publish connector=connectors/source-postgres
if you have connectors that successfully published but failed definition generation, follow step 4 here |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
approving cause PR cant be merged without my approval
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work!
I'd like to chat a little about how tracing is enabled - do we pass the connector an API Key or a Collector URL? Is there a related Platform PR that handles this?
I'm also curious as to why we went with DD directly, and not open-telemetry to keep the connectors themself as agnostic as possible.
@@ -6,6 +6,9 @@ RUN yum install -y tar openssl && yum clean all | |||
|
|||
WORKDIR /airbyte | |||
|
|||
# Add the Datadog Java APM agent | |||
ADD https://dtdg.co/latest-java-tracer dd-java-agent.jar |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this mean that we are using the in-process collector or will we be forwarding metrics to the DD collector pod?
* Source postgres: add dd for env running locally * Source postgres: add dd for running in cloud * auto-bump connector version * Source postgres: bump postgres strict-encrypt version * Source postgres: filter datadog agent env variables just for postgres source * Source postgres: format * Source postgres: clean code * Source postgres: pass java opts for all connectors * Source postgres: temp removing dd agent from image * Source postgres: add dd agent to image * Source postgres: temp revert adding dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp removing hardcoded dd env variable * Source postgres: temp added hardcoded dd env variable * Source postgres: temp added hardcoded dd env variable * Source postgres: temp added hardcoded dd env variable * Source postgres: rename to java_opts and pass data dog host * Source postgres: add vars to kube pods * Source postgres: add vars to kube pods * Source postgres: add vars to kube pods * Source postgres: add Trace to more methods * Source postgres: add Trace to more methods * Source postgres: add Trace to more methods * Source postgres: temp reverting service name removing * Source postgres: temp reverting service name removing * Source postgres: temp reverting service name removing * Source postgres: temp adding trace to integration runner * Source postgres: temp adding trace to integration runner * Source postgres: bump postgres source dd version * Source postgres: bump postgres source dd version * Source postgres: revert temp changes * Source postgres: merge with master * Automated Commit - Formatting Changes * Source postgres: move dd java agent to base java * Source postgres: move dd java agent to base java * Source postgres: clean up * Source postgres: clean up * Automated Change * Source postgres: clean up * Source postgres: bump version * Source postgres: bump version for test * Source postgres: temp bump version * Source postgres: bump version * Automated Change * Source postgres: bump version * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: sashaNeshcheret <sashaNeshcheret@users.noreply.github.com>
* Source postgres: add dd for env running locally * Source postgres: add dd for running in cloud * auto-bump connector version * Source postgres: bump postgres strict-encrypt version * Source postgres: filter datadog agent env variables just for postgres source * Source postgres: format * Source postgres: clean code * Source postgres: pass java opts for all connectors * Source postgres: temp removing dd agent from image * Source postgres: add dd agent to image * Source postgres: temp revert adding dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp hardcoded dd env variable * Source postgres: temp removing hardcoded dd env variable * Source postgres: temp added hardcoded dd env variable * Source postgres: temp added hardcoded dd env variable * Source postgres: temp added hardcoded dd env variable * Source postgres: rename to java_opts and pass data dog host * Source postgres: add vars to kube pods * Source postgres: add vars to kube pods * Source postgres: add vars to kube pods * Source postgres: add Trace to more methods * Source postgres: add Trace to more methods * Source postgres: add Trace to more methods * Source postgres: temp reverting service name removing * Source postgres: temp reverting service name removing * Source postgres: temp reverting service name removing * Source postgres: temp adding trace to integration runner * Source postgres: temp adding trace to integration runner * Source postgres: bump postgres source dd version * Source postgres: bump postgres source dd version * Source postgres: revert temp changes * Source postgres: merge with master * Automated Commit - Formatting Changes * Source postgres: move dd java agent to base java * Source postgres: move dd java agent to base java * Source postgres: clean up * Source postgres: clean up * Automated Change * Source postgres: clean up * Source postgres: bump version * Source postgres: bump version for test * Source postgres: temp bump version * Source postgres: bump version * Automated Change * Source postgres: bump version * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: sashaNeshcheret <sashaNeshcheret@users.noreply.github.com>
What
Add datadog support for postgres source
How
We added dd-agent to postgres source. All other work will be provided from platform side
Added utils for tracing, and marked needed methods with @trace
Recommended reading order
airbyte-integrations/bases/base-java/Dockerfile
ApmTraceUtils.java
🚨 User Impact 🚨
Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampledocs/integrations/README.md
airbyte-integrations/builds.md
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereUpdating a connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereConnector Generator
-scaffold
in their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates
then checking in your changesTests
Unit
Put your unit tests output here.
Integration
Put your integration tests output here.
Acceptance
Put your acceptance tests output here.