From 755235e19f9c811b3bb4310f5898d94508fc6f4d Mon Sep 17 00:00:00 2001 From: LiRen Tu Date: Fri, 29 Apr 2022 23:14:58 -0700 Subject: [PATCH] =?UTF-8?q?=F0=9F=8E=89=20Jdbc=20sources:=20publish=20new?= =?UTF-8?q?=20version=20with=20adaptive=20fetch=20size=20(#12480)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Default scaffold to use adaptive streaming config * Switch more connectors to use adaptive streaming config * Bump version for cockroach db * Bump version for db2 * Bump mssql version * Bump mysql version * Bump oracle version * Bump postgres version * Bump redshift version * Bump snowflake version * Bump tidb version * auto-bump connector version * Fix db2 findbug issue * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * Fix more findbug issues * auto-bump connector version * auto-bump connector version * auto-bump connector version * Fix findbug issue for mysql-strict-encrypt * Fix findbugs issue for oracle source * auto-bump connector version * Remove suppress warnings annotation * Fix oracle encrypt tests * Fix oracle encrypt acceptance test Co-authored-by: Octavia Squidington III --- .../resources/seed/source_definitions.yaml | 18 +++++------ .../src/main/resources/seed/source_specs.yaml | 31 +++++++++++++------ .../java/io/airbyte/db/jdbc/JdbcUtils.java | 12 +++++-- .../{{pascalCase name}}Source.java.hbs | 8 ++--- ...trictEncryptDestinationAcceptanceTest.java | 6 ++-- .../Dockerfile | 2 +- .../connectors/source-cockroachdb/Dockerfile | 2 +- .../source/cockroachdb/CockroachDbSource.java | 4 +-- .../source-db2-strict-encrypt/Dockerfile | 2 +- ...ncryptSourceCertificateAcceptanceTest.java | 3 +- .../Db2JdbcSourceAcceptanceTest.java | 3 +- .../connectors/source-db2/Dockerfile | 2 +- .../Db2Source.java | 3 +- .../Db2SourceCertificateAcceptanceTest.java | 3 +- .../source/jdbc/AbstractJdbcSource.java | 6 +++- .../jdbc/test/JdbcSourceAcceptanceTest.java | 11 +++++-- .../source-mssql-strict-encrypt/Dockerfile | 2 +- .../connectors/source-mssql/Dockerfile | 2 +- .../source-mysql-strict-encrypt/Dockerfile | 2 +- ...StrictEncryptJdbcSourceAcceptanceTest.java | 3 +- .../connectors/source-mysql/Dockerfile | 2 +- .../source-oracle-strict-encrypt/Dockerfile | 2 +- .../OracleSourceNneAcceptanceTest.java | 18 +++++------ ...acleStrictEncryptSourceAcceptanceTest.java | 4 +-- ...StrictEncryptJdbcSourceAcceptanceTest.java | 31 ++++++++++--------- .../connectors/source-oracle/Dockerfile | 2 +- .../source/oracle/OracleSource.java | 10 ++++-- .../OracleJdbcSourceAcceptanceTest.java | 18 +++++------ .../source/oracle/OracleSourceTest.java | 5 +-- .../source/oracle/OracleStressTest.java | 13 ++++---- .../source-postgres-strict-encrypt/Dockerfile | 2 +- .../connectors/source-postgres/Dockerfile | 2 +- .../connectors/source-redshift/Dockerfile | 2 +- .../ScaffoldJavaJdbcSource.java | 8 ++--- .../connectors/source-snowflake/Dockerfile | 2 +- .../connectors/source-tidb/Dockerfile | 2 +- .../integrations/source/tidb/TiDBSource.java | 4 +-- docs/integrations/sources/cockroachdb.md | 3 +- docs/integrations/sources/db2.md | 2 +- docs/integrations/sources/mssql.md | 2 +- docs/integrations/sources/mysql.md | 1 + docs/integrations/sources/oracle.md | 2 +- docs/integrations/sources/postgres.md | 2 +- docs/integrations/sources/redshift.md | 1 + docs/integrations/sources/snowflake.md | 1 + docs/integrations/sources/tidb.md | 1 + 46 files changed, 155 insertions(+), 112 deletions(-) diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml index f10bb01214b7a..908b78df77c44 100644 --- a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml +++ b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml @@ -147,7 +147,7 @@ - name: Cockroachdb sourceDefinitionId: 9fa5862c-da7c-11eb-8d19-0242ac130003 dockerRepository: airbyte/source-cockroachdb - dockerImageTag: 0.1.11 + dockerImageTag: 0.1.12 documentationUrl: https://docs.airbyte.io/integrations/sources/cockroachdb icon: cockroachdb.svg sourceType: database @@ -348,7 +348,7 @@ - name: IBM Db2 sourceDefinitionId: 447e0381-3780-4b46-bb62-00a4e3c8b8e2 dockerRepository: airbyte/source-db2 - dockerImageTag: 0.1.9 + dockerImageTag: 0.1.10 documentationUrl: https://docs.airbyte.io/integrations/sources/db2 icon: db2.svg sourceType: database @@ -459,7 +459,7 @@ - name: Microsoft SQL Server (MSSQL) sourceDefinitionId: b5ea17b1-f170-46dc-bc31-cc744ca984c1 dockerRepository: airbyte/source-mssql - dockerImageTag: 0.3.21 + dockerImageTag: 0.3.22 documentationUrl: https://docs.airbyte.io/integrations/sources/mssql icon: mssql.svg sourceType: database @@ -501,7 +501,7 @@ - name: MySQL sourceDefinitionId: 435bb9a5-7887-4809-aa58-28c27df0d7ad dockerRepository: airbyte/source-mysql - dockerImageTag: 0.5.9 + dockerImageTag: 0.5.10 documentationUrl: https://docs.airbyte.io/integrations/sources/mysql icon: mysql.svg sourceType: database @@ -535,7 +535,7 @@ - name: Oracle DB sourceDefinitionId: b39a7370-74c3-45a6-ac3a-380d48520a83 dockerRepository: airbyte/source-oracle - dockerImageTag: 0.3.14 + dockerImageTag: 0.3.15 documentationUrl: https://docs.airbyte.io/integrations/sources/oracle icon: oracle.svg sourceType: database @@ -625,7 +625,7 @@ - name: Postgres sourceDefinitionId: decd338e-5647-4c0b-adf4-da0e75f5a750 dockerRepository: airbyte/source-postgres - dockerImageTag: 0.4.11 + dockerImageTag: 0.4.12 documentationUrl: https://docs.airbyte.io/integrations/sources/postgres icon: postgresql.svg sourceType: database @@ -667,7 +667,7 @@ - name: Redshift sourceDefinitionId: e87ffa8e-a3b5-f69c-9076-6011339de1f6 dockerRepository: airbyte/source-redshift - dockerImageTag: 0.3.9 + dockerImageTag: 0.3.10 documentationUrl: https://docs.airbyte.io/integrations/sources/redshift icon: redshift.svg sourceType: database @@ -751,7 +751,7 @@ - name: Snowflake sourceDefinitionId: e2d65910-8c8b-40a1-ae7d-ee2416b2bfa2 dockerRepository: airbyte/source-snowflake - dockerImageTag: 0.1.11 + dockerImageTag: 0.1.12 documentationUrl: https://docs.airbyte.io/integrations/sources/snowflake icon: snowflake.svg sourceType: database @@ -792,7 +792,7 @@ - name: TiDB sourceDefinitionId: 0dad1a35-ccf8-4d03-b73e-6788c00b13ae dockerRepository: airbyte/source-tidb - dockerImageTag: 0.1.0 + dockerImageTag: 0.1.1 documentationUrl: https://docs.airbyte.io/integrations/sources/tidb icon: tidb.svg sourceType: database diff --git a/airbyte-config/init/src/main/resources/seed/source_specs.yaml b/airbyte-config/init/src/main/resources/seed/source_specs.yaml index 76795b74b9a1e..95d0bd4860680 100644 --- a/airbyte-config/init/src/main/resources/seed/source_specs.yaml +++ b/airbyte-config/init/src/main/resources/seed/source_specs.yaml @@ -1305,7 +1305,7 @@ supportsNormalization: false supportsDBT: false supported_destination_sync_modes: [] -- dockerImage: "airbyte/source-cockroachdb:0.1.11" +- dockerImage: "airbyte/source-cockroachdb:0.1.12" spec: documentationUrl: "https://docs.airbyte.io/integrations/sources/cockroachdb" connectionSpecification: @@ -3578,7 +3578,7 @@ - - "client_secret" oauthFlowOutputParameters: - - "refresh_token" -- dockerImage: "airbyte/source-db2:0.1.9" +- dockerImage: "airbyte/source-db2:0.1.10" spec: documentationUrl: "https://docs.airbyte.io/integrations/sources/db2" connectionSpecification: @@ -4669,7 +4669,7 @@ supportsNormalization: false supportsDBT: false supported_destination_sync_modes: [] -- dockerImage: "airbyte/source-mssql:0.3.21" +- dockerImage: "airbyte/source-mssql:0.3.22" spec: documentationUrl: "https://docs.airbyte.io/integrations/destinations/mssql" connectionSpecification: @@ -4687,6 +4687,7 @@ description: "The hostname of the database." title: "Host" type: "string" + order: 0 port: description: "The port of the database." title: "Port" @@ -4695,21 +4696,32 @@ maximum: 65536 examples: - "1433" + order: 1 database: description: "The name of the database." title: "Database" type: "string" examples: - "master" + order: 2 username: description: "The username which is used to access the database." title: "Username" type: "string" + order: 3 password: description: "The password associated with the username." title: "Password" type: "string" airbyte_secret: true + order: 4 + jdbc_url_params: + title: "JDBC URL Params" + description: "Additional properties to pass to the JDBC URL string when\ + \ connecting to the database formatted as 'key=value' pairs separated\ + \ by the symbol '&'. (example: key1=value1&key2=value2&key3=value3)." + type: "string" + order: 5 ssl_method: title: "SSL Method" type: "object" @@ -4774,6 +4786,7 @@ enum: - "STANDARD" - "CDC" + order: 8 tunnel_method: type: "object" title: "SSH Tunnel Method" @@ -5368,7 +5381,7 @@ supportsNormalization: false supportsDBT: false supported_destination_sync_modes: [] -- dockerImage: "airbyte/source-mysql:0.5.9" +- dockerImage: "airbyte/source-mysql:0.5.10" spec: documentationUrl: "https://docs.airbyte.io/integrations/sources/mysql" connectionSpecification: @@ -5757,7 +5770,7 @@ supportsNormalization: false supportsDBT: false supported_destination_sync_modes: [] -- dockerImage: "airbyte/source-oracle:0.3.14" +- dockerImage: "airbyte/source-oracle:0.3.15" spec: documentationUrl: "https://docs.airbyte.io/integrations/sources/oracle" connectionSpecification: @@ -6478,7 +6491,7 @@ supportsNormalization: false supportsDBT: false supported_destination_sync_modes: [] -- dockerImage: "airbyte/source-postgres:0.4.11" +- dockerImage: "airbyte/source-postgres:0.4.12" spec: documentationUrl: "https://docs.airbyte.com/integrations/sources/postgres" connectionSpecification: @@ -6914,7 +6927,7 @@ supportsNormalization: false supportsDBT: false supported_destination_sync_modes: [] -- dockerImage: "airbyte/source-redshift:0.3.9" +- dockerImage: "airbyte/source-redshift:0.3.10" spec: documentationUrl: "https://docs.airbyte.io/integrations/destinations/redshift" connectionSpecification: @@ -7972,7 +7985,7 @@ - - "client_secret" oauthFlowOutputParameters: - - "refresh_token" -- dockerImage: "airbyte/source-snowflake:0.1.11" +- dockerImage: "airbyte/source-snowflake:0.1.12" spec: documentationUrl: "https://docs.airbyte.io/integrations/sources/snowflake" connectionSpecification: @@ -8493,7 +8506,7 @@ supportsNormalization: false supportsDBT: false supported_destination_sync_modes: [] -- dockerImage: "airbyte/source-tidb:0.1.0" +- dockerImage: "airbyte/source-tidb:0.1.1" spec: documentationUrl: "https://docs.airbyte.io/integrations/sources/tidb" connectionSpecification: diff --git a/airbyte-db/lib/src/main/java/io/airbyte/db/jdbc/JdbcUtils.java b/airbyte-db/lib/src/main/java/io/airbyte/db/jdbc/JdbcUtils.java index 05caf59336b02..50471c0da2034 100644 --- a/airbyte-db/lib/src/main/java/io/airbyte/db/jdbc/JdbcUtils.java +++ b/airbyte-db/lib/src/main/java/io/airbyte/db/jdbc/JdbcUtils.java @@ -29,17 +29,25 @@ public static String getFullyQualifiedTableName(final String schemaName, final S } public static Map parseJdbcParameters(final JsonNode config, final String jdbcUrlParamsKey) { + return parseJdbcParameters(config, jdbcUrlParamsKey, "&"); + } + + public static Map parseJdbcParameters(final JsonNode config, final String jdbcUrlParamsKey, final String delimiter) { if (config.has(jdbcUrlParamsKey)) { - return parseJdbcParameters(config.get(jdbcUrlParamsKey).asText()); + return parseJdbcParameters(config.get(jdbcUrlParamsKey).asText(), delimiter); } else { return Maps.newHashMap(); } } public static Map parseJdbcParameters(final String jdbcPropertiesString) { + return parseJdbcParameters(jdbcPropertiesString, "&"); + } + + public static Map parseJdbcParameters(final String jdbcPropertiesString, final String delimiter) { final Map parameters = new HashMap<>(); if (!jdbcPropertiesString.isBlank()) { - final String[] keyValuePairs = jdbcPropertiesString.split("&"); + final String[] keyValuePairs = jdbcPropertiesString.split(delimiter); for (final String kv : keyValuePairs) { final String[] split = kv.split("="); if (split.length == 2) { diff --git a/airbyte-integrations/connector-templates/source-java-jdbc/src/main/java/io/airbyte/integrations/source/{{snakeCase name}}/{{pascalCase name}}Source.java.hbs b/airbyte-integrations/connector-templates/source-java-jdbc/src/main/java/io/airbyte/integrations/source/{{snakeCase name}}/{{pascalCase name}}Source.java.hbs index d7babcc57203b..3d3350f9a0106 100644 --- a/airbyte-integrations/connector-templates/source-java-jdbc/src/main/java/io/airbyte/integrations/source/{{snakeCase name}}/{{pascalCase name}}Source.java.hbs +++ b/airbyte-integrations/connector-templates/source-java-jdbc/src/main/java/io/airbyte/integrations/source/{{snakeCase name}}/{{pascalCase name}}Source.java.hbs @@ -6,7 +6,7 @@ package io.airbyte.integrations.source.{{snakeCase name}}; import com.fasterxml.jackson.databind.JsonNode; import io.airbyte.db.jdbc.JdbcUtils; -import io.airbyte.db.jdbc.streaming.NoOpStreamingQueryConfig; +import io.airbyte.db.jdbc.streaming.AdaptiveStreamingQueryConfig; import io.airbyte.integrations.base.IntegrationRunner; import io.airbyte.integrations.base.Source; import io.airbyte.integrations.source.jdbc.AbstractJdbcSource; @@ -23,9 +23,9 @@ public class {{pascalCase name}}Source extends AbstractJdbcSource impl static final String DRIVER_CLASS = "driver_name_here"; public {{pascalCase name}}Source() { - // By default, NoOpStreamingQueryConfig class is used. If the JDBC supports custom - // fetch size, change it to AdaptiveStreamingQueryConfig for better performance. - super(DRIVER_CLASS, NoOpStreamingQueryConfig::new, JdbcUtils.getDefaultSourceOperations()); + // TODO: if the JDBC driver does not support custom fetch size, use NoOpStreamingQueryConfig + // instead of AdaptiveStreamingQueryConfig. + super(DRIVER_CLASS, AdaptiveStreamingQueryConfig::new, JdbcUtils.getDefaultSourceOperations()); } // TODO The config is based on spec.json, update according to your DB diff --git a/airbyte-integrations/connectors/destination-oracle-strict-encrypt/src/test-integration/java/io/airbyte/integrations/destination/oracle_strict_encrypt/OracleStrictEncryptDestinationAcceptanceTest.java b/airbyte-integrations/connectors/destination-oracle-strict-encrypt/src/test-integration/java/io/airbyte/integrations/destination/oracle_strict_encrypt/OracleStrictEncryptDestinationAcceptanceTest.java index b360964abcc4f..6cfde5013241d 100644 --- a/airbyte-integrations/connectors/destination-oracle-strict-encrypt/src/test-integration/java/io/airbyte/integrations/destination/oracle_strict_encrypt/OracleStrictEncryptDestinationAcceptanceTest.java +++ b/airbyte-integrations/connectors/destination-oracle-strict-encrypt/src/test-integration/java/io/airbyte/integrations/destination/oracle_strict_encrypt/OracleStrictEncryptDestinationAcceptanceTest.java @@ -178,8 +178,7 @@ public void testEncryption() throws SQLException { config.get("sid").asText()), "oracle.jdbc.driver.OracleDriver", JdbcUtils.parseJdbcParameters("oracle.net.encryption_client=REQUIRED;" + - "oracle.net.encryption_types_client=( " - + algorithm + " )")); + "oracle.net.encryption_types_client=( " + algorithm + " )", ";")); final String network_service_banner = "select network_service_banner from v$session_connect_info where sid in (select distinct sid from v$mystat)"; @@ -204,8 +203,7 @@ public void testCheckProtocol() throws SQLException { clone.get("sid").asText()), "oracle.jdbc.driver.OracleDriver", JdbcUtils.parseJdbcParameters("oracle.net.encryption_client=REQUIRED;" + - "oracle.net.encryption_types_client=( " - + algorithm + " )")); + "oracle.net.encryption_types_client=( " + algorithm + " )", ";")); final String network_service_banner = "SELECT sys_context('USERENV', 'NETWORK_PROTOCOL') as network_protocol FROM dual"; final List collect = database.unsafeQuery(network_service_banner).collect(Collectors.toList()); diff --git a/airbyte-integrations/connectors/source-cockroachdb-strict-encrypt/Dockerfile b/airbyte-integrations/connectors/source-cockroachdb-strict-encrypt/Dockerfile index 5e07282f51124..f68558cfd6077 100644 --- a/airbyte-integrations/connectors/source-cockroachdb-strict-encrypt/Dockerfile +++ b/airbyte-integrations/connectors/source-cockroachdb-strict-encrypt/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-cockroachdb-strict-encrypt COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.1.8 +LABEL io.airbyte.version=0.1.12 LABEL io.airbyte.name=airbyte/source-cockroachdb-strict-encrypt diff --git a/airbyte-integrations/connectors/source-cockroachdb/Dockerfile b/airbyte-integrations/connectors/source-cockroachdb/Dockerfile index 8a8456ed239de..8ae1e6922049a 100644 --- a/airbyte-integrations/connectors/source-cockroachdb/Dockerfile +++ b/airbyte-integrations/connectors/source-cockroachdb/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-cockroachdb COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.1.11 +LABEL io.airbyte.version=0.1.12 LABEL io.airbyte.name=airbyte/source-cockroachdb diff --git a/airbyte-integrations/connectors/source-cockroachdb/src/main/java/io/airbyte/integrations/source/cockroachdb/CockroachDbSource.java b/airbyte-integrations/connectors/source-cockroachdb/src/main/java/io/airbyte/integrations/source/cockroachdb/CockroachDbSource.java index bd2ebe01a3b36..dfe64063cf592 100644 --- a/airbyte-integrations/connectors/source-cockroachdb/src/main/java/io/airbyte/integrations/source/cockroachdb/CockroachDbSource.java +++ b/airbyte-integrations/connectors/source-cockroachdb/src/main/java/io/airbyte/integrations/source/cockroachdb/CockroachDbSource.java @@ -12,7 +12,7 @@ import io.airbyte.db.Databases; import io.airbyte.db.jdbc.JdbcDatabase; import io.airbyte.db.jdbc.JdbcUtils; -import io.airbyte.db.jdbc.streaming.NoOpStreamingQueryConfig; +import io.airbyte.db.jdbc.streaming.AdaptiveStreamingQueryConfig; import io.airbyte.integrations.base.IntegrationRunner; import io.airbyte.integrations.base.Source; import io.airbyte.integrations.base.ssh.SshWrappedSource; @@ -41,7 +41,7 @@ public class CockroachDbSource extends AbstractJdbcSource { public static final List PORT_KEY = List.of("port"); public CockroachDbSource() { - super(DRIVER_CLASS, NoOpStreamingQueryConfig::new, new CockroachJdbcSourceOperations()); + super(DRIVER_CLASS, AdaptiveStreamingQueryConfig::new, new CockroachJdbcSourceOperations()); } public static Source sshWrappedSource() { diff --git a/airbyte-integrations/connectors/source-db2-strict-encrypt/Dockerfile b/airbyte-integrations/connectors/source-db2-strict-encrypt/Dockerfile index f992a81d9cc05..dfa81a644c1e6 100644 --- a/airbyte-integrations/connectors/source-db2-strict-encrypt/Dockerfile +++ b/airbyte-integrations/connectors/source-db2-strict-encrypt/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-db2-strict-encrypt COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.1.6 +LABEL io.airbyte.version=0.1.10 LABEL io.airbyte.name=airbyte/source-db2-strict-encrypt diff --git a/airbyte-integrations/connectors/source-db2-strict-encrypt/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/Db2StrictEncryptSourceCertificateAcceptanceTest.java b/airbyte-integrations/connectors/source-db2-strict-encrypt/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/Db2StrictEncryptSourceCertificateAcceptanceTest.java index 63c23e74177a4..a21c00c2f0100 100644 --- a/airbyte-integrations/connectors/source-db2-strict-encrypt/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/Db2StrictEncryptSourceCertificateAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-db2-strict-encrypt/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/Db2StrictEncryptSourceCertificateAcceptanceTest.java @@ -25,6 +25,7 @@ import java.io.File; import java.io.IOException; import java.io.PrintWriter; +import java.nio.charset.StandardCharsets; import java.util.HashMap; import java.util.concurrent.TimeUnit; import org.testcontainers.containers.Db2Container; @@ -181,7 +182,7 @@ private String getCertificate() throws IOException, InterruptedException { private static void convertAndImportCertificate(final String certificate) throws IOException, InterruptedException { final Runtime run = Runtime.getRuntime(); - try (final PrintWriter out = new PrintWriter("certificate.pem")) { + try (final PrintWriter out = new PrintWriter("certificate.pem", StandardCharsets.UTF_8)) { out.print(certificate); } runProcess("openssl x509 -outform der -in certificate.pem -out certificate.der", run); diff --git a/airbyte-integrations/connectors/source-db2-strict-encrypt/src/test/java/io/airbyte/integrations/source/db2_strict_encrypt/Db2JdbcSourceAcceptanceTest.java b/airbyte-integrations/connectors/source-db2-strict-encrypt/src/test/java/io/airbyte/integrations/source/db2_strict_encrypt/Db2JdbcSourceAcceptanceTest.java index 0627551bfeb2a..61f79b0655d99 100644 --- a/airbyte-integrations/connectors/source-db2-strict-encrypt/src/test/java/io/airbyte/integrations/source/db2_strict_encrypt/Db2JdbcSourceAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-db2-strict-encrypt/src/test/java/io/airbyte/integrations/source/db2_strict_encrypt/Db2JdbcSourceAcceptanceTest.java @@ -19,6 +19,7 @@ import java.io.File; import java.io.IOException; import java.io.PrintWriter; +import java.nio.charset.StandardCharsets; import java.sql.JDBCType; import java.util.Collections; import java.util.Set; @@ -186,7 +187,7 @@ private static String getCertificate() throws IOException, InterruptedException private static void convertAndImportCertificate(final String certificate) throws IOException, InterruptedException { final Runtime run = Runtime.getRuntime(); - try (final PrintWriter out = new PrintWriter("certificate.pem")) { + try (final PrintWriter out = new PrintWriter("certificate.pem", StandardCharsets.UTF_8)) { out.print(certificate); } runProcess("openssl x509 -outform der -in certificate.pem -out certificate.der", run); diff --git a/airbyte-integrations/connectors/source-db2/Dockerfile b/airbyte-integrations/connectors/source-db2/Dockerfile index f658419be0068..d2e3e152e7cc9 100644 --- a/airbyte-integrations/connectors/source-db2/Dockerfile +++ b/airbyte-integrations/connectors/source-db2/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-db2 COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.1.9 +LABEL io.airbyte.version=0.1.10 LABEL io.airbyte.name=airbyte/source-db2 diff --git a/airbyte-integrations/connectors/source-db2/src/main/java/io.airbyte.integrations.source.db2/Db2Source.java b/airbyte-integrations/connectors/source-db2/src/main/java/io.airbyte.integrations.source.db2/Db2Source.java index 74b897c7d93b2..f48de8c9e2dc1 100644 --- a/airbyte-integrations/connectors/source-db2/src/main/java/io.airbyte.integrations.source.db2/Db2Source.java +++ b/airbyte-integrations/connectors/source-db2/src/main/java/io.airbyte.integrations.source.db2/Db2Source.java @@ -16,6 +16,7 @@ import io.airbyte.integrations.source.jdbc.dto.JdbcPrivilegeDto; import java.io.IOException; import java.io.PrintWriter; +import java.nio.charset.StandardCharsets; import java.sql.Connection; import java.sql.JDBCType; import java.sql.PreparedStatement; @@ -143,7 +144,7 @@ private static String getKeyStorePassword(final JsonNode encryptionKeyStorePassw private static void convertAndImportCertificate(final String certificate, final String keyStorePassword) throws IOException, InterruptedException { final Runtime run = Runtime.getRuntime(); - try (final PrintWriter out = new PrintWriter("certificate.pem")) { + try (final PrintWriter out = new PrintWriter("certificate.pem", StandardCharsets.UTF_8)) { out.print(certificate); } runProcess("openssl x509 -outform der -in certificate.pem -out certificate.der", run); diff --git a/airbyte-integrations/connectors/source-db2/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/Db2SourceCertificateAcceptanceTest.java b/airbyte-integrations/connectors/source-db2/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/Db2SourceCertificateAcceptanceTest.java index 3aa28e9dcaf21..f35bbb52e2095 100644 --- a/airbyte-integrations/connectors/source-db2/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/Db2SourceCertificateAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-db2/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/Db2SourceCertificateAcceptanceTest.java @@ -25,6 +25,7 @@ import java.io.File; import java.io.IOException; import java.io.PrintWriter; +import java.nio.charset.StandardCharsets; import java.util.HashMap; import java.util.concurrent.TimeUnit; import org.testcontainers.containers.Db2Container; @@ -175,7 +176,7 @@ private String getCertificate() throws IOException, InterruptedException { private static void convertAndImportCertificate(final String certificate) throws IOException, InterruptedException { final Runtime run = Runtime.getRuntime(); - try (final PrintWriter out = new PrintWriter("certificate.pem")) { + try (final PrintWriter out = new PrintWriter("certificate.pem", StandardCharsets.UTF_8)) { out.print(certificate); } runProcess("openssl x509 -outform der -in certificate.pem -out certificate.der", run); diff --git a/airbyte-integrations/connectors/source-jdbc/src/main/java/io/airbyte/integrations/source/jdbc/AbstractJdbcSource.java b/airbyte-integrations/connectors/source-jdbc/src/main/java/io/airbyte/integrations/source/jdbc/AbstractJdbcSource.java index 3ca4791c2981c..d7ee2d9380ee3 100644 --- a/airbyte-integrations/connectors/source-jdbc/src/main/java/io/airbyte/integrations/source/jdbc/AbstractJdbcSource.java +++ b/airbyte-integrations/connectors/source-jdbc/src/main/java/io/airbyte/integrations/source/jdbc/AbstractJdbcSource.java @@ -295,7 +295,7 @@ public JdbcDatabase createDatabase(final JsonNode config) throws SQLException { jdbcConfig.get("jdbc_url").asText(), driverClass, streamingQueryConfigProvider, - JdbcUtils.parseJdbcParameters(jdbcConfig, "connection_properties"), + JdbcUtils.parseJdbcParameters(jdbcConfig, "connection_properties", getJdbcParameterDelimiter()), sourceOperations); quoteString = (quoteString == null ? database.getMetaData().getIdentifierQuoteString() : quoteString); @@ -303,4 +303,8 @@ public JdbcDatabase createDatabase(final JsonNode config) throws SQLException { return database; } + protected String getJdbcParameterDelimiter() { + return "&"; + } + } diff --git a/airbyte-integrations/connectors/source-jdbc/src/testFixtures/java/io/airbyte/integrations/source/jdbc/test/JdbcSourceAcceptanceTest.java b/airbyte-integrations/connectors/source-jdbc/src/testFixtures/java/io/airbyte/integrations/source/jdbc/test/JdbcSourceAcceptanceTest.java index 8511ad9d05206..283de93b8d284 100644 --- a/airbyte-integrations/connectors/source-jdbc/src/testFixtures/java/io/airbyte/integrations/source/jdbc/test/JdbcSourceAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-jdbc/src/testFixtures/java/io/airbyte/integrations/source/jdbc/test/JdbcSourceAcceptanceTest.java @@ -25,6 +25,7 @@ import io.airbyte.db.jdbc.JdbcDatabase; import io.airbyte.db.jdbc.JdbcSourceOperations; import io.airbyte.db.jdbc.JdbcUtils; +import io.airbyte.db.jdbc.streaming.AdaptiveStreamingQueryConfig; import io.airbyte.integrations.base.Source; import io.airbyte.integrations.source.jdbc.AbstractJdbcSource; import io.airbyte.integrations.source.relationaldb.models.DbState; @@ -185,6 +186,10 @@ protected String primaryKeyClause(final List columns) { return clause.toString(); } + protected String getJdbcParameterDelimiter() { + return "&"; + } + public void setup() throws Exception { source = getSource(); config = getConfig(); @@ -192,12 +197,14 @@ public void setup() throws Exception { streamName = TABLE_NAME; - database = Databases.createJdbcDatabase( + database = Databases.createStreamingJdbcDatabase( jdbcConfig.get("username").asText(), jdbcConfig.has("password") ? jdbcConfig.get("password").asText() : null, jdbcConfig.get("jdbc_url").asText(), getDriverClass(), - JdbcUtils.parseJdbcParameters(jdbcConfig, "connection_properties")); + AdaptiveStreamingQueryConfig::new, + JdbcUtils.parseJdbcParameters(jdbcConfig, "connection_properties", getJdbcParameterDelimiter()), + JdbcUtils.getDefaultSourceOperations()); if (supportsSchemas()) { createSchemas(); diff --git a/airbyte-integrations/connectors/source-mssql-strict-encrypt/Dockerfile b/airbyte-integrations/connectors/source-mssql-strict-encrypt/Dockerfile index 015453ecde72a..ab3e22af5dcbc 100644 --- a/airbyte-integrations/connectors/source-mssql-strict-encrypt/Dockerfile +++ b/airbyte-integrations/connectors/source-mssql-strict-encrypt/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-mssql-strict-encrypt COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.1.11 +LABEL io.airbyte.version=0.3.22 LABEL io.airbyte.name=airbyte/source-mssql-strict-encrypt diff --git a/airbyte-integrations/connectors/source-mssql/Dockerfile b/airbyte-integrations/connectors/source-mssql/Dockerfile index f826af5a2e528..8091bbc9be8f0 100644 --- a/airbyte-integrations/connectors/source-mssql/Dockerfile +++ b/airbyte-integrations/connectors/source-mssql/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-mssql COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.3.21 +LABEL io.airbyte.version=0.3.22 LABEL io.airbyte.name=airbyte/source-mssql diff --git a/airbyte-integrations/connectors/source-mysql-strict-encrypt/Dockerfile b/airbyte-integrations/connectors/source-mysql-strict-encrypt/Dockerfile index a7759c8c9a5d8..4a95c3c9cd047 100644 --- a/airbyte-integrations/connectors/source-mysql-strict-encrypt/Dockerfile +++ b/airbyte-integrations/connectors/source-mysql-strict-encrypt/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-mysql-strict-encrypt COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.1.12 +LABEL io.airbyte.version=0.5.10 LABEL io.airbyte.name=airbyte/source-mysql-strict-encrypt diff --git a/airbyte-integrations/connectors/source-mysql-strict-encrypt/src/test/java/io/airbyte/integrations/source/mysql_strict_encrypt/MySqlStrictEncryptJdbcSourceAcceptanceTest.java b/airbyte-integrations/connectors/source-mysql-strict-encrypt/src/test/java/io/airbyte/integrations/source/mysql_strict_encrypt/MySqlStrictEncryptJdbcSourceAcceptanceTest.java index 6484aed0460a1..d6d38b2419c83 100644 --- a/airbyte-integrations/connectors/source-mysql-strict-encrypt/src/test/java/io/airbyte/integrations/source/mysql_strict_encrypt/MySqlStrictEncryptJdbcSourceAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-mysql-strict-encrypt/src/test/java/io/airbyte/integrations/source/mysql_strict_encrypt/MySqlStrictEncryptJdbcSourceAcceptanceTest.java @@ -36,7 +36,6 @@ class MySqlStrictEncryptJdbcSourceAcceptanceTest extends JdbcSourceAcceptanceTes protected static final String TEST_PASSWORD = "test"; protected static MySQLContainer container; - protected JsonNode config; protected Database database; @BeforeAll @@ -47,7 +46,7 @@ static void init() throws SQLException { .withEnv("MYSQL_ROOT_HOST", "%") .withEnv("MYSQL_ROOT_PASSWORD", TEST_PASSWORD); container.start(); - final Connection connection = DriverManager.getConnection(container.getJdbcUrl(), "root", TEST_PASSWORD); + final Connection connection = DriverManager.getConnection(container.getJdbcUrl(), "root", container.getPassword()); connection.createStatement().execute("GRANT ALL PRIVILEGES ON *.* TO '" + TEST_USER + "'@'%';\n"); } diff --git a/airbyte-integrations/connectors/source-mysql/Dockerfile b/airbyte-integrations/connectors/source-mysql/Dockerfile index 952f96f2c0d7c..1072e6d436724 100644 --- a/airbyte-integrations/connectors/source-mysql/Dockerfile +++ b/airbyte-integrations/connectors/source-mysql/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-mysql COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.5.9 +LABEL io.airbyte.version=0.5.10 LABEL io.airbyte.name=airbyte/source-mysql diff --git a/airbyte-integrations/connectors/source-oracle-strict-encrypt/Dockerfile b/airbyte-integrations/connectors/source-oracle-strict-encrypt/Dockerfile index f64d7feb84a27..96726701554ad 100644 --- a/airbyte-integrations/connectors/source-oracle-strict-encrypt/Dockerfile +++ b/airbyte-integrations/connectors/source-oracle-strict-encrypt/Dockerfile @@ -17,5 +17,5 @@ ENV TZ UTC COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.1.6 +LABEL io.airbyte.version=0.3.15 LABEL io.airbyte.name=airbyte/source-oracle-strict-encrypt diff --git a/airbyte-integrations/connectors/source-oracle-strict-encrypt/src/test-integration/java/io/airbyte/integrations/source/oracle_strict_encrypt/OracleSourceNneAcceptanceTest.java b/airbyte-integrations/connectors/source-oracle-strict-encrypt/src/test-integration/java/io/airbyte/integrations/source/oracle_strict_encrypt/OracleSourceNneAcceptanceTest.java index 0705682671001..f012936d4649e 100644 --- a/airbyte-integrations/connectors/source-oracle-strict-encrypt/src/test-integration/java/io/airbyte/integrations/source/oracle_strict_encrypt/OracleSourceNneAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-oracle-strict-encrypt/src/test-integration/java/io/airbyte/integrations/source/oracle_strict_encrypt/OracleSourceNneAcceptanceTest.java @@ -16,15 +16,14 @@ import io.airbyte.db.jdbc.JdbcUtils; import java.sql.SQLException; import java.util.List; -import java.util.stream.Collectors; import org.junit.jupiter.api.Test; public class OracleSourceNneAcceptanceTest extends OracleStrictEncryptSourceAcceptanceTest { @Test - public void testEncrytion() throws SQLException { - final JsonNode clone = Jsons.clone(getConfig()); - ((ObjectNode) clone).put("encryption", Jsons.jsonNode(ImmutableMap.builder() + public void testEncryption() throws SQLException { + final ObjectNode clone = (ObjectNode) Jsons.clone(getConfig()); + clone.set("encryption", Jsons.jsonNode(ImmutableMap.builder() .put("encryption_method", "client_nne") .put("encryption_algorithm", "3DES168") .build())); @@ -45,7 +44,7 @@ public void testEncrytion() throws SQLException { final String network_service_banner = "select network_service_banner from v$session_connect_info where sid in (select distinct sid from v$mystat)"; - final List collect = database.unsafeQuery(network_service_banner).collect(Collectors.toList()); + final List collect = database.unsafeQuery(network_service_banner).toList(); assertTrue(collect.get(2).get("NETWORK_SERVICE_BANNER").asText() .contains(algorithm + " Encryption")); @@ -53,8 +52,8 @@ public void testEncrytion() throws SQLException { @Test public void testCheckProtocol() throws SQLException { - final JsonNode clone = Jsons.clone(getConfig()); - ((ObjectNode) clone).put("encryption", Jsons.jsonNode(ImmutableMap.builder() + final ObjectNode clone = (ObjectNode) Jsons.clone(getConfig()); + clone.set("encryption", Jsons.jsonNode(ImmutableMap.builder() .put("encryption_method", "client_nne") .put("encryption_algorithm", "AES256") .build())); @@ -70,11 +69,10 @@ public void testCheckProtocol() throws SQLException { clone.get("sid").asText()), "oracle.jdbc.driver.OracleDriver", JdbcUtils.parseJdbcParameters("oracle.net.encryption_client=REQUIRED;" + - "oracle.net.encryption_types_client=( " - + algorithm + " )")); + "oracle.net.encryption_types_client=( " + algorithm + " )", ";")); final String network_service_banner = "SELECT sys_context('USERENV', 'NETWORK_PROTOCOL') as network_protocol FROM dual"; - final List collect = database.unsafeQuery(network_service_banner).collect(Collectors.toList()); + final List collect = database.unsafeQuery(network_service_banner).toList(); assertEquals("tcp", collect.get(0).get("NETWORK_PROTOCOL").asText()); } diff --git a/airbyte-integrations/connectors/source-oracle-strict-encrypt/src/test-integration/java/io/airbyte/integrations/source/oracle_strict_encrypt/OracleStrictEncryptSourceAcceptanceTest.java b/airbyte-integrations/connectors/source-oracle-strict-encrypt/src/test-integration/java/io/airbyte/integrations/source/oracle_strict_encrypt/OracleStrictEncryptSourceAcceptanceTest.java index 150c47cc90458..0106bd75a5129 100644 --- a/airbyte-integrations/connectors/source-oracle-strict-encrypt/src/test-integration/java/io/airbyte/integrations/source/oracle_strict_encrypt/OracleStrictEncryptSourceAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-oracle-strict-encrypt/src/test-integration/java/io/airbyte/integrations/source/oracle_strict_encrypt/OracleStrictEncryptSourceAcceptanceTest.java @@ -61,8 +61,8 @@ protected void setupEnvironment(final TestDestinationEnv environment) throws Exc config.get("port").asText(), config.get("sid").asText()), "oracle.jdbc.driver.OracleDriver", - JdbcUtils.parseJdbcParameters("oracle.net.encryption_client=REQUIRED&" + - "oracle.net.encryption_types_client=( 3DES168 )")); + JdbcUtils.parseJdbcParameters("oracle.net.encryption_client=REQUIRED;" + + "oracle.net.encryption_types_client=( 3DES168 )", ";")); database.execute(connection -> { connection.createStatement().execute("CREATE USER JDBC_SPACE IDENTIFIED BY JDBC_SPACE DEFAULT TABLESPACE USERS QUOTA UNLIMITED ON USERS"); diff --git a/airbyte-integrations/connectors/source-oracle-strict-encrypt/src/test/java/io/airbyte/integrations/source/oracle_strict_encrypt/OracleStrictEncryptJdbcSourceAcceptanceTest.java b/airbyte-integrations/connectors/source-oracle-strict-encrypt/src/test/java/io/airbyte/integrations/source/oracle_strict_encrypt/OracleStrictEncryptJdbcSourceAcceptanceTest.java index 437869f2de463..6311c64b820a0 100644 --- a/airbyte-integrations/connectors/source-oracle-strict-encrypt/src/test/java/io/airbyte/integrations/source/oracle_strict_encrypt/OracleStrictEncryptJdbcSourceAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-oracle-strict-encrypt/src/test/java/io/airbyte/integrations/source/oracle_strict_encrypt/OracleStrictEncryptJdbcSourceAcceptanceTest.java @@ -56,14 +56,6 @@ class OracleStrictEncryptJdbcSourceAcceptanceTest extends JdbcSourceAcceptanceTe @BeforeAll static void init() { - ORACLE_DB = new OracleContainer("epiclabs/docker-oracle-xe-11g") - .withEnv("NLS_DATE_FORMAT", "YYYY-MM-DD"); - ORACLE_DB.start(); - } - - @BeforeEach - public void setup() throws Exception { - SCHEMA_NAME = "JDBC_INTEGRATION_TEST1"; SCHEMA_NAME2 = "JDBC_INTEGRATION_TEST2"; TEST_SCHEMAS = ImmutableSet.of(SCHEMA_NAME, SCHEMA_NAME2); @@ -84,6 +76,13 @@ public void setup() throws Exception { ID_VALUE_4 = new BigDecimal(4); ID_VALUE_5 = new BigDecimal(5); + ORACLE_DB = new OracleContainer("epiclabs/docker-oracle-xe-11g") + .withEnv("NLS_DATE_FORMAT", "YYYY-MM-DD"); + ORACLE_DB.start(); + } + + @BeforeEach + public void setup() throws Exception { config = Jsons.jsonNode(ImmutableMap.builder() .put("host", ORACLE_DB.getHost()) .put("port", ORACLE_DB.getFirstMappedPort()) @@ -126,9 +125,8 @@ void cleanUpTables() throws SQLException { conn.createStatement().executeQuery(String.format("SELECT TABLE_NAME FROM ALL_TABLES WHERE OWNER = '%s'", schemaName)); while (resultSet.next()) { final String tableName = resultSet.getString("TABLE_NAME"); - final String tableNameProcessed = tableName.contains(" ") ? sourceOperations - .enquoteIdentifier(conn, tableName) : tableName; - conn.createStatement().executeQuery(String.format("DROP TABLE %s.%s", schemaName, tableNameProcessed)); + final String tableNameProcessed = tableName.contains(" ") ? sourceOperations.enquoteIdentifier(conn, tableName) : tableName; + conn.createStatement().executeQuery("DROP TABLE " + schemaName + "." + tableNameProcessed); } } if (!conn.isClosed()) @@ -180,7 +178,12 @@ public void createSchemas() throws SQLException { } } - public void executeOracleStatement(final String query) throws SQLException { + @Override + protected String getJdbcParameterDelimiter() { + return ";"; + } + + public void executeOracleStatement(final String query) { try (final Connection conn = DriverManager.getConnection( ORACLE_DB.getJdbcUrl(), ORACLE_DB.getUsername(), @@ -194,8 +197,8 @@ public void executeOracleStatement(final String query) throws SQLException { public static void logSQLException(final SQLException ex) { for (final Throwable e : ex) { - if (e instanceof SQLException) { - if (ignoreSQLException(((SQLException) e).getSQLState()) == false) { + if (e instanceof final SQLException sqlException) { + if (!ignoreSQLException(sqlException.getSQLState())) { LOGGER.info("SQLState: " + ((SQLException) e).getSQLState()); LOGGER.info("Error Code: " + ((SQLException) e).getErrorCode()); LOGGER.info("Message: " + e.getMessage()); diff --git a/airbyte-integrations/connectors/source-oracle/Dockerfile b/airbyte-integrations/connectors/source-oracle/Dockerfile index dc2ba25e64557..cfc0c7aeaa398 100644 --- a/airbyte-integrations/connectors/source-oracle/Dockerfile +++ b/airbyte-integrations/connectors/source-oracle/Dockerfile @@ -8,5 +8,5 @@ ENV TZ UTC COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar RUN tar xf ${APPLICATION}.tar --strip-components=1 -LABEL io.airbyte.version=0.3.14 +LABEL io.airbyte.version=0.3.15 LABEL io.airbyte.name=airbyte/source-oracle diff --git a/airbyte-integrations/connectors/source-oracle/src/main/java/io/airbyte/integrations/source/oracle/OracleSource.java b/airbyte-integrations/connectors/source-oracle/src/main/java/io/airbyte/integrations/source/oracle/OracleSource.java index 94813f54fff3e..acd8e04c6a58b 100644 --- a/airbyte-integrations/connectors/source-oracle/src/main/java/io/airbyte/integrations/source/oracle/OracleSource.java +++ b/airbyte-integrations/connectors/source-oracle/src/main/java/io/airbyte/integrations/source/oracle/OracleSource.java @@ -18,6 +18,7 @@ import io.airbyte.protocol.models.CommonField; import java.io.IOException; import java.io.PrintWriter; +import java.nio.charset.StandardCharsets; import java.sql.JDBCType; import java.util.ArrayList; import java.util.List; @@ -92,7 +93,7 @@ public JsonNode toDatabaseConfig(final JsonNode config) { } } if (!additionalParameters.isEmpty()) { - final String connectionParams = String.join(";", additionalParameters); + final String connectionParams = String.join(getJdbcParameterDelimiter(), additionalParameters); configBuilder.put("connection_properties", connectionParams); } @@ -129,7 +130,7 @@ private Protocol obtainConnectionProtocol(final JsonNode encryption, final List< private static void convertAndImportCertificate(final String certificate) throws IOException, InterruptedException { final Runtime run = Runtime.getRuntime(); - try (final PrintWriter out = new PrintWriter("certificate.pem")) { + try (final PrintWriter out = new PrintWriter("certificate.pem", StandardCharsets.UTF_8)) { out.print(certificate); } runProcess("openssl x509 -outform der -in certificate.pem -out certificate.der", run); @@ -170,6 +171,11 @@ public Set getExcludedInternalNameSpaces() { return Set.of(); } + @Override + protected String getJdbcParameterDelimiter() { + return ";"; + } + public static void main(final String[] args) throws Exception { final Source source = OracleSource.sshWrappedSource(); LOGGER.info("starting source: {}", OracleSource.class); diff --git a/airbyte-integrations/connectors/source-oracle/src/test/java/io/airbyte/integrations/source/oracle/OracleJdbcSourceAcceptanceTest.java b/airbyte-integrations/connectors/source-oracle/src/test/java/io/airbyte/integrations/source/oracle/OracleJdbcSourceAcceptanceTest.java index 0d5c7a604403f..a608165340c96 100644 --- a/airbyte-integrations/connectors/source-oracle/src/test/java/io/airbyte/integrations/source/oracle/OracleJdbcSourceAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-oracle/src/test/java/io/airbyte/integrations/source/oracle/OracleJdbcSourceAcceptanceTest.java @@ -55,15 +55,6 @@ class OracleJdbcSourceAcceptanceTest extends JdbcSourceAcceptanceTest { @BeforeAll static void init() { // Oracle returns uppercase values - - ORACLE_DB = new OracleContainer("epiclabs/docker-oracle-xe-11g") - .withEnv("NLS_DATE_FORMAT", "YYYY-MM-DD"); - ORACLE_DB.start(); - } - - @BeforeEach - public void setup() throws Exception { - SCHEMA_NAME = "JDBC_INTEGRATION_TEST1"; SCHEMA_NAME2 = "JDBC_INTEGRATION_TEST2"; TEST_SCHEMAS = ImmutableSet.of(SCHEMA_NAME, SCHEMA_NAME2); @@ -84,6 +75,13 @@ public void setup() throws Exception { ID_VALUE_4 = new BigDecimal(4); ID_VALUE_5 = new BigDecimal(5); + ORACLE_DB = new OracleContainer("epiclabs/docker-oracle-xe-11g") + .withEnv("NLS_DATE_FORMAT", "YYYY-MM-DD"); + ORACLE_DB.start(); + } + + @BeforeEach + public void setup() throws Exception { config = Jsons.jsonNode(ImmutableMap.builder() .put("host", ORACLE_DB.getHost()) .put("port", ORACLE_DB.getFirstMappedPort()) @@ -124,7 +122,7 @@ void cleanUpTables() throws SQLException { final String tableName = resultSet.getString("TABLE_NAME"); final String tableNameProcessed = tableName.contains(" ") ? sourceOperations .enquoteIdentifier(conn, tableName) : tableName; - conn.createStatement().executeQuery(String.format("DROP TABLE %s.%s", schemaName, tableNameProcessed)); + conn.createStatement().executeQuery("DROP TABLE " + schemaName + "." + tableNameProcessed); } } if (!conn.isClosed()) diff --git a/airbyte-integrations/connectors/source-oracle/src/test/java/io/airbyte/integrations/source/oracle/OracleSourceTest.java b/airbyte-integrations/connectors/source-oracle/src/test/java/io/airbyte/integrations/source/oracle/OracleSourceTest.java index 529cbaeff4c92..ff86b31c011fd 100644 --- a/airbyte-integrations/connectors/source-oracle/src/test/java/io/airbyte/integrations/source/oracle/OracleSourceTest.java +++ b/airbyte-integrations/connectors/source-oracle/src/test/java/io/airbyte/integrations/source/oracle/OracleSourceTest.java @@ -24,6 +24,7 @@ import io.airbyte.protocol.models.JsonSchemaType; import io.airbyte.protocol.models.SyncMode; import java.math.BigDecimal; +import java.nio.charset.StandardCharsets; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -45,8 +46,8 @@ class OracleSourceTest { Field.of("IMAGE", JsonSchemaType.STRING)) .withSupportedSyncModes(Lists.newArrayList(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL)))); private static final ConfiguredAirbyteCatalog CONFIGURED_CATALOG = CatalogHelpers.toDefaultConfiguredCatalog(CATALOG); - private static final Set ASCII_MESSAGES = Sets.newHashSet( - createRecord(STREAM_NAME, map("ID", new BigDecimal("1.0"), "NAME", "user", "IMAGE", "last_summer.png".getBytes()))); + private static final Set ASCII_MESSAGES = Sets.newHashSet(createRecord(STREAM_NAME, + map("ID", new BigDecimal("1.0"), "NAME", "user", "IMAGE", "last_summer.png".getBytes(StandardCharsets.UTF_8)))); private static OracleContainer ORACLE_DB; diff --git a/airbyte-integrations/connectors/source-oracle/src/test/java/io/airbyte/integrations/source/oracle/OracleStressTest.java b/airbyte-integrations/connectors/source-oracle/src/test/java/io/airbyte/integrations/source/oracle/OracleStressTest.java index cdca797efd006..cbc25fd3aa208 100644 --- a/airbyte-integrations/connectors/source-oracle/src/test/java/io/airbyte/integrations/source/oracle/OracleStressTest.java +++ b/airbyte-integrations/connectors/source-oracle/src/test/java/io/airbyte/integrations/source/oracle/OracleStressTest.java @@ -38,18 +38,18 @@ class OracleStressTest extends JdbcStressTest { @BeforeAll static void init() { - ORACLE_DB = new OracleContainer("epiclabs/docker-oracle-xe-11g"); - ORACLE_DB.start(); - } - - @BeforeEach - public void setup() throws Exception { TABLE_NAME = "ID_AND_NAME"; COL_ID = "ID"; COL_NAME = "NAME"; COL_ID_TYPE = "NUMBER(38,0)"; INSERT_STATEMENT = "INTO id_and_name (id, name) VALUES (%s,'picard-%s')"; + ORACLE_DB = new OracleContainer("epiclabs/docker-oracle-xe-11g"); + ORACLE_DB.start(); + } + + @BeforeEach + public void setup() throws Exception { config = Jsons.jsonNode(ImmutableMap.builder() .put("host", ORACLE_DB.getHost()) .put("port", ORACLE_DB.getFirstMappedPort()) @@ -57,7 +57,6 @@ public void setup() throws Exception { .put("username", ORACLE_DB.getUsername()) .put("password", ORACLE_DB.getPassword()) .build()); - super.setup(); } diff --git a/airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile b/airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile index 6b6b28beb9abb..e2fdc14b44d95 100644 --- a/airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile +++ b/airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-postgres-strict-encrypt COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.1.11 +LABEL io.airbyte.version=0.4.12 LABEL io.airbyte.name=airbyte/source-postgres-strict-encrypt diff --git a/airbyte-integrations/connectors/source-postgres/Dockerfile b/airbyte-integrations/connectors/source-postgres/Dockerfile index 6e5cc0d203778..9ae29ae6eb225 100644 --- a/airbyte-integrations/connectors/source-postgres/Dockerfile +++ b/airbyte-integrations/connectors/source-postgres/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-postgres COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.4.11 +LABEL io.airbyte.version=0.4.12 LABEL io.airbyte.name=airbyte/source-postgres diff --git a/airbyte-integrations/connectors/source-redshift/Dockerfile b/airbyte-integrations/connectors/source-redshift/Dockerfile index af8cc61f41873..a743455d2227c 100644 --- a/airbyte-integrations/connectors/source-redshift/Dockerfile +++ b/airbyte-integrations/connectors/source-redshift/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-redshift COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.3.9 +LABEL io.airbyte.version=0.3.10 LABEL io.airbyte.name=airbyte/source-redshift diff --git a/airbyte-integrations/connectors/source-scaffold-java-jdbc/src/main/java/io/airbyte/integrations/source/scaffold_java_jdbc/ScaffoldJavaJdbcSource.java b/airbyte-integrations/connectors/source-scaffold-java-jdbc/src/main/java/io/airbyte/integrations/source/scaffold_java_jdbc/ScaffoldJavaJdbcSource.java index 1f8e82d00496f..e2be871f2000e 100644 --- a/airbyte-integrations/connectors/source-scaffold-java-jdbc/src/main/java/io/airbyte/integrations/source/scaffold_java_jdbc/ScaffoldJavaJdbcSource.java +++ b/airbyte-integrations/connectors/source-scaffold-java-jdbc/src/main/java/io/airbyte/integrations/source/scaffold_java_jdbc/ScaffoldJavaJdbcSource.java @@ -6,7 +6,7 @@ import com.fasterxml.jackson.databind.JsonNode; import io.airbyte.db.jdbc.JdbcUtils; -import io.airbyte.db.jdbc.streaming.NoOpStreamingQueryConfig; +import io.airbyte.db.jdbc.streaming.AdaptiveStreamingQueryConfig; import io.airbyte.integrations.base.IntegrationRunner; import io.airbyte.integrations.base.Source; import io.airbyte.integrations.source.jdbc.AbstractJdbcSource; @@ -23,9 +23,9 @@ public class ScaffoldJavaJdbcSource extends AbstractJdbcSource impleme static final String DRIVER_CLASS = "driver_name_here"; public ScaffoldJavaJdbcSource() { - // By default, NoOpStreamingQueryConfig class is used. If the JDBC supports custom - // fetch size, change it to AdaptiveStreamingQueryConfig for better performance. - super(DRIVER_CLASS, NoOpStreamingQueryConfig::new, JdbcUtils.getDefaultSourceOperations()); + // TODO: if the JDBC driver does not support custom fetch size, use NoOpStreamingQueryConfig + // instead of AdaptiveStreamingQueryConfig. + super(DRIVER_CLASS, AdaptiveStreamingQueryConfig::new, JdbcUtils.getDefaultSourceOperations()); } // TODO The config is based on spec.json, update according to your DB diff --git a/airbyte-integrations/connectors/source-snowflake/Dockerfile b/airbyte-integrations/connectors/source-snowflake/Dockerfile index f2d1461977b56..7d904bc10687f 100644 --- a/airbyte-integrations/connectors/source-snowflake/Dockerfile +++ b/airbyte-integrations/connectors/source-snowflake/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-snowflake COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.1.11 +LABEL io.airbyte.version=0.1.12 LABEL io.airbyte.name=airbyte/source-snowflake diff --git a/airbyte-integrations/connectors/source-tidb/Dockerfile b/airbyte-integrations/connectors/source-tidb/Dockerfile index 6cd5e75bcfbe1..6179f1f2b654b 100755 --- a/airbyte-integrations/connectors/source-tidb/Dockerfile +++ b/airbyte-integrations/connectors/source-tidb/Dockerfile @@ -17,5 +17,5 @@ ENV APPLICATION source-tidb COPY --from=build /airbyte /airbyte # Airbyte's build system uses these labels to know what to name and tag the docker images produced by this Dockerfile. -LABEL io.airbyte.version=0.1.0 +LABEL io.airbyte.version=0.1.1 LABEL io.airbyte.name=airbyte/source-tidb diff --git a/airbyte-integrations/connectors/source-tidb/src/main/java/io/airbyte/integrations/source/tidb/TiDBSource.java b/airbyte-integrations/connectors/source-tidb/src/main/java/io/airbyte/integrations/source/tidb/TiDBSource.java index 370236b8d6917..f10efc8b6e163 100644 --- a/airbyte-integrations/connectors/source-tidb/src/main/java/io/airbyte/integrations/source/tidb/TiDBSource.java +++ b/airbyte-integrations/connectors/source-tidb/src/main/java/io/airbyte/integrations/source/tidb/TiDBSource.java @@ -8,7 +8,7 @@ import com.google.common.collect.ImmutableMap; import com.mysql.cj.MysqlType; import io.airbyte.commons.json.Jsons; -import io.airbyte.db.jdbc.streaming.NoOpStreamingQueryConfig; +import io.airbyte.db.jdbc.streaming.AdaptiveStreamingQueryConfig; import io.airbyte.integrations.base.IntegrationRunner; import io.airbyte.integrations.base.Source; import io.airbyte.integrations.base.ssh.SshWrappedSource; @@ -33,7 +33,7 @@ public static Source sshWrappedSource() { } public TiDBSource() { - super(DRIVER_CLASS, NoOpStreamingQueryConfig::new, new TiDBSourceOperations()); + super(DRIVER_CLASS, AdaptiveStreamingQueryConfig::new, new TiDBSourceOperations()); } @Override diff --git a/docs/integrations/sources/cockroachdb.md b/docs/integrations/sources/cockroachdb.md index d31a5571f4cba..3de1b54f3b0f3 100644 --- a/docs/integrations/sources/cockroachdb.md +++ b/docs/integrations/sources/cockroachdb.md @@ -95,6 +95,7 @@ Your database user should now be ready for use with Airbyte. | Version | Date | Pull Request | Subject | |:--------|:-----------| :--- | :--- | +| 0.1.12 | 2022-04-29 | [12480](https://github.com/airbytehq/airbyte/pull/12480) | Query tables with adaptive fetch size to optimize JDBC memory consumption | | 0.1.11 | 2022-04-06 | [11729](https://github.com/airbytehq/airbyte/pull/11729) | Bump mina-sshd from 2.7.0 to 2.8.0 | | 0.1.10 | 2022-02-24 | [10235](https://github.com/airbytehq/airbyte/pull/10235) | Fix Replication Failure due Multiple portal opens | | 0.1.9 | 2022-02-21 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats | @@ -110,6 +111,7 @@ Your database user should now be ready for use with Airbyte. | Version | Date | Pull Request | Subject | |:--------| :--- | :--- | :--- | +| 0.1.12 | 2022-04-29 | [12480](https://github.com/airbytehq/airbyte/pull/12480) | Query tables with adaptive fetch size to optimize JDBC memory consumption | | 0.1.8 | 2022-04-06 | [11729](https://github.com/airbytehq/airbyte/pull/11729) | Bump mina-sshd from 2.7.0 to 2.8.0 | | 0.1.6 | 2022-02-21 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats | | 0.1.5 | 2022-02-18 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Updated timestamp transformation with microseconds | @@ -118,4 +120,3 @@ Your database user should now be ready for use with Airbyte. | 0.1.2 | 2021-12-24 | [9004](https://github.com/airbytehq/airbyte/pull/9004) | User can see only permmited tables during discovery | | 0.1.1 | 2021-12-24 | [8958](https://github.com/airbytehq/airbyte/pull/8958) | Add support for JdbcType.ARRAY | | 0.1.0 | 2021-11-23 | [7457](https://github.com/airbytehq/airbyte/pull/7457) | CockroachDb source: Add only encrypted version for the connector | - diff --git a/docs/integrations/sources/db2.md b/docs/integrations/sources/db2.md index 173f7973d94a6..7a029ecb619ed 100644 --- a/docs/integrations/sources/db2.md +++ b/docs/integrations/sources/db2.md @@ -62,6 +62,7 @@ You can also enter your own password for the keystore, but if you don't, the pas | Version | Date | Pull Request | Subject | | :--- | :--- | :--- | :--- | +| 0.1.10 | 2022-04-29 | [12480](https://github.com/airbytehq/airbyte/pull/12480) | Query tables with adaptive fetch size to optimize JDBC memory consumption | | 0.1.9 | 2022-02-21 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats | | 0.1.8 | 2022-02-18 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Updated timestamp transformation with microseconds | | 0.1.7 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | Add `-XX:+ExitOnOutOfMemoryError` JVM option |**** @@ -72,4 +73,3 @@ You can also enter your own password for the keystore, but if you don't, the pas | 0.1.2 | 2021-10-25 | [7355](https://github.com/airbytehq/airbyte/pull/7355) | Added ssl support | | 0.1.1 | 2021-08-13 | [4699](https://github.com/airbytehq/airbyte/pull/4699) | Added json config validator | | 0.1.0 | 2021-06-22 | [4197](https://github.com/airbytehq/airbyte/pull/4197) | New Source: IBM DB2 | - diff --git a/docs/integrations/sources/mssql.md b/docs/integrations/sources/mssql.md index 69495aec35560..43790f189caf4 100644 --- a/docs/integrations/sources/mssql.md +++ b/docs/integrations/sources/mssql.md @@ -294,6 +294,7 @@ If you do not see a type in this list, assume that it is coerced into a string. | Version | Date | Pull Request | Subject | |:--------|:-----------| :----------------------------------------------------- | :------------------------------------- | +| 0.3.22 | 2022-04-29 | [12480](https://github.com/airbytehq/airbyte/pull/12480) | Query tables with adaptive fetch size to optimize JDBC memory consumption | | 0.3.21 | 2022-04-11 | [11729](https://github.com/airbytehq/airbyte/pull/11729) | Bump mina-sshd from 2.7.0 to 2.8.0 | | 0.3.19 | 2022-03-31 | [11495](https://github.com/airbytehq/airbyte/pull/11495) | Adds Support to Chinese MSSQL Server Agent | | 0.3.18 | 2022-03-29 | [11010](https://github.com/airbytehq/airbyte/pull/11010) | Adds JDBC Params | @@ -326,4 +327,3 @@ If you do not see a type in this list, assume that it is coerced into a string. | 0.1.6 | 2020-12-09 | [1172](https://github.com/airbytehq/airbyte/pull/1172) | Support incremental sync | | | 0.1.5 | 2020-11-30 | [1038](https://github.com/airbytehq/airbyte/pull/1038) | Change JDBC sources to discover more than standard schemas | | | 0.1.4 | 2020-11-30 | [1046](https://github.com/airbytehq/airbyte/pull/1046) | Add connectors using an index YAML file | | - diff --git a/docs/integrations/sources/mysql.md b/docs/integrations/sources/mysql.md index e9d8298526d00..811c26f111f98 100644 --- a/docs/integrations/sources/mysql.md +++ b/docs/integrations/sources/mysql.md @@ -185,6 +185,7 @@ If you do not see a type in this list, assume that it is coerced into a string. | Version | Date | Pull Request | Subject | |:--------|:-----------|:-----------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------| +| 0.5.10 | 2022-04-29 | [12480](https://github.com/airbytehq/airbyte/pull/12480) | Query tables with adaptive fetch size to optimize JDBC memory consumption | | 0.5.9 | 2022-04-06 | [11729](https://github.com/airbytehq/airbyte/pull/11729) | Bump mina-sshd from 2.7.0 to 2.8.0 | | 0.5.6 | 2022-02-21 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats | | 0.5.5 | 2022-02-18 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Updated timestamp transformation with microseconds | diff --git a/docs/integrations/sources/oracle.md b/docs/integrations/sources/oracle.md index 0b2432879f37f..031e55def8e21 100644 --- a/docs/integrations/sources/oracle.md +++ b/docs/integrations/sources/oracle.md @@ -132,6 +132,7 @@ Airbite has the ability to connect to the Oracle source with 3 network connectiv | Version | Date | Pull Request | Subject | |:--------| :--- | :--- |:------------------------------------------------| +| 0.3.15 | 2022-04-29 | [12480](https://github.com/airbytehq/airbyte/pull/12480) | Query tables with adaptive fetch size to optimize JDBC memory consumption | | 0.3.14 | 2022-02-21 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats | | 0.3.13 | 2022-02-18 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Updated timestamp transformation with microseconds | | 0.3.12 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | Add `-XX:+ExitOnOutOfMemoryError` JVM option | @@ -145,4 +146,3 @@ Airbite has the ability to connect to the Oracle source with 3 network connectiv | 0.3.4 | 2021-09-01 | [6038](https://github.com/airbytehq/airbyte/pull/6038) | Remove automatic filtering of system schemas. | | 0.3.3 | 2021-09-01 | [5779](https://github.com/airbytehq/airbyte/pull/5779) | Ability to only discover certain schemas. | | 0.3.2 | 2021-08-13 | [4699](https://github.com/airbytehq/airbyte/pull/4699) | Added json config validator. | - diff --git a/docs/integrations/sources/postgres.md b/docs/integrations/sources/postgres.md index 6b349289c4800..f5220081c57f5 100644 --- a/docs/integrations/sources/postgres.md +++ b/docs/integrations/sources/postgres.md @@ -270,6 +270,7 @@ According to Postgres [documentation](https://www.postgresql.org/docs/14/datatyp | Version | Date | Pull Request | Subject | |:--------|:-----------|:-------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------| +| 0.4.12 | 2022-04-29 | [12480](https://github.com/airbytehq/airbyte/pull/12480) | Query tables with adaptive fetch size to optimize JDBC memory consumption | | 0.4.11 | 2022-04-11 | [11729](https://github.com/airbytehq/airbyte/pull/11729) | Bump mina-sshd from 2.7.0 to 2.8.0 | | 0.4.10 | 2022-04-08 | [11798](https://github.com/airbytehq/airbyte/pull/11798) | Fixed roles for fetching materialized view processing | | 0.4.8 | 2022-02-21 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats | @@ -313,4 +314,3 @@ According to Postgres [documentation](https://www.postgresql.org/docs/14/datatyp | 0.1.6 | 2020-12-09 | [1172](https://github.com/airbytehq/airbyte/pull/1172) | Support incremental sync | | 0.1.5 | 2020-11-30 | [1038](https://github.com/airbytehq/airbyte/pull/1038) | Change JDBC sources to discover more than standard schemas | | 0.1.4 | 2020-11-30 | [1046](https://github.com/airbytehq/airbyte/pull/1046) | Add connectors using an index YAML file | - diff --git a/docs/integrations/sources/redshift.md b/docs/integrations/sources/redshift.md index 34ca674590756..33bd04918bc26 100644 --- a/docs/integrations/sources/redshift.md +++ b/docs/integrations/sources/redshift.md @@ -54,6 +54,7 @@ All Redshift connections are encrypted using SSL | Version | Date | Pull Request | Subject | | :------ | :-------- | :----- | :------ | +| 0.3.10 | 2022-04-29 | [12480](https://github.com/airbytehq/airbyte/pull/12480) | Query tables with adaptive fetch size to optimize JDBC memory consumption |0 | 0.3.9 | 2022-02-21 | [9744](https://github.com/airbytehq/airbyte/pull/9744) | List only the tables on which the user has SELECT permissions. | 0.3.8 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | Add `-XX:+ExitOnOutOfMemoryError` JVM option | | 0.3.7 | 2022-01-26 | [9721](https://github.com/airbytehq/airbyte/pull/9721) | Added schema selection | diff --git a/docs/integrations/sources/snowflake.md b/docs/integrations/sources/snowflake.md index f500081c8b2a6..ddeca9ac1d035 100644 --- a/docs/integrations/sources/snowflake.md +++ b/docs/integrations/sources/snowflake.md @@ -103,6 +103,7 @@ Field | Description | | Version | Date | Pull Request | Subject | | :--- | :--- | :--- | :--- | +| 0.1.12 | 2022-04-29 | [12480](https://github.com/airbytehq/airbyte/pull/12480) | Query tables with adaptive fetch size to optimize JDBC memory consumption | | 0.1.11 | 2022-04-27 | [10953](https://github.com/airbytehq/airbyte/pull/10953) | Implement OAuth flow | | 0.1.9 | 2022-02-21 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats | | 0.1.8 | 2022-02-18 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Updated timestamp transformation with microseconds | diff --git a/docs/integrations/sources/tidb.md b/docs/integrations/sources/tidb.md index 93e6864d77be3..5cea0d66f7331 100644 --- a/docs/integrations/sources/tidb.md +++ b/docs/integrations/sources/tidb.md @@ -120,4 +120,5 @@ Using this feature requires additional configuration, when creating the source. | Version | Date | Pull Request | Subject | | :------ | :--- | :----------- | ------- | +| 0.1.1 | 2022-04-29 | [12480](https://github.com/airbytehq/airbyte/pull/12480) | Query tables with adaptive fetch size to optimize JDBC memory consumption | | 0.1.0 | 2022-04-19 | [11283](https://github.com/airbytehq/airbyte/pull/11283) | Initial Release |