-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🎉 Destination MySQl - Added support for MySQL destination via TLS/SSL #6506
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
b35a363
add support SSL for MySQL destination
andriikorotkov d9763bd
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov fc1d7af
updated ssl tests and add documentation
andriikorotkov e27a485
updated code style
andriikorotkov 33f4468
changed default ssl value as true
andriikorotkov File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
22 changes: 1 addition & 21 deletions
22
.../java/io/airbyte/integrations/destination/mysql/SshKeyMySQLDestinationAcceptanceTest.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
22 changes: 1 addition & 21 deletions
22
...ion/java/io/airbyte/integrations/destination/mysql/SshMySQLDestinationAcceptanceTest.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
22 changes: 1 addition & 21 deletions
22
.../io/airbyte/integrations/destination/mysql/SshPasswordMySQLDestinationAcceptanceTest.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
145 changes: 145 additions & 0 deletions
145
...ion/java/io/airbyte/integrations/destination/mysql/SslMySQLDestinationAcceptanceTest.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| /* | ||
| * Copyright (c) 2021 Airbyte, Inc., all rights reserved. | ||
| */ | ||
|
|
||
| package io.airbyte.integrations.destination.mysql; | ||
|
|
||
| import com.fasterxml.jackson.databind.JsonNode; | ||
| import com.google.common.collect.ImmutableMap; | ||
| import io.airbyte.commons.json.Jsons; | ||
| import io.airbyte.db.Databases; | ||
| import io.airbyte.integrations.base.JavaBaseConstants; | ||
| import io.airbyte.integrations.destination.ExtendedNameTransformer; | ||
| import java.sql.SQLException; | ||
| import java.util.List; | ||
| import java.util.stream.Collectors; | ||
| import org.jooq.JSONFormat; | ||
| import org.jooq.JSONFormat.RecordFormat; | ||
| import org.jooq.SQLDialect; | ||
| import org.junit.jupiter.api.Test; | ||
| import org.testcontainers.containers.MySQLContainer; | ||
|
|
||
| public class SslMySQLDestinationAcceptanceTest extends MySQLDestinationAcceptanceTest { | ||
|
|
||
| private static final JSONFormat JSON_FORMAT = new JSONFormat().recordFormat(RecordFormat.OBJECT); | ||
|
|
||
| private MySQLContainer<?> db; | ||
| private final ExtendedNameTransformer namingResolver = new MySQLNameTransformer(); | ||
|
|
||
| @Override | ||
| protected JsonNode getConfig() { | ||
| return Jsons.jsonNode(ImmutableMap.builder() | ||
| .put("host", db.getHost()) | ||
| .put("username", db.getUsername()) | ||
| .put("password", db.getPassword()) | ||
| .put("database", db.getDatabaseName()) | ||
| .put("port", db.getFirstMappedPort()) | ||
| .put("ssl", true) | ||
| .build()); | ||
| } | ||
|
|
||
| @Override | ||
| protected JsonNode getFailCheckConfig() { | ||
| return Jsons.jsonNode(ImmutableMap.builder() | ||
| .put("host", db.getHost()) | ||
| .put("username", db.getUsername()) | ||
| .put("password", "wrong password") | ||
| .put("database", db.getDatabaseName()) | ||
| .put("port", db.getFirstMappedPort()) | ||
| .put("ssl", false) | ||
| .build()); | ||
| } | ||
|
|
||
| @Override | ||
| protected List<JsonNode> retrieveRecords(TestDestinationEnv testEnv, | ||
| String streamName, | ||
| String namespace, | ||
| JsonNode streamSchema) | ||
| throws Exception { | ||
| return retrieveRecordsFromTable(namingResolver.getRawTableName(streamName), namespace) | ||
| .stream() | ||
| .map(r -> Jsons.deserialize(r.get(JavaBaseConstants.COLUMN_NAME_DATA).asText())) | ||
| .collect(Collectors.toList()); | ||
| } | ||
|
|
||
| @Override | ||
| protected List<JsonNode> retrieveNormalizedRecords(TestDestinationEnv testEnv, String streamName, String namespace) throws Exception { | ||
| String tableName = namingResolver.getIdentifier(streamName); | ||
| String schema = namingResolver.getIdentifier(namespace); | ||
| return retrieveRecordsFromTable(tableName, schema); | ||
| } | ||
|
|
||
| @Override | ||
| @Test | ||
| public void testCustomDbtTransformations() { | ||
| // We need to create view for testing custom dbt transformations | ||
| executeQuery("GRANT CREATE VIEW ON *.* TO " + db.getUsername() + "@'%';"); | ||
| // overrides test with a no-op until https://github.com/dbt-labs/jaffle_shop/pull/8 is merged | ||
| // super.testCustomDbtTransformations(); | ||
| } | ||
|
|
||
| @Override | ||
| protected void setup(TestDestinationEnv testEnv) { | ||
| db = new MySQLContainer<>("mysql:8.0"); | ||
| db.start(); | ||
| setLocalInFileToTrue(); | ||
| revokeAllPermissions(); | ||
| grantCorrectPermissions(); | ||
| } | ||
|
|
||
| @Override | ||
| protected void tearDown(TestDestinationEnv testEnv) { | ||
| db.stop(); | ||
| db.close(); | ||
| } | ||
|
|
||
| private List<JsonNode> retrieveRecordsFromTable(String tableName, String schemaName) throws SQLException { | ||
| return Databases.createDatabase( | ||
| db.getUsername(), | ||
| db.getPassword(), | ||
| String.format("jdbc:mysql://%s:%s/%s?useSSL=true&requireSSL=true&verifyServerCertificate=false", | ||
| db.getHost(), | ||
| db.getFirstMappedPort(), | ||
| db.getDatabaseName()), | ||
| "com.mysql.cj.jdbc.Driver", | ||
| SQLDialect.MYSQL).query( | ||
| ctx -> ctx | ||
| .fetch(String.format("SELECT * FROM %s.%s ORDER BY %s ASC;", schemaName, tableName, | ||
| JavaBaseConstants.COLUMN_NAME_EMITTED_AT)) | ||
| .stream() | ||
| .map(r -> r.formatJSON(JSON_FORMAT)) | ||
| .map(Jsons::deserialize) | ||
| .collect(Collectors.toList())); | ||
| } | ||
|
|
||
| private void setLocalInFileToTrue() { | ||
| executeQuery("set global local_infile=true"); | ||
| } | ||
|
|
||
| private void revokeAllPermissions() { | ||
| executeQuery("REVOKE ALL PRIVILEGES, GRANT OPTION FROM " + db.getUsername() + "@'%';"); | ||
| } | ||
|
|
||
| private void grantCorrectPermissions() { | ||
| executeQuery("GRANT ALTER, CREATE, INSERT, SELECT, DROP ON *.* TO " + db.getUsername() + "@'%';"); | ||
| } | ||
|
|
||
| private void executeQuery(String query) { | ||
| try { | ||
| Databases.createDatabase( | ||
| "root", | ||
| "test", | ||
| String.format("jdbc:mysql://%s:%s/%s?useSSL=true&requireSSL=true&verifyServerCertificate=false", | ||
| db.getHost(), | ||
| db.getFirstMappedPort(), | ||
| db.getDatabaseName()), | ||
| "com.mysql.cj.jdbc.Driver", | ||
| SQLDialect.MYSQL).query( | ||
| ctx -> ctx | ||
| .execute(query)); | ||
| } catch (SQLException e) { | ||
| throw new RuntimeException(e); | ||
| } | ||
| } | ||
|
|
||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you make SSL the default if it is not specified? This will make it easier to create a secure fork of the connector in #6423
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sherifnada, I changed default ssl value to true in spec