Mssql destination: enable DAT tests, use nvarchar and datetime2 by default #12305
Conversation
…ar and datetime2 by default
github-actions
bot
added
area/connectors
sashaNeshcheret
added
connectors/destination/mssql
and removed
area/worker
| @@ -51,3 +51,4 @@ | |||
| {"type": "RECORD", "record": {"stream": "1_prefix_startwith_number", "emitted_at": 1602637990800, "data": { "id": 2, "date": "2020-09-01", "text": "hi 4"}}} | |||
|
|
|||
| {"type":"RECORD","record":{"stream":"multiple_column_names_conflicts","data":{"id":1,"User Id":"chris","user_id":42,"User id":300,"user id": 102,"UserId":101},"emitted_at":1623959926}} | |||
| {"type":"RECORD","record":{"stream":"special_characters","data":{"id":1,"character_1":"some random special characters: ࠈൡሗ","user_id":42,"character_2":"some random special characters: žõ⥁🌞"},"emitted_at":1623959932}} | |||
There was a problem hiding this comment.
If I'm not mistaken, this test will effect normalization of all destinations. Are you sure that we are safe to do it?
There was a problem hiding this comment.
Yes, you are right. @grubberr will check if it works for all destinations and will make changes in scope of the PR if not.
| @@ -91,6 +92,8 @@ protected boolean compareJsonNodes(final JsonNode expectedValue, final JsonNode | |||
| return compareDateValues(expectedValue.asText(), actualValue.asText()); | |||
| } else if (expectedValue.isArray() && actualValue.isArray()) { | |||
| return compareArrays(expectedValue, actualValue); | |||
| } else if (expectedValue.isArray() && isQuotedArray(actualValue)) { | |||
There was a problem hiding this comment.
this is a specific case of your destination. It should be handled inside your destination implementation.
I already fixed few destinations with the similar problem. You need to parse text values on the destination and provide actual value as true array here.
There was a problem hiding this comment.
Thanks, will do it.
|
/test connector=bases/base-normalization
|
github-actions
bot
added
area/platform
|
/test connector=connectors/destination-mssql
|
|
/test connector=connectors/destination-mssql-strict-encrypt
|
|
/test connector=connectors/destination-mssql-strict-encrypt
|
|
/test connector=connectors/destination-mssql
|
.../io/airbyte/integrations/standardtest/destination/comparator/AdvancedTestDataComparator.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
github-actions
bot
added
the
area/documentation
|
/publish connector=bases/base-normalization
|
|
/publish connector=bases/base-normalization
|
| @@ -1,6 +1,6 @@ | |||
| SELECT | |||
| forms | |||
| FROM | |||
| {{ REF('pokemon') }} | |||
| {{ ref('pokemon') }} | |||
There was a problem hiding this comment.
Looks like it's an innocent victim of the Format.
There was a problem hiding this comment.
if you know how to fix
there is an issue for this :)
#12384
|
/test connector=connectors/destination-mssql
|
|
/publish connector=bases/base-normalization
|
|
/publish connector=bases/base-normalization
|
|
/publish connector=bases/base-normalization
|
|
/publish connector=bases/base-normalization
|
This reverts commit 67503fe.
|
I see that it's published |
…fault (#12305) * Mssql destination: enable DAT tests for mssql destination, use nvarchar and datetime2 by default * Mssql destination: update array handling in test * Mssql destination: update array and JSON handling in test * Mssql destination: remove unused method * bugfix bigquery tests, dataset_location added * basic-normalization.md updated Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * Mssql destination: change parent class for mssql test Co-authored-by: Sergey Chvalyuk <grubberr@gmail.com>
What
Enabled DAT tests for Mssql destination, updated data types.
How
Overridden methods for enabling DAT tests for Mssql destination, replace varchar to nvarchar for supporting special characters, and datetime to datetime2 for increasing range of values.
Recommended reading order
datatypes.sqlMSSQLDestinationAcceptanceTest.javaMSSQLTestDataComparator.javax.java🚨 User Impact 🚨
Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/SUMMARY.mddocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampledocs/integrations/README.mdairbyte-integrations/builds.mdAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing/publishcommand described hereUpdating a connector
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing/publishcommand described hereConnector Generator
-scaffoldin their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplatesthen checking in your changesTests
Unit
Put your unit tests output here.
Integration
Put your integration tests output here.
Acceptance
Put your acceptance tests output here.