Source MSSQL - Sync CDC with New Changes Only option not working.

[quannh-uet]

Environment

• Airbyte version: 0.35.65-alpha
• OS Version / Instance: AWS EC2
• Deployment: Docker
• Source Connector and version: MSSQL 0.4.24
• Destination Connector and version: Snowflake 0.4.38
• Step where error happened: Setup new connection and sync data

Current Behavior

I setup new source connector as CDC mode with New Changes Only option
After that, when I sync data, job read a table as Existing and New options.

Expected Behavior

Sync job need run as New Changes Only option

Logs

Source connector config:

{
  "host": "xx.xx.xx.xx",
  "port": 1433,
  "schemas": [
    "dbo"
  ],
  "database": "xxx",
  "password": "xxx",
  "username": "xxx",
  "ssl_method": {
    "ssl_method": "unencrypted"
  },
  "tunnel_method": {
    "tunnel_method": "NO_TUNNEL"
  },
  "replication_method": {
    "method": "CDC",
    "data_to_sync": "New Changes Only",
    "snapshot_isolation": "Snapshot"
  }
}

Debezium config on log:
snapshot.mode = initial.

2022-11-02 12:14:13 �[44msource�[0m > Starting SqlServerConnectorTask with configuration:
2022-11-02 12:14:13 �[44msource�[0m >    connector.class = io.debezium.connector.sqlserver.SqlServerConnector
2022-11-02 12:14:13 �[44msource�[0m >    max.queue.size = 8192
2022-11-02 12:14:13 �[44msource�[0m >    provide.transaction.metadata = false
2022-11-02 12:14:13 �[44msource�[0m >    include.schema.changes = false
2022-11-02 12:14:13 �[44msource�[0m >    offset.storage.file.filename = /tmp/cdc-state-offset757126953703380547/offset.dat
2022-11-02 12:14:13 �[44msource�[0m >    decimal.handling.mode = string
2022-11-02 12:14:13 �[44msource�[0m >    converters = mssql_converter
2022-11-02 12:14:13 �[44msource�[0m >    database.history.file.filename = /tmp/cdc-db-history6392748082955249532/dbhistory.dat
2022-11-02 12:14:13 �[44msource�[0m >    database.user = xxx
2022-11-02 12:14:13 �[44msource�[0m >    database.dbname = xxx
2022-11-02 12:14:13 �[44msource�[0m >    offset.storage = org.apache.kafka.connect.storage.FileOffsetBackingStore
2022-11-02 12:14:13 �[44msource�[0m >    mssql_converter.type = io.airbyte.integrations.debezium.internals.MSSQLConverter
2022-11-02 12:14:13 �[44msource�[0m >    database.server.name = xxx
2022-11-02 12:14:13 �[44msource�[0m >    snapshot.isolation.mode = snapshot
2022-11-02 12:14:13 �[44msource�[0m >    database.port = 1433
2022-11-02 12:14:13 �[44msource�[0m >    offset.flush.interval.ms = 1000
2022-11-02 12:14:13 �[44msource�[0m >    key.converter.schemas.enable = false
2022-11-02 12:14:13 �[44msource�[0m >    database.hostname = 172.38.17.93
2022-11-02 12:14:13 �[44msource�[0m >    database.password = ********
2022-11-02 12:14:13 �[44msource�[0m >    name = xx
2022-11-02 12:14:13 �[44msource�[0m >    value.converter.schemas.enable = false
2022-11-02 12:14:13 �[44msource�[0m >    max.batch.size = 2048
2022-11-02 12:14:13 �[44msource�[0m >    table.include.list = dbo.xxx
2022-11-02 12:14:13 �[44msource�[0m >    snapshot.mode = initial
2022-11-02 12:14:13 �[44msource�[0m >    database.history = io.airbyte.integrations.debezium.internals.FilteredFileDatabaseHistory
2022-11-02 12:14:13 �[44msource�[0m >    database.include.list = xxx

Steps to Reproduce

1. Setup new source connector with CDC mode and New Changes Only option
2. Run sync job
3. View log

Are you willing to submit a PR?

Yes, I'll create a PR fixing that.

• PR MSSQL Source : Standardize spec.json for DB connectors that support log-based CDC replication #16215 do not change logic correctly when you setup as New Changes Only option.
• https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mssql/src/main/java/io/airbyte/integrations/source/mssql/MssqlCdcHelper.java#L125

[natalyjazzviolin]

Hi @quannh-uet ! Could you please doublecheck the Airbyte version you listed, and correct it if needed?

[quannh-uet]

Hi @natalyjazzviolin, Airbyte, Source connector and Destination connector version is correct.
I'll create a PR fix this bug.

[quannh-uet]

Hi @natalyjazzviolin could you review my PR above?

[natalyjazzviolin]

Hey @quannh-uet , we have a code freeze in place until next week, then the team would be able to review and merge your PR. Meanwhile, could you run your integration tests locally, make sure everything passes and include the logs in the PR? This will make the review process quicker!