Source: MongoDB - Timestamp microseconds missing in Cursor - Incremental change capture

[prconstruct]

Enviroment

• Airbyte version: v0.30.24-alpha
• OS Version / Instance: Ubuntu 20.04 lts
• Deployment: Docker (in local)
• Source Connector and version: airbyte/source-mongodb-v2:0.1.3
• Destination Connector and version: S3 - JSONL
• Severity: Critical
• Step where error happened: Sync job

Current Behavior

Incremental change capture (cursor) is missing microseconds. Due to this same records are being captured repeatedly.
For example, Lets assume, I have chosen "UpdatedAt" field as cursor. The last processed record "UpdatedAt" timestamp is "2021-11-01 10:59:01.567Z". But the cursor is being captured as "2021-11-01 10:59:01Z". Eventhough there is no changes done at the source, in the next run, the same set of record is captured again because its checking for the records GREATER THAN "2021-11-01 10:59:01Z" so obviously the record which has timestamp "2021-11-01 10:59:01.567Z" is processed again.

Expected Behavior

The cursor should capture the exact data with microseconds (in the above example case, it should capture the cursor as "2021-11-01 10:59:01.567Z")

Steps to Reproduce

1. Create a collection and Insert some records in MongoDB. Make sure record has a timestamp field that will be used as a cursor and also the timestamp has microseconds.
2. Configure source (MongoDb), destination(S3), and connector as Incremental Append. selected a timestamp field as cursor.
3. Run the connector or run docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/{workspace} --network host --log-driver none airbyte/source-mongodb-v2:0.1.0 read --config source_config.json --catalog source_catalog.json
4. Rerun the connection again without making any changes in source. This should process the record again.

Are you willing to submit a PR?

Since I'm not proficient in Java, I'm unable to figure the issue in code.

[prconstruct]

@sherifnada , @irynakruk : Could you please prioritize the issue? Due to this, the same set of records is being processed again

[prconstruct]

@andriikorotkov : Getting an error after upgrading to source_mogodb_v2:0.1.5.

�[34msource�[0m - 2021-11-18 14:08:52 ERROR () LineGobbler(voidCall):82 - Exception in thread "main" java.util.NoSuchElementException
�[34msource�[0m - 2021-11-18 14:08:52 ERROR () LineGobbler(voidCall):82 - 	at com.mongodb.client.internal.MongoBatchCursorAdapter.next(MongoBatchCursorAdapter.java:60)
�[34msource�[0m - 2021-11-18 14:08:52 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.db.mongodb.MongoUtils.getFieldsName(MongoUtils.java:198)
�[34msource�[0m - 2021-11-18 14:08:52 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.db.mongodb.MongoUtils.getUniqueFields(MongoUtils.java:184)
�[34msource�[0m - 2021-11-18 14:08:52 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.source.mongodb.MongoDbSource.discoverInternal(MongoDbSource.java:119)
�[34msource�[0m - 2021-11-18 14:08:52 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.source.mongodb.MongoDbSource.discoverInternal(MongoDbSource.java:41)
�[34msource�[0m - 2021-11-18 14:08:52 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.source.relationaldb.AbstractDbSource.discoverWithoutSystemTables(AbstractDbSource.java:129)
�[34msource�[0m - 2021-11-18 14:08:52 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.source.relationaldb.AbstractDbSource.read(AbstractDbSource.java:105)
�[34msource�[0m - 2021-11-18 14:08:52 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:117)
�[34msource�[0m - 2021-11-18 14:08:52 ERROR () LineGobbler(voidCall):82 - 	at io.airbyte.integrations.source.mongodb.MongoDbSource.main(MongoDbSource.java:65)

[andriikorotkov]

@prconstruct, understood you. I'll post a fix today.

[sherifnada]

@andriikorotkov I reopened this issue -- could you add a regression test as part of the fix?