🐛 📝 don't missparse +/-infinity in postgres-source #31856

stephane-airbyte · 2023-10-25T22:49:50Z

fixes #19952

Postgres has the ability to store +infinity and -infinity in its date, timestamp and timestamp with timezone datatypes. Unfortunately, we don't handle them, and PG-JDBC does some unnatural stuff to still return a java.sql.* java type: It returns a special value of a Timestamp/Date/OffsetDateTime, which is not their max value either. Debezium will do the same thing.

So in this change, I look for those special values and return +infinity and -infinity instead of a non-sensical date.

There's some changes to the CDK because the postgres debezum converter is in there.

I added some tests to the AbstractPostgresSourceDatatypeTest, but I'm not 100% clear what the destination's behavior will be WRT the new values

vercel · 2023-10-25T22:49:56Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Nov 9, 2023 6:22pm

CLAassistant · 2023-10-25T22:49:56Z

All committers have signed the CLA.

github-actions · 2023-10-25T22:50:11Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

stephane-airbyte · 2023-10-31T23:46:08Z

...te/integrations/io/airbyte/integration_tests/sources/AbstractPostgresSourceDatatypeTest.java

@@ -634,23 +635,6 @@ protected void addTimeWithTimeZoneTest() {
    }
  }

-  protected void addTimestampWithInfinityValuesTest() {


removed the test because 1) it was doing the wrong thing and 2) we added a lot more tests in the original test block

stephane-airbyte · 2023-10-31T23:46:43Z

.../integrations/io/airbyte/integration_tests/sources/CdcWalLogsPostgresSourceDatatypeTest.java

@@ -196,25 +196,6 @@ protected void addTimeWithTimeZoneTest() {
    }
  }

-  @Override
-  protected void addTimestampWithInfinityValuesTest() {


removed this because 1) it's testing the wrong thing and 2) we have a lot more infinity tests in the abstract class above

stephane-airbyte · 2023-11-07T22:10:25Z

airbyte-cdk/java/airbyte-cdk/airbyte-commons-protocol/src/test/resources/WellKnownTypes.json

+          "enum": ["+infinity", "-infinity"]
+        }
+      ],
+      "description": "An RFC 3339\u00a75.6 full-time, extended with +/-infinity"
    },
    "TimeWithoutTimezone": {
      "type": "string",


stephane-airbyte · 2023-11-07T22:13:02Z

airbyte-integrations/connectors/source-postgres/build.gradle

@@ -9,7 +9,7 @@ plugins {
 airbyteJavaConnector {
    cdkVersionRequired = '0.4.1'


push new CDK

github-actions · 2023-11-07T23:55:57Z

Coverage report for source-postgres

File	Coverage [35.11%]	❌
PostgresSourceOperations.java	35.11%	❌

Total Project Coverage	71.75%	🍏

postamar · 2023-11-08T21:27:10Z

airbyte-integrations/connectors/source-postgres/build.gradle

    features = ['db-sources']
-    useLocalCdk = false
+    useLocalCdk = true


Reminder to unset this before merging.

stephane-airbyte · 2023-11-08T22:37:04Z

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/6804841041
❌ Publish Java CDK version=0.4.7version=0.4.6 failed!

stephane-airbyte · 2023-11-08T23:29:35Z

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/6805251415
❌ Publish Java CDK version=0.4.7version=0.4.6 failed!

stephane-airbyte · 2023-11-08T23:37:54Z

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/6805316543
✅ Successfully published Java CDK version=0.4.7!

akashkulk · 2023-11-08T23:57:46Z

airbyte-cdk/java/airbyte-cdk/airbyte-commons-protocol/src/test/resources/WellKnownTypes.json

I think we should comment that this isn't currently used (if that hasn't already been said)

…postgres-source add tests. Add support for arrays of timestamps refactor more fixes Automated Commit - Formatting Changes fix add infinityHandling for CDC fix infinity for CDC Automated Commit - Formatting Changes simplify comparisons update CDK version revert change revert change change +/-infinity to (-)Infinity Update build.gradle Update version.properties revert last changes push CDK to 0.4.7

stephane-airbyte · 2023-11-09T18:23:54Z

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/6815901227
❌ Publish Java CDK version=0.4.7 failed!

octavia-squidington-iii added area/connectors Connector related issues connectors/source/postgres labels Oct 25, 2023

stephane-airbyte force-pushed the stephane-19952-postgres-infinity branch from e650134 to b7737ab Compare October 31, 2023 23:43

octavia-squidington-iii added the CDK Connector Development Kit label Oct 31, 2023

stephane-airbyte commented Oct 31, 2023

View reviewed changes

stephane-airbyte force-pushed the stephane-19952-postgres-infinity branch 2 times, most recently from 1e2de43 to 95d3cb7 Compare November 6, 2023 20:57

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Nov 7, 2023

stephane-airbyte force-pushed the stephane-19952-postgres-infinity branch from 553e00d to 4733fac Compare November 7, 2023 21:34

vercel bot deployed to Preview November 7, 2023 21:38 View deployment

stephane-airbyte commented Nov 7, 2023

View reviewed changes

stephane-airbyte marked this pull request as ready for review November 8, 2023 21:02

stephane-airbyte requested a review from a team as a code owner November 8, 2023 21:02

stephane-airbyte requested review from rodireich, edgao and akashkulk November 8, 2023 21:03

stephane-airbyte changed the title ~~don't missparse +/-infinity in postgres-source~~ 🐛 📝 don't missparse +/-infinity in postgres-source Nov 8, 2023

postamar reviewed Nov 8, 2023

View reviewed changes

postamar approved these changes Nov 8, 2023

View reviewed changes

vercel bot deployed to Preview November 8, 2023 21:57 View deployment

vercel bot deployed to Preview November 8, 2023 22:38 View deployment

stephane-airbyte force-pushed the stephane-19952-postgres-infinity branch from 0d1b3e6 to d7b8e89 Compare November 8, 2023 23:35

vercel bot deployed to Preview November 8, 2023 23:38 View deployment

stephane-airbyte enabled auto-merge (squash) November 8, 2023 23:53

akashkulk reviewed Nov 8, 2023

View reviewed changes

akashkulk approved these changes Nov 9, 2023

View reviewed changes

stephane-airbyte force-pushed the stephane-19952-postgres-infinity branch from 8e61a39 to 58e6f4c Compare November 9, 2023 01:17

stephane-airbyte and others added 3 commits November 9, 2023 10:22

Update build.gradle

c826f82

fix spotbugs

3460729

stephane-airbyte force-pushed the stephane-19952-postgres-infinity branch from 58e6f4c to 3460729 Compare November 9, 2023 18:22

stephane-airbyte merged commit 3f42418 into master Nov 9, 2023

stephane-airbyte deleted the stephane-19952-postgres-infinity branch November 9, 2023 18:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 📝 don't missparse +/-infinity in postgres-source #31856

🐛 📝 don't missparse +/-infinity in postgres-source #31856

stephane-airbyte commented Oct 25, 2023 •

edited

Loading

vercel bot commented Oct 25, 2023 •

edited

Loading

CLAassistant commented Oct 25, 2023 •

edited

Loading

github-actions bot commented Oct 25, 2023

stephane-airbyte Oct 31, 2023

stephane-airbyte Oct 31, 2023

stephane-airbyte Nov 7, 2023

stephane-airbyte Nov 7, 2023

github-actions bot commented Nov 7, 2023 •

edited

Loading

postamar Nov 8, 2023

stephane-airbyte commented Nov 8, 2023 •

edited by github-actions bot

Loading

stephane-airbyte commented Nov 8, 2023 •

edited by github-actions bot

Loading

stephane-airbyte commented Nov 8, 2023 •

edited by github-actions bot

Loading

akashkulk Nov 8, 2023

stephane-airbyte commented Nov 9, 2023 •

edited by github-actions bot

Loading

		@@ -9,7 +9,7 @@ plugins {
		airbyteJavaConnector {
		cdkVersionRequired = '0.4.1'

🐛 📝 don't missparse +/-infinity in postgres-source #31856

🐛 📝 don't missparse +/-infinity in postgres-source #31856

Conversation

stephane-airbyte commented Oct 25, 2023 • edited Loading

vercel bot commented Oct 25, 2023 • edited Loading

CLAassistant commented Oct 25, 2023 • edited Loading

github-actions bot commented Oct 25, 2023

Before Merging a Connector Pull Request

stephane-airbyte Oct 31, 2023

Choose a reason for hiding this comment

stephane-airbyte Oct 31, 2023

Choose a reason for hiding this comment

stephane-airbyte Nov 7, 2023

Choose a reason for hiding this comment

stephane-airbyte Nov 7, 2023

Choose a reason for hiding this comment

github-actions bot commented Nov 7, 2023 • edited Loading

Coverage report for source-postgres

postamar Nov 8, 2023

Choose a reason for hiding this comment

stephane-airbyte commented Nov 8, 2023 • edited by github-actions bot Loading

stephane-airbyte commented Nov 8, 2023 • edited by github-actions bot Loading

stephane-airbyte commented Nov 8, 2023 • edited by github-actions bot Loading

akashkulk Nov 8, 2023

Choose a reason for hiding this comment

stephane-airbyte commented Nov 9, 2023 • edited by github-actions bot Loading

stephane-airbyte commented Oct 25, 2023 •

edited

Loading

vercel bot commented Oct 25, 2023 •

edited

Loading

CLAassistant commented Oct 25, 2023 •

edited

Loading

github-actions bot commented Nov 7, 2023 •

edited

Loading

stephane-airbyte commented Nov 8, 2023 •

edited by github-actions bot

Loading

stephane-airbyte commented Nov 8, 2023 •

edited by github-actions bot

Loading

stephane-airbyte commented Nov 8, 2023 •

edited by github-actions bot

Loading

stephane-airbyte commented Nov 9, 2023 •

edited by github-actions bot

Loading