low-code connectors: fix parse and format methods #15326
Conversation
| if self._datetime_format == "timestamp": | ||
| return dt.timestamp() | ||
| if self._datetime_format == TIMESTAMP_FORMAT: | ||
| return int(dt.timestamp()) |
There was a problem hiding this comment.
cast to int because dt.timestamp() returns a flow
| return datetime.datetime.fromtimestamp(int(date)).replace(tzinfo=self._timezone) | ||
| return date | ||
| elif isinstance(date, str): | ||
| return datetime.datetime.strptime(str(date), self._datetime_format).replace(tzinfo=self._timezone) |
There was a problem hiding this comment.
better to use the specified datetime_format instead of relying on dateutil.parser's best guess
girarda
changed the title
| else: | ||
| return dateutil.parser.parse(date).replace(tzinfo=self._timezone) | ||
| elif isinstance(date, int): | ||
| def parse_date(self, date: Union[str, datetime.datetime]) -> datetime.datetime: |
There was a problem hiding this comment.
can't place the comment in the right spot so adding it here. Can you clarify in the doc string for datetime_format what the reference for formats is? eg add a link to it
There was a problem hiding this comment.
added a link in the docstring and in MinMaxDatetime's https://github.com/airbytehq/airbyte/pull/15326/files#diff-3a6ff2f6eca71b4e166b1c8c852becb143ad4e079f5308967ff52828e747f035R34
@sherifnada the contract is now stated in the docstring. I removed the "timestamp" magic keyword because "%s" is equivalent |
|
/publish-cdk dry-run=false
|
|
/publish-cdk dry-run=false
|
What
Fixes 2 bugs in the datetime stream slicer
self._datetime_formatHow
_format_datetimeto an int if the format is "timestamp"strptimeandself._datetime_formatRecommended reading order
airbyte-cdk/python/airbyte_cdk/sources/declarative/stream_slicers/datetime_stream_slicer.pyairbyte-cdk/python/unit_tests/sources/declarative/stream_slicers/test_datetime_stream_slicer.py🚨 User Impact 🚨
Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampledocs/integrations/README.mdairbyte-integrations/builds.mdAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing/publishcommand described hereUpdating a connector
Community member or Airbyter
airbyte_secret./gradlew :airbyte-integrations:connectors:<name>:integrationTest.README.mdbootstrap.md. See description and examplesdocs/integrations/<source or destination>/<name>.mdincluding changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>command is passing/publishcommand described hereConnector Generator
-scaffoldin their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplatesthen checking in your changesTests
Unit
Put your unit tests output here.
Integration
Put your integration tests output here.
Acceptance
Put your acceptance tests output here.