| description |
|---|
Be sure to not miss out on new features and improvements! |
This is the changelog for Airbyte Platform. For our connector changelog, please visit our Connector Changelog page.
- Add an endpoint that specify that the feedback have been given after the first sync.
- No major changes to Airbyte Core.
- No major changes to Airbyte Core.
- Improve error handling when additional sources/destinations cannot be read.
- Implement connector config dependency for OAuth consent URL.
- Treat oauthFlowInitParameters just as hidden instead of getting rid of them.
- Stop using gentle close with heartbeat.
- Add to the new connection flow form with an existing source and destination dropdown.
- Implement protocol change for OAuth outputs.
- Enhance API for use by cloud to provide per-connector billing info.
- This is a MAJOR version update. You need to update to this version before updating to any version newer than
0.32.0
- No major changes to Airbyte Core.
- We migrated our secret management to Google Secret Manager, allowing us to scale how many connectors we support.
- No major changes to Airbyte Core.
- No major changes to Airbyte Core.
- No major changes to Airbyte Core.
- No major changes to Airbyte Core.
- No major changes to Airbyte Core.
- Airbyte Server no longer crashes from having too many open files.
- No major changes to Airbyte Core.
- No major changes to Airbyte Core.
- Incremental normalization is live. Basic normalization no longer runs on already normalized data, making it way faster and cheaper.
- No major changes to Airbyte Core.
- We now support experimental deployment of Airbyte on Macbooks with M1 chips!
{% hint style="info" %}
This interim patch period mostly contained stability changes for Airbyte Cloud, so we skipped from 0.30.16 to 0.30.22.
{% endhint %}
- On Kubernetes deployments, you can now configure the Airbyte Worker Pod's image pull policy.
{% hint style="info" %}
This interim patch period mostly contained stability changes for Airbyte Cloud, so we skipped from 0.30.2 to 0.30.16.
{% endhint %}
- Fixed a bug that would fail Airbyte upgrades for deployments with sync notifications.
- We now have integration tests for SSH.
- You can now deploy Airbyte on Kubernetes with a Helm Chart!
- Fixes a breaking bug that prevents Airbyte upgrading from older versions.
- Building images is now optional in the CI build.
- You can now properly cancel deployments when deploying on K8s.
- You can now send notifications via webhook for successes and failures on Airbyte syncs.
- Scheduling jobs and worker jobs are now separated, allowing for workers to be scaled horizontally.
- Fixed a bug that made it possible for connector definitions to be duplicated, violating uniqueness.
- Nothing of note.
- The scheduler now waits for the server before it creates any databases.
- You can now apply tolerations for Airbyte Pods on K8s deployments.
- Syncs now have a
max_sync_timeoutthat times them out after 3 days. - Fixed Kube deploys when logging with Minio.
- Nothing of note.
- Migration of Python connector template images to Alpine Docker images to reduce size.
- Nothing of note.
- Nothing of note.
- Re-release: Fixed errant ENV variable in
0.29.6
- Connector pods no longer fail with edge case names for the associated Docker images.
- Nothing of note.
- Introduced implementation for date-time support in normalization.
- Importing configuration no longer removes available but unused connectors.
- Fixed nil pointer exception in version migrations.
- When migrating, types represented in the config archive need to be a subset of the types declared in the schema.
- Deprecated
DEFAULT_WORKSPACE_ID; default workspace no longer exists by default.
- Backend now handles workspaceId for WebBackend operations.
- K8s: Overly-sensitive logs are now silenced.
- Acceptance test dependencies fixed.
- Fixed unreliable logging on Kubernetes deployments.
- Introduced pre-commit to auto-format files on commits.
- Config persistence is now migrated to the internal Airbyte database.
- Source connector ports now properly close when deployed on Kubernetes.
- Missing dependencies added that allow acceptance tests to run.
- Fixed some minor API spec errors.
- GCP environment variable is now stubbed out to prevent noisy and harmless errors.
- New API endpoint: List workspaces
- K8s: Server doesn't start up before Temporal is ready to operate now.
- Silent source failures caused by last patch fixed to throw exceptions.
- Airbyte now automatically upgrades on server startup!
- Airbyte will check whether your
.envAirbyte version is compatible with the Airbyte version in the database and upgrade accordingly.
- Airbyte will check whether your
- When running Airbyte on K8s logs will automatically be stored in a Minio bucket unless configured otherwise.
- CDC for MySQL now handles decimal types correctly.
- First-Class Kubernetes support!
- Custom dbt transformations!
- You can now configure your destination namespace at the table level when setting up a connection!
- Migrate basic normalization settings to the sync operations.
- Bugfix: Handle TINYINT(1) and BOOLEAN correctly and fix target file comparison for MySQL CDC.
- Bugfix: Updating the source/destination name in the UI now works as intended.
- Bugfix: Ensure that logs from threads created by replication workers are added to the log file.
- Remove hash from table names when it's not necessary for normalization outputs.
- PythonCDK: change minimum Python version to 3.7.0
- Minor fixes to documentation
- Reliability updates in preparation for custom transformations
- Limit Docker log size to 500 MB (#3702)
- Fix for file names being too long in Windows deployments (#3625)
- Allow users to access the API and WebApp from the same port (#3603)
- Checkpointing for incremental syncs that will now continue where they left off even if they fail! (#3290)
- Avoid dbt runtime exception "maximum recursion depth exceeded" in ephemeral materialization (#3470)
- Documentation to deploy locally on Windows is now available (#3425)
- Connector icons are now displayed in the UI
- Restart core containers if they fail automatically (#3423)
- Progress on supporting custom transformation using dbt. More updates on this soon!
- Bump K8s deployment version to latest stable version, thanks to Coetzee van Staden
- Added tutorial to deploy Airbyte on Azure VM (#3171), thanks to geekwhocodes
- Progress on checkpointing to support rate limits better
- Upgrade normalization to use dbt from docker images (#3186)
- Split replication and normalization into separate temporal activities (#3136)
- Fix normalization Nesting bug (#3110)
- Replace timeout for sources (#3031)
- Fix UI issue where tables with the same name are selected together (#3032)
- Fix feed handling when feeds are unavailable (#2964)
- Export whitelisted tables (#3055)
- Create a contributor bootstrap script (#3028) (#3054), thanks to nclsbayona
- Namespace support: supported source-destination pairs will now sync data into the same namespace as the source (#2862)
- Add “Refresh Schema” button (#2943)
- In the Settings, you can now add a webhook to get notified when a sync fails
- Add destinationSyncModes to connection form
- Add tooltips for connection status icons
- Change Data Capture (CDC) is now supported for Postgres, thanks to @jrhizor and @cgardens. We will now expand it to MySQL and MSSQL in the coming weeks.
- When displaying the schema for a source, you can now search for table names, thanks to @jamakase
- Better feedback UX when manually triggering a sync with “Sync now”
- New Connections page where you can see the list of all your connections and their statuses.
- New Settings page to update your preferences.
- Bugfix where very large schemas caused schema discovery to fail.
- Surface the health of each connection so that a user can spot any problems at a glance.
- Added support for deduplicating records in the destination using a primary key using incremental dedupe -
- A source’s extraction mode (incremental, full refresh) is now decoupled from the destination’s write mode -- so you can repeatedly append full refreshes to get repeated snapshots of data in your source.
- New Upgrade all button in Admin to upgrade all your connectors at once
- New Cancel job button in Connections Status page when a sync job is running, so you can stop never-ending processes.
- Improved the speed of get spec, check connection, and discover schema by migrating to the Temporal workflow engine.
- Exposed cancellation for sync jobs in the API (will be exposed in the UI in the next week!).
- Bug fix: Fix issue where migration app was OOMing.
- Creating and deleting multiple workspaces is now supported via the API. Thanks to @Samuel Gordalina for contributing this feature!
- Normalization now supports numeric types with precision greater than 32 bits
- Normalization now supports union data types
- Support longform text inputs in the UI for cases where you need to preserve formatting on connector inputs like .pem keys
- Expose the latest available connector versions in the API
- Airflow: published a new tutorial for how to use the Airbyte operator. Thanks @Marcos Marx for writing the tutorial!
- Connector Contributions: All connectors now describe how to contribute to them without having to touch Airbyte’s monorepo build system -- just work on the connector in your favorite dev setup!
- Integration with Airflow is here. Thanks to @Marcos Marx, you can now run Airbyte jobs from Airflow directly. A tutorial is on the way and should be coming this week!
- Add a prefix for tables, so that tables with the same name don't clobber each other in the destination
-
We made some progress to address nested tables in our normalization.
Previously, basic normalization would output nested tables as-is and append a number for duplicate tables. For example, Stripe’s nested address fields go from:
Address address_1To
Charges_source_owner_755_address customers_shipping_c70_addressAfter the change, the parent tables are combined with the name of the nested table to show where the nested table originated. This is a breaking change for the consumers of nested tables. Consumers will need to update to point at the new tables.
- We now handle nested tables with the normalization steps. Check out the video below to see how it works.
{% embed url="https://youtu.be/I4fngMnkJzY" caption="" %}
- Front-end changes:
- Display Airbyte's version number
- Describe schemas using JsonSchema
- Better feedback on buttons
Beta launch - 0.13 - Released 02/02/2021
- Add connector build status dashboard
- Support Schema Changes in Sources
- Support Import / Export of Airbyte Data in the Admin section of the UI
- Bug fixes:
- If Airbyte is closed during a sync the running job is not marked as failed
- Airbyte should fail when deployment version doesn't match data version
- Upgrade Airbyte Version without losing existing configuration / data
0.12-alpha - Released 01/20/2021
- Ability to skip onboarding
- Miscellaneous bug fixes:
- A long discovery request causes a timeout in the UI type/bug
- Out of Memory when replicating large table from MySQL
- Increase timeout for long running catalog discovery operations from 3 minutes to 30 minutes to avoid prematurely failing long-running operations
- Writing boolean columns to Redshift destination now works correctly
0.11.0-alpha - Delivered 01/14/2021
- Allow skipping the onboarding flow in the UI
- Add the ability to reset a connection's schema when the underlying data source schema changes
- Fix UI race condition which showed config for the wrong connector when rapidly choosing between different connector
- Fix a bug in MSSQL and Redshift source connectors where custom SQL types weren't being handled correctly. Pull request
- Support incremental sync for Salesforce, Slack, and Braintree sources
- Gracefully handle invalid nuemric values (e.g NaN or Infinity) in MySQL, MSSQL, and Postgtres DB sources
- Fix flashing red sources/destinations fields after success submit
- Fix a bug which caused getting a connector's specification to hang indefinitely if the connector docker image failed to download
- Tempo
- Appstore
0.10.0 - delivered on 01/04/2021
- You can now deploy Airbyte on Kuberbetes **(alpha version)
- Support incremental sync for Mixpanel and HubSpot sources
- Fixes on connectors:
- Fixed a bug in the GitHub connector where the connector didn’t verify the provided API token was granted the correct permissions
- Fixed a bug in the Google Sheets connector where rate limits were not always respected
- Alpha version of Facebook marketing API v9. This connector is a native Airbyte connector (current is Singer based).
- New source: Plaid (contributed by @tgiardina - thanks Thomas!)
0.9.0 - delivered on 12/23/2020
- New chat app from the web app so you can directly chat with the team for any issues you run into
- Debugging has been made easier in the UI, with checks, discover logs, and sync download logs
- Support of Kubernetes in local. GKE will come at the next release.
- New source: Looker **
0.8.0 - delivered on 12/17/2020
- Incremental - Append"
- We now allow sources to replicate only new or modified data. This enables to avoid re-fetching data that you have already replicated from a source.
- The delta from a sync will be appended to the existing data in the data warehouse.
- Here are all the details of this feature.
- It has been released for 15 connectors, including Postgres, MySQL, Intercom, Zendesk, Stripe, Twilio, Marketo, Shopify, GitHub, and all the destination connectors. We will expand it to all the connectors in the next couple of weeks.
- Other features:
- Improve interface for writing python sources (should make writing new python sources easier and clearer).
- Add support for running Standard Source Tests with files (making them easy to run for any language a source is written in)
- Add ability to reset data for a connection.
- Bug fixes:
- Update version of test containers we use to avoid pull issues while running tests.
- Fix issue where jobs were not sorted by created at in connection detail view.
- New sources: Intercom, Mixpanel, Jira Cloud, Zoom, Drift, Microsoft Teams
0.7.0 - delivered on 12/07/2020
- New destination: our own Redshift warehouse connector. You can also use this connector for Panoply.
- New sources: 8 additional source connectors including Recurly, Twilio, Freshdesk. Greenhouse, Redshift (source), Braintree, Slack, Zendesk Support
- Bug fixes
0.6.0 - delivered on 11/23/2020
- Support multiple destinations
- New source: Sendgrid
- Support basic normalization
- Bug fixes
0.5.0 - delivered on 11/18/2020
- New sources: 10 additional source connectors, including Files (CSV, HTML, JSON...), Shopify, MSSQL, Mailchimp
0.4.0 - delivered on 11/04/2020
Here is what we are working on right now:
- New destination: our own Snowflake warehouse connector
- New sources: Facebook Ads, Google Ads.
0.3.0 - delivered on 10/30/2020
- New sources: Salesforce, GitHub, Google Sheets, Google Analytics, HubSpot, Rest APIs, and MySQL
- Integration test suite for sources
- Improve build speed
0.2.0 - delivered on 10/21/2020
- a new Admin section to enable users to add their own connectors, in addition to upgrading the ones they currently use
- improve the developer experience (DX) for contributing new connectors with additional documentation and a connector protocol
- our own BigQuery warehouse connector
- our own Postgres warehouse connector
- simplify the process of supporting new Singer taps, ideally make it a 1-day process
0.1.0 - delivered on 09/23/2020
This is our very first release after 2 months of work.
- New sources: Stripe, Postgres
- New destinations: BigQuery, Postgres
- Only one destination: we only support one destination in that 1st release, but you will soon be able to add as many as you need.
- Logs & monitoring: you can now see your detailed logs
- Scheduler: you now have 10 different frequency options for your recurring syncs
- Deployment: you can now deploy Airbyte via a simple Docker image, or directly on AWS and GCP
- New website: this is the day we launch our website - airbyte.io. Let us know what you think
- New documentation: this is the 1st day for our documentation too
- New blog: we published a few articles on our startup journey, but also about our vision to making data integrations a commodity.
Stay tuned, we will have new sources and destinations very soon! Don't hesitate to subscribe to our newsletter to receive our product updates and community news.