OAuth Flow: new authSpecifications supporting field references from connector config as dependency

[bazarnov]

Tell us about the problem you're trying to solve

We need a way to pass the user input config parameters into the OAuthFlow 2.0.

Use-case:
We have Source Shopify with implemented Oauth2.0 connection method.
The url base for the consent and the access token url should include the Shop Name which user can input on the stage of connection setup (see screenshot, for shop).
So the final link for the consent screen should look like this:

FORMAT: "https://{shop}.myshopify.com/admin/oauth/access_token", where {shop} is the Shop Name of the user's Shop.

Describe the solution you’d like

1. Probably to extend the OAuth Specification to have something like:
   "oauthFlowConfigParameters" or "oauthFlowCustomParameters" and pass the config-required fields there, and as the result to the <connector>OauthFlow.java
2. The button of Authenticate your Shopify account in this case should be active only after the Shop field is populated.

Describe the alternative you’ve considered or used

Other way to pass the config-required parameters into the OAuthFlow.

Additional context

[bazarnov]

@sherifnada @ChristopheDuong @jrhizor

The list of blocked source-connectors on this:

• Shopify
• Zendesk Support
• Zendesk Sunshine
• Zendesk Talk
• WooCommerce
• Marketo
• Mixpanel
• Freshdesk
• GitLab
• Cart.com
• BigCommerce
• Jira
• Okta
• Posthog
• Prestashop
• Bing-Ads

In order to OAuth 2.0 implementation work properly for these connectors, there should be a mechanism to pass the config-required fields into each *OauthFlow.java class and use these fields to construct AUTH_CODE / ACCESS_TOKEN urls dynamically, based on user's input.

[sherifnada]

great work @bazarnov , thanks for tracking this down. Do you want to make a proposal/RFC for how this should be supported in the protocol?

[bazarnov]

@sherifnada
The simplest way to have all the necessary parameters to construct the urls is to pass them though UI > any mapping object , which is passing along with source_definition, workspace_id. We plan to pass all the fields from the spec.json, that are required and not airbyte_secret.
@avida got the idea of how to implement this, without changes to airbyte-protocol.

Another option is to pass the necessary fields using the AuthSpecification section by setting them inside of authFlowAdditionalParameters. But we believe this change could be implemented without modifications to airbyte-protocol.

[sherifnada]

I would prefer that we don’t hardcode special rules in the UI/server for each connector. As it is, hardcoding the oauth flow in the server couples the connector lifecycle with the server lifecycle. Because their versions are released totally separately, it’s not a good experience for the user.

I propose that we add the following struct to the oauth specification:

// represents a list of pointers to the fields required to launch the oauth flow.
// For example, to authenticate a shopify user, we need to embed the shop ID in the consentgrant URL,
// shop ID is a separate parameter in the spec
// Therefore this array would contain a list of pointers (a pointer is a string array denoting a path in the config)
// pointing to the required fields
oauth_flow_required_fields: Array<Array<String>>

In the shopify case this would be something like:

---- spec.json

{
"properties": {
  shop_id: {type: string},
  refresh_token: {type: string}
  client_id: // etc...
 ,
 authSpecification: {
  type: oauth2
  oauth2Specification:
   {
    oauth_flow_input_fields: [["shop_id"]]
    // remaining fields
    }
 }
}

The UI then sends these parameters alongside the get_consent_url and the complete_oauth_flow calls

[avida]

@sherifnada It makes sense. Lets call it oauthFlowInputFields similar to other fields.