Build middle layer in Cloud API for spec mutations

[subodh1810]

Part of airbytehq/airbyte-internal-issues#310
Tech spec : https://docs.google.com/document/d/18_Ezbl7Z1aSpHBdFFUy6BHMlDfYTktI4Yp8h2DbX4RE/edit#heading=h.fa72b39y2m99
As part of this issue we should introduce a middle layer in ConfigurationApiWrapped in cloud which would sit in between and handle the API requests from client.

The following APIs are the major ones to handle (non-exhaustive list, need to identify the API calls)

1. getDestinationDefinitionSpecification
2. createDestination
3. updateDestination
4. listDestinationsForWorkspace
5. getDestination
6. checkConnectionToDestinationForUpdate
7. getSourceDefinition
8. updateSourceDefinition
9. getSourceDefinitionSpecification
10. createSource
11. updateSource
12. listSourcesForWorkspace
13. getSource
14. checkConnectionToSourceForUpdate
15. ??

We would have a registry of handlers, each connector (source/destination) will have a handler and the handler would contain the mutation logic i.e. what needs to be added or removed from the spec/connector config before sending it back to client or saving it in the database.
When a request would be received in the ConfigurationAPI, the middle layer would find the handler of the connector from the registry and then invoke the concerned method.

For instance, a user requests for the setup form of Redshift destination connector, the middle layer would get the original spec from the DB and then pass it to the Redshift destination handler which will mutate it to add the info related to Airbyte staging and then send it back to the client. Similarly when a user fills in all the info and tries to create the destination, the middle layer would pass the config to Redshift handler which would identify if the user has selected Airbyte staging and if yes, then query the DB for the credentials of Airbyte staging bucket and then fill in the connector config and then save it in the database.

We would also need a framework to make sure that the mutation logic of the each connector handler is in sync in with the latest spec of the connector. This can be done by comparing the latest spec which can be found here against an expected spec and if the expected spec doesnt match the latest spec, the test should fail.

So as part of this issue

• Create the middle layer to handle the API calls, a registry of handlers, find and invoke the right handler for the API call
• Create the interface of the handler with all the right methods (to handle different API calls) so that we can introduce new handlers easily
• Create a framework to allow comparison of an expected spec against the latest spec so that its easier for developers to introduce new handlers. This should also enforce developers to add a expected spec whenever they introduce a new handler

[girarda]

This is almost done. PR has been reviewed. Only blocker is deploying to dev which I'd like to do with the shopify spec handler to confirm the feature works as expected.

The shopify spec handler is done, but deploying cloud with a custom version of OSS is convoluted. I'm hoping to be done with this by end of day

[girarda]

Going through one last round of QA/smoke check. will merge, release and close this by end of day

[subodh1810]

Done in the PR https://github.com/airbytehq/airbyte-cloud/pull/1091