Add UI, PostgreSQL, Minio and S3 relations

[kelkawi-a]

This PR does the following:

• Adds a relation to the PostgreSQL charm to act as Airbyte's DB.
• Adds a relation to the Minio charm as an option for object storage.
• Adds a relation to the S3 integrator charm as another option for object storage.
• Adds a relation to the Airbyte UI charm to enable communication of server status and service name between the two. The corresponding UI PR can be found here.
• Adds a configuration value for logs TTL, which gets added as a lifecycle policy for the selected object storage bucket.
• Adds logic for creating object storage buckets on Minio/S3 storage on relation creation or on config change.
• Refactors charm code.

Note: I've tested the functionality of adding source and destination connectors, and running a sync job. The only shortcoming thus far is that I am not able to get the Airbyte worker to store the logs in object storage (the bucket creation at the charm level works, Airbyte is just not communicating with the object storage properly to upload the logs). I am currently investigating this issue on different Airbyte slack channels and threads [1,2,3]. I have also tested the S3 integration with RadosGW, but Airbyte seems to have hardcoded this to AWS S3 storage, not just any S3-compatible storage. Regardless, this PR can be considered ready for review, with minor fixes to come potentially on some environment variables later.

[mertalpt]

Before I review the PR, I noticed something:

I have also tested the S3 integration with RadosGW, but Airbyte seems to have hardcoded this to AWS S3 storage, not just any S3-compatible storage.

@kelkawi-a How does minio work in the current Airbyte deployment? Does it use a different storage class?

[mertalpt]

LGTM overall but not approving until the "issue" thread is resolved.

[kelkawi-a]

Before I review the PR, I noticed something:

I have also tested the S3 integration with RadosGW, but Airbyte seems to have hardcoded this to AWS S3 storage, not just any S3-compatible storage.

@kelkawi-a How does minio work in the current Airbyte deployment? Does it use a different storage class?

Minio on k8s stores on persistent storage.

[gtato]

left some minor comments and questions
it's nice to see this taking shape
great job!

[gtato]

so if you don't delete the peer from meta, then what.. get_interfaces fails?
is there a way to get a specific interface... since you don't care about the others?

[kelkawi-a]

Yes, get_interfaces iterates through all the relations and searches for them under the list of relations defined in requires and provides, but the peer relation is a special case which does not exist under them, so the method fails. Anyway, we do not care for searching for the peer relation, hence why it can be removed.

[gtato]

are these 2 files s3 and s3_helpers coming from somewhere? another charm?

[kelkawi-a]

These are defined for this charm, but there are occurences of the helpers in other charms that relate to the s3 integrator as well (e.g. the create_bucket_if_not_exists method)