Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Destination Postgres: CloudSQL allow verify-ca with client certificate and client key #22660

Open
bffrost opened this issue Feb 9, 2023 · 2 comments

Comments

@bffrost
Copy link
Contributor

bffrost commented Feb 9, 2023

Tell us about the problem you're trying to solve

I'm trying to connect from Airbyte (self-hosted, in GKE) to a CloudSQL Postgres instance. When I require SSL on my CloudSQL instance, the connection requires a server CA certificate, a Client Certificate and a Client Key. I have this connection working from other database clients, using sslmode=verify-ca.

Currently when the Airbyte Postgres Destination is in verify-ca mode, only the server CA certificate and Client key password can be supplied, so I cannot connect to my CloudSQL instance:
Screen Shot 2023-02-09 at 8 49 16 AM

When I switch to the verify-full SSL mode, it does allow inputting server CA cert, client cert, and client key, but the connection fails when trying to verify the CloudSQL instance's hostname (in my case an IP address) against the server CA certificate, which does not contain a hostname.

Describe the solution you’d like

I'd like the Postgres Destination SSL settings to behave the same way the Postgres Source settings currently do, allowing the client certificate and client key as optional configs when in verify-ca mode:
Screen Shot 2023-02-09 at 8 37 48 AM

Describe the alternative you’ve considered or used

As a workaround, I'm not using SSL because I'm able to connect from Airbyte to the CloudSQL instance over a private network.

Additional context

I noticed that this difference exists within the spec.json of the Postgres Source vs Postgres Destination.
Screen Shot 2023-02-09 at 9 07 57 AM

but I'm not sure if that is the only location that would need to be changed.
PostgresDestination.java uses this method to get SSL connection options:
io.airbyte.integrations.util.PostgresSslConnectionUtils.obtainConnectionOptions
while PostgresSource.java seems to use AbstractJdbcSource.parseSSLConfig.

Possibly related PRs & issues:
#14743
#17989

Are you willing to submit a PR?

With some guidance, I'm willing to submit a PR, but I'd be happy for someone more experienced to take this if it's either a quick fix or a more involved fix. I'm very new to Airbyte and I don't want to break this valuable connector!

@banditelol
Copy link

Hi @rodireich are you still working on it? if not I'd like to take this on.
As currently I need to use this in a use case in my company

@bleonard bleonard added the frozen Not being actively worked on label Mar 22, 2024
@hpadman
Copy link

hpadman commented Jul 17, 2024

This looks like very common requirement of using Postgres with SSL. We highly depend on postgres as destination for all of our sources and will be a big roadblock. Please prioritize this to highest possible level. We are using docker compose deployment for on-premise VM's.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants