Implement proactive error tracking for connectors

[sherifnada]

Tell us about the problem you're trying to solve

as an airbyte maintainer I'd love to get proactive error reporting when connectors break while a user is running Airbyte.

Describe the solution you’d like

I would like to receive an error email/slack message/github issue when a user trying a connector out encounters an exception.

Describe the alternative you’ve considered or used

We could use Sentry for this potentially

Additional context

Thanks to @eugene-kulak for the idea

┆Issue is synchronized with this Asana task by Unito

[cgardens]

Neat. This project might be sneaky hard to do. Worth speccing out exactly what info we want first. Then figure out how we will pull it.

I'm saying this, because I think many of the errors we want to see are on the backend, so I'm not sure that the sentry FE solution will be the right option. but admittedly i don't know the sentry offering all that well.

[eugene-kulak]

@cgardens historically sentry was a system to track and report errors on BE systems (Django, etc), so there should be no issues to sent error from BE, or am I missing something.
btw, this is another good reason to use a standard python logger because Sentry integrates with it natively.

[cgardens]

nope. i've just only used it with FE so i don't know what the BE offer looks like or how it will integrate will all of our stuff running across multiple docker containers.

btw, this is another good reason to use a standard python logger because Sentry integrates with it natively.
hahaha. finally!!!!

[marcosmarxm]

@sherifnada can i take this?

[sherifnada]

@marcosmarxm go for it!

[cgardens]

I'm just going to make the observation that this issue has gotten kicked down the lane a lot. My hunch is that because it's just not super clear what we're trying to implement here. Might be good to have someone (or a brainstorm) to figure out what we want to do here. Otherwise it's a sneaky hard and large issue. Gonna kick it out of the milestones until now.

[keu]

Having a cloud solution makes this much more logical to implement.
The final point that I see is that:
The issue got detected and fixed before cloud users even notice it.

Draft of the plan for MVP:

• investigate env configuration of Sentry (make it optional)
• investigate what Sentry provides for obfuscation and dealing with sensitive data
• make test integration for one of the connectors and deploy it in sandbox cloud env (if any exist).

[sherifnada]

@keu let's start breaking this down and making progress on it: can you follow the [EPIC] pattern to break it down into several tickets on github? I've moved it into our next sprint to investigate.

[evantahler]

#12423 and our analytics around it probably satisfies this!

[evantahler]

Duplicate of #12119