Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a self-healing scheduler services (x2) #15218

Closed
evantahler opened this issue Aug 2, 2022 · 5 comments
Closed

Create a self-healing scheduler services (x2) #15218

evantahler opened this issue Aug 2, 2022 · 5 comments
Assignees

Comments

@evantahler
Copy link
Contributor

evantahler commented Aug 2, 2022

Relies on Micronaut

There will likely be 2 services for each above links. Keeping separate services per job means that a failure is isolated and the other services can continue to run. There might be a "talks to our DB & services" service, and then a "talks to temporal" service.

In a future story, we will emit metrics when these jobs run.

Be sure to add an E2E test for the temporal schedule healer

@evantahler
Copy link
Contributor Author

Note from sprint planning for @jdpgrailsdev: As the temporal work wants a scheduler, this is likely the first Micronaut story to prioritize

@evantahler evantahler changed the title Create a self-healing scheduler service Create a self-healing scheduler services (x2) Aug 16, 2022
@davinchia
Copy link
Contributor

davinchia commented Aug 16, 2022

Can we also emit a warning/metric when this happens? In theory we should never run into self-determinism errors on Temporal. It indicates a big error, and I think it's worth shouting out to us.

@evantahler
Copy link
Contributor Author

Metrics will be emitted here #15699

@evantahler
Copy link
Contributor Author

Once this is done, we can then merge #14758 as well

@gosusnp
Copy link
Contributor

gosusnp commented Sep 15, 2022

Depending on how we handle workflow versioning once we have this change ready, we may want to consider #16762

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants