[Specs] Scheduler V2 Job Retention Cron.

[davinchia]

Tell us about the problem you're trying to solve

Scheduler V1 includes job history purging to minimise DB usage. Scheduler V2 is done with Temporal and there is currently no job sweeping available.

We need to figure out how to sweep jobs as Scheduler V2 gets stabilised and released to OSS. This is also important for Cloud to minimise cost.

Describe the solution you’d like

The eventual solution should:

• be compatible across Docker and Kube.
• be usable in both OSS and Cloud.
• allow for sane sweeping configuration probably centered around days.

Describe the alternative you’ve considered or used

A clear and concise description of any alternative solutions or features you've considered or are using today.

Additional context

Add any other context or screenshots about the feature request here.

Are you willing to submit a PR?

Remove this with your answer :-)

[davinchia]

fyi @benmoriceau and @cgardens

[benmoriceau]

@davinchia I tagged it as a spec issue and will try to make it to be in the next sprint. Do you know if there is some legal duration which we need to respect in order to define the priority of this ticket.

[jrhizor]

Looks like the likely solution here is to use Temporal (either as a scheduled Temporal cron that cleans up for all jobs or something that cleans up for a connection itself after each run).

[davinchia]

@benmoriceau no legal requirements for now. We've historically do so for operational/cost reasons.

@nataliekwong @bmcgilli are either of you aware of any legal requirements to keep data/are there product requirements around there that have surfaced since we last discussed this?

fyi @andyjih.

[nataliekwong]

I'm not aware of any.

[bmcgilli]

What's changing? Previously there was no concern from our legal council.

Also with Patsy starting in 2 weeks we can have her flag anything legal-related as she onboards

[benmoriceau]

It will be about fixing a regression. We had a sweeping of the old logs and DB in order to save cost which hasn't been added with the scheduler v2. We need to re-introduce it. I wonder if we promised to our clients that we will keeps logs up to X days/months

[davinchia]

@benmoriceau sounds like there are no new reqs. I think using the default of sweeping after 60 days is good enough for now.

[cgardens]

Context: For the cloud product we will want to hold logs for 1 year for security compliance reasons.

[davinchia]

This is out of date.