state: avoid infinite recursion in JobStore

[radeksimko]

Fixes #1065

I couldn't come up with a more light-weight test which would reproduce the bug. As mentioned in the comment, it (scheduling over 1 million jobs) takes about 3-4 minutes to finish on M1 Pro and consumes decent chunk of almost all cores + ~5-6GB of memory at peak.

The Windows GHA environment runs out of memory before the test can even finish. 🙊

There are a few options to consider:

1. remove the test entirely (some risk that we re-introduce the bug)
2. gate the test on ENV variable, such as LONG_TESTS=1
3. gate the test on build tags, such as -tags=longtests

We could also consider not running the test on non-empty CI ENV variable (which most CI systems set, incl. GitHub Actions), but I'd argue that it's probably not as valuable to run that test locally for most of the time either.

[dbanck]

We (especially you) did quite some work to reduce CI run time over the last couple of weeks. Therefore, I would instead not run this as part of our usual CI pipeline.

What do you think about running this as a nightly job, similar to the benchmarks? That would still ensure that we catch this bug before a release.

[radeksimko]

What do you think about running this as a nightly job, similar to the benchmarks?

We could, but that would have to be limited to linux and/or macos as it's just impossible to run that in the Windows GHA environment.

[radeksimko]

@dbanck I have gated the test on longtest build tag and added a nightly workflow, as suggested, PTAL

[dbanck]

LGTM! I would just update the name

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.