SlurmScheduler: Detect broken submission scripts for invalid account
#5850
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sphuber
force-pushed
the
feature/2955/slurm-invalid-account
branch
from
8004cdb to
dfc4da7
Compare
|
This PR is a follow-up to #5849 which should be reviewed and merged first. This PR then adds a commit on top which implements the new feature for the |
If an invalid combination of the `account` and `partition` options are
provided the submission will fail. Currently the scheduler plugin will
raise a generic exception causing the expontential backoff mechanism to
kick in. This is pointless, however, as the problem is not transient and
the submission will always fail, unless the scheduler script is updated,
which is not possible, since it breaks provenance.
The solution is to make use of the recently introduced feature for the
`_parse_submit_output` method to return an instance of `ExitCode` which
will trigger the engine to terminate the process. If an invalid account
or combination of account and partition are defined, the error will be:
Invalid account or account/partition combination specified
This error is printed to the `stderr`. When detected, the new exit code
`ERROR_SCHEDULER_INVALID_ACCOUNT` is returned.
The `ERROR_SCHEDULER_INVALID_ACCOUNT` exit code uses status `131`. The
idea is that the range 130 - 139 is reserved for errors that occur when
the job script submission fails. The status 130 is kept open for a more
general exit code that may be defined in the future.
sphuber
force-pushed
the
feature/2955/slurm-invalid-account
branch
from
dfc4da7 to
d0903ac
Compare
ltalirz
approved these changes
Dec 20, 2022
|
|
||
| if 'Invalid account' in stderr: | ||
| return CalcJob.exit_codes.ERROR_SCHEDULER_INVALID_ACCOUNT | ||
|
|
||
| raise SchedulerError(f'Error during submission, retval={retval}\nstdout={stdout}\nstderr={stderr}') |
There was a problem hiding this comment.
I see, so an exception-based mechanism is actually in place as well...
There was a problem hiding this comment.
Yes, but this gets trapped low-down in the expontential backoff mechanism. This was exactly the direct source of the linked issue. This exception was caught and the engine kept retrying until pausing the job. The point of the exit code is that we need to get out of that, and bubble all the way up.
| @@ -422,8 +422,14 @@ def _parse_submit_output(self, retval, stdout, stderr): | |||
|
|
|||
| Return a string with the JobID. | |||
| """ | |||
| from aiida.engine import CalcJob | |||
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #2955
If an invalid combination of the
accountandpartitionoptions areprovided the submission will fail. Currently the scheduler plugin will
raise a generic exception causing the expontential backoff mechanism to
kick in. This is pointless, however, as the problem is not transient and
the submission will always fail, unless the scheduler script is updated,
which is not possible, since it breaks provenance.
The solution is to make use of the recently introduced feature for the
_parse_submit_outputmethod to return an instance ofExitCodewhichwill trigger the engine to terminate the process. If an invalid account
or combination of account and partition are defined, the error will be:
This error is printed to the
stderr. When detected, the new exit codeERROR_SCHEDULER_INVALID_ACCOUNTis returned.The
ERROR_SCHEDULER_INVALID_ACCOUNTexit code uses status131. Theidea is that the range 130 - 139 is reserved for errors that occur when
the job script submission fails. The status 130 is kept open for a more
general exit code that may be defined in the future.