Failing submission to GPU partition in CI run

[laraPPr]

All rfm_job.sh fail to be submitted by reframe on the GPU partitions. with the following error.

sbatch: error: Invalid generic resource (gres) specification

However when I go to the Stage directory and run the command that failed the job gets submitted and runs. Their must be something that I'm missing that is set by ReFrame but not in the jobscript of launch command. That causes the job not to get submitted. I've already investigated it a little bit and I can't find what the difference might be.

ReFrame version: 4.7.3 & 4.7.2
Test-suite version: 0.5.1

[boegel]

@laraPPr Can you easily obtain the contents of the job script, and the exact sbatch command that is being used?

[laraPPr]

I did some further digging and the problem can't be with the job script itself, the test-suite or reframe

if reframe outside of the CI the error does not occur. And the test-suite runs fine.

So it is gonna be something that is picked up in the CI environment that causes the error. but I haven't tracked it down yet.

[laraPPr]

@laraPPr Can you easily obtain the contents of the job script, and the exact sbatch command that is being used?

Yes you can find the job script in the stage directory and the run command you can find in the logs but the problem is not with either of those.

[laraPPr]

Fixed with:

• Resolve failing to submit to hortense GPU partition #250

[laraPPr]

I tried to find what the environment is right before th jobs gets submitted by reframe by adding this to the following file: reframe/reframe/core/schedulers/slurm.py.

+ completed = _run_strict('env > /dodrio/scratch/projects/gadminforever/vsc46128/ReFrame_EESSI_CI/reframe_env.out`)

but than it was complaining that their is no such file or directory named '>'.

Maybe i should look at what ReFrame purges from the environment when 'purge_environment': True, is set.

[laraPPr]

I'm able to reproduce it without reframe by doing this

module --force purge
cd PATH/TO/STAGEDIR/hortense/gpu_rome_a100_40/default/EESSI_LAMMPS_lj_05f2d3fa
sbatch --export=NONE --get-user-env=60L --partition=gpu_rome_a100_40 rfm_job.sh

As with reframe it only causes trouble when trying to submit to the GPU and not the CPU partition

[laraPPr]

sbatch: error: Invalid generic resource (gres) specification
occurs when the following environment varible is not set:
export SLURM_CONF=/etc/slurm/slurm.conf_dodrio