ACCESS-OM2 driver insists on their being an atmosphere subdirectory when restarting

[aidanheerdegen]

When trying to use some older restarts from a COSIMA model (/g/data/ik11/outputs/access-om2-025/025deg_jra55_ryf9091_gadi/restart250) payu threw an error complaining about the absence of an atmosphere subdirectory in the restart directory.

$ payu setup
laboratory path:  /scratch/tm70/aph502/access-om2
binary path:  /scratch/tm70/aph502/access-om2/bin
input path:  /scratch/tm70/aph502/access-om2/input
work path:  /scratch/tm70/aph502/access-om2/work
archive path:  /scratch/tm70/aph502/access-om2/archive                                                                                             Found experiment archive: /scratch/tm70/aph502/access-om2/archive/025deg_jra55_ryf_backport-expt-7440fc32                                          payu: Found modules in /opt/Modules/v4.3.0                                                                                                         Loading access-om2/2023.11.23                                                                                                                     
  Loading requirement: oasis3-mct/2023.11.09_2023.11.23 libaccessom2/2023.10.26_2023.11.23 cice5/2023.10.19_2023.11.23 mom5/2023.11.09_2023.11.23
Loading input manifest: manifests/input.yaml        
Loading restart manifest: manifests/restart.yaml                                                                                                  
Loading exe manifest: manifests/exe.yaml               
Setting up atmosphere
No prior restart files found: [Errno 2] No such file or directory: '/g/data/ik11/outputs/access-om2-025/025deg_jra55_ryf9091_gadi/restart250/atmosphere'                                                                                                                                             
Setting up ocean                                      
Setting up ice                                         
Setting up access-om2                          
Checking exe, input and restart manifests          
Writing manifests/input.yaml                         
Writing manifests/restart.yaml                           
$ ls /g/data/ik11/outputs/access-om2-025/025deg_jra55_ryf9091_gadi/restart250/atmosphere
ls: cannot access '/g/data/ik11/outputs/access-om2-025/025deg_jra55_ryf9091_gadi/restart250/atmosphere': No such file or directory

It seems ACCESS-OM2 is now producing an empty atmosphere restart directory and then insisting on that existing in older restart directories.

payu version info:

$ which payu
/g/data/vk83/prerelease/apps/payu/dev/bin/payu
$ payu --version
payu 1.1.5+37.g1186c46

I used this released 0.25 configuration

https://github.com/ACCESS-NRI/access-om2-configs/tree/release-025deg_jra55_ryf-2.0

[aidanheerdegen]

Note that this doesn't stop payu running, but it is confusing for users if payu reports an error that should be ignored.

[jo-basevi]

I think it's being checked in the general payu model driver during setup where it creates the restart manifest and sets up symlinks to the work directory:

payu/payu/models/model.py

Lines 262 to 273 in 1186c46

	# Add restart files from prior run to restart manifest
	if self.prior_restart_path:
	restart_files = self.get_prior_restart_files()
	for f_name in restart_files:
	f_orig = os.path.join(self.prior_restart_path, f_name)
	f_link = os.path.join(self.work_init_path_local, f_name)
	self.expt.manifest.add_filepath(
	'restart',
	f_link,
	f_orig,
	self.copy_restarts
	)

payu/payu/models/model.py

Lines 176 to 184 in 1186c46

	def get_prior_restart_files(self):
	try:
	respath = self.prior_restart_path
	return [f for f in os.listdir(respath)
	if os.path.isfile(os.path.join(respath, f))]
	except Exception as e:
	print("No prior restart files found: {error}".format(error=str(e)))
	return []

Should the error print statement not run for just the yatm model? It looks like the yatm model creates an empty directory during the archive step.

[aidanheerdegen]

It looks like the yatm model creates an empty directory during the archive step.

You're right

https://github.com/payu-org/payu/blame/master/payu/models/yatm.py#L41

I can only assume that is because get_prior_restart_files expected there to be one.

I think this solution is to remove that from the yatm driver and add a no-op get_prior_restart_files method to the yatm driver class that just returns an empty array.

[jo-basevi]

Yeah I think that'll work :)

[jo-basevi]

Do you know if the yatm model never uses prior restart files? Just double checking whether it is safe to over-ride the method and return [], or to have mostly the same method apart from the error log message?

[aidanheerdegen]

yam reads from data files, so should never require prior restarts. So fine to assume so. In some ways overriding is better because it is explicit that there are no restarts.