cleanup TaskType names

[belforte]

in the context of Modernize HTC calls
see #8337 (comment)

• rename TaskType ad to CRAB_DAGType
• rename ROOT type to MAIN. So that dagman jobs will have CRAB_DagType in ["MAIN", "PROCESSING", "TAIL"]. "normal" tasks will only have "MAIN". "automatic" will use "MAIN" for running probes, followed by "PROCESSING" and up to 3 "TAIL".
  But I am still not happy here. ROOT is more descriptive for a DAG which can be either the full thing standalone or the probe step for automatic splitting. Maybe "BASE" ?
  I will start with renaming the classAd, at least it will make it easier later on to identify where it is used and possibly change the value.

BEWARE these changes will not be backward compatible ! i.e. crab kill and crab resubmit will not work on tasks which have TaskType but not CRAB_DAGType.

• add a backward-compatibility layer (to be remove after 2 months in production)

Note: ROOT is assigned in initial submission (so it i the same for everybody) in DagamSubmitter. Later on PreDag uses the stage name when it submits the various subdags. Stage name is set in DagmanCreator and used in PreDag and PreJob

List of possible values. Note that CRAB_DAGType is the classAd of a job !! Simply it takes a non trivial value when the job runs in the Scheduler universe to execut a DagMan

kind of job	universe	CRAB_DAGType	stage name
main Dagman w/o automatic splitting	scheduler (7)	BASE	conventional
main Dagman with automatic splitting	scheduler (7)	BASE	probe
grid job	vanilla (5)	Job	not used
task_process	local (12)	undefined	not used
processing subDag	scheduler (7)	PROCESSING	processing
tail subDag	scheduler (7)	TAIL	tail

Possible alternative names for ROOT: MAIN , BASE

TaskType ad is defined in

CRABServer/src/python/TaskWorker/Actions/DagmanSubmitter.py

Line 529 in 3c826e3

jobJDL["+TaskType"] = classad.quote("ROOT") # we want the ad value to be "ROOT", not ROOT

and

CRABServer/src/python/TaskWorker/Actions/DagmanCreator.py

Line 104 in 43208f8

+TaskType = "Job"

and in case of automatic splitting in

CRABServer/src/python/TaskWorker/Actions/PreDAG.py

Line 338 in 3c826e3

'-append', '+TaskType = "{0}"'.format(stage.upper()), subdag])

the stage in PreDAG is passed as argument when PreDAG is called during Dag execution:

CRABServer/src/python/TaskWorker/Actions/DagmanCreator.py

Line 56 in 3c826e3

SCRIPT DEFER 4 300 PRE Job{count}SubJobs dag_bootstrap.sh PREDAG {stage} {completion} {count}

and the code which managed the various DAGs creation is the somehow cryptic

CRABServer/src/python/TaskWorker/Actions/DagmanCreator.py

Lines 998 to 1024 in 3c826e3

	## Write down the DAG as needed by DAGMan.
	restHostForSchedd = kwargs['task']['resthost']
	dag = DAG_HEADER.format(nodestate='.{0}'.format(parent) if parent else ('.0' if stage == 'processing' else ''),
	resthost=restHostForSchedd)
	if stage == 'probe':
	dagSpecs = dagSpecs[:getattr(self.config.TaskWorker, 'numAutomaticProbes', 5)]
	for dagSpec in dagSpecs:
	dag += DAG_FRAGMENT.format(**dagSpec)
	if stage in ('probe', 'processing'):
	# default for probe DAG: only one processing DAG after 100% of the probe jobs have completed
	subdagCompletions = [100]
	nextStage = {'probe': 'processing', 'processing': 'tail'}[stage]
	if stage == 'processing' and len(dagSpecs) > getattr(self.config.TaskWorker, 'minAutomaticTailSize', 100):
	subdagCompletions = getattr(self.config.TaskWorker, 'minAutomaticTailTriggers', [50, 80, 100])
	for n, percent in enumerate(subdagCompletions, 0 if stage == 'probe' else 1):
	subdagSpec = {
	'count': n,
	'stage': nextStage,
	'completion': (len(dagSpecs) * percent) // 100
	}
	dag += SUBDAG_FRAGMENT.format(**subdagSpec)
	subdag = "RunJobs{count}.subdag".format(**subdagSpec)
	with open(subdag, "w", encoding='utf-8') as fd:
	fd.write("")
	subdags.append(subdag)

TaskType ad is also used in

CRABServer/scripts/task_process/task_proc_wrapper.sh

Lines 73 to 77 in 43208f8

	function perform_condorq {
	DAG_INFO=$(condor_q -constr 'CRAB_ReqName =?= "'$REQUEST_NAME'" && stringListMember(TaskType, "ROOT PROCESSING TAIL", " ")' -af ClusterId JobStatus EnteredCurrentStatus)
	TIME_OF_LAST_QUERY=$(date +"%s")
	log "HTCondor query of DAG status done on $(date '+%Y/%m/%d %H:%M:%S %Z')"
	}

CRABServer/scripts/AdjustSites.py

Line 452 in 43208f8

tailconst = "TaskType =?= \"TAIL\" && CRAB_ReqName =?= %s" % classad.quote(ad.get("CRAB_ReqName"))

CRABServer/src/python/TaskWorker/Actions/DagmanResubmitter.py

Line 73 in 43208f8

rootConst = f"TaskType =?= \"ROOT\" && CRAB_ReqName =?= {classad.quote(workflow)}"

CRABServer/src/python/TaskWorker/Actions/Recurring/RenewRemoteProxies.py

Line 153 in 43208f8

task_ads = schedd.query('TaskType =?= "ROOT" && CRAB_HC =!= "True"', QUERY_ATTRS)

CRABServer/src/python/TaskWorker/Actions/DagmanKiller.py

Line 63 in 43208f8

const = f'CRAB_ReqName =?= {classad.quote(self.workflow)} && TaskType=?="Job"'

CRABServer/src/python/TaskWorker/Actions/DagmanKiller.py

Line 75 in 43208f8

rootConst = f'stringListMember(TaskType, "ROOT PROCESSING TAIL", " ") && CRAB_ReqName =?= {classad.quote(self.workflow)}'

[belforte]

do not put priority very high, since in the end it is quite some work, need to changes a lot of places, for "small" gain