add a per-user policy to tape recall

[belforte]

last week two users managed to put tape recall requests for a total of about 7PB

We can't deliver that "timely" nor can they process all that data. Clearly they do not know what they are doing.

We want to put a limit on how much each individual user can request at any given time, i.e. until all of her/his previous requests are satisfied.

We had an initial discussion in DM Ops MatterMost, moving here to converge on a concrete proposal and implement it.

@dynamic-entropy

[belforte]

Rahul, currently each rule that we create has a comment like this

Comment:                    Recall 605 GBytes for user: sleontsi dataset: /Charmonium/Run2016E-21Feb2020_UL2016_HIPM-v1/MINIAOD

Format is fixed becasue we use it for our monitoring

I am open to change to (or add as) a rule metadata.

I think that what we need is a small database of how much a given user has already in recall in order to make a decision whether to accept a request or not.

Can you summarize how you do this for automatic recall ?

[dynamic-entropy]

Hello, Stefano
I saw your mail. Did they just repeat the jobs that triggered the recalls?
Does this need to be taken urgently? I wanted to do some tests on my own to pitch a concrete proposal before saying anything, hence the delay.

[belforte]

Simply I had not done anything until now. I think he will be careful. But we do not want to continuously watch things.

[belforte]

On Hold, waiting for a proposal from @dynamic-entropy on how to track usage by user.
As a reminder currently limits to each request are applied here

CRABServer/src/python/TaskWorker/Actions/DBSDataDiscovery.py

Line 474 in 5ca7cae

def executeTapeRecallPolicy(self, inputDataset, inputBlocks, totalSizeBytes):

And currently those are in English:

• tiers in ['AOD', 'AODSIM', 'MINIAOD', 'MINIAODSIM', 'NANOAOD', 'NANOAODSIM'] are recalled no matter how big
• other data tiers are recalled if recall request is <50TB
  • users can indicate a list of blocks which fits if dataset is too big

Beyond that the old plan was to have a global quota cap on crab_tape_recall account. Couple of years ago Nick told me that this cap was there but for technical reasons was not implemented in Rucio. I do not know any details on that.

[belforte]

Stefano and Rahul had a chat. Proposal:

• tape recall rules will be created with account=username and activity="Analysis Tape Recall" and askApproval=True
• Rahul will make RUcio accept that activity.
• Rahul will make sure that only crab_tape_recall account can create rules in this activity and they will be automatically approved. This will avoid that the rule is charged to the user Rucio quota on the selected RSE
• CRAB will ask Rucio for all rule belonging to the user wich this activity and compute the overall size (Rahul has code example to do it https://github.com/dmwm/CMSRucio/blob/5469bb52226ce9cde59eb7b538fcb96cde386a1f/src/policy/CMSRucioPolicy/permission.py#L179-L186 )
• CRAB will then apply the limit and communicate to user accordingly
• per-user-limit will be set in coordination with Comp.Ops. We can start with 200TB total

we prefer to do the check in CRAB code because Rucio does not have a way (yet) to communicate to the user what the usage is etc. Only a yes/no.

If computing the total size for a user takes too long, we will reconsider

[belforte]

implementation tasks in https://github.com/dmwm/CRABServer/blob/master/src/python/TaskWorker/Actions/RucioActions.py

• add the quota check in DBSDataDiscovery/executeTapeRecallPolicy ?
• add user limit in ServerUtilities with all other constants ? Or in TW config like e.g. maxTierToBlockRecallSize ? i.e. in https://gitlab.cern.ch/ai/it-puppet-hostgroup-vocmsglidein/-/blob/master/code/templates/crabtaskworker/taskworker/TaskWorkerConfig.py.erb#L107-108
• pass activity as arguments to createOrReuseRucioRule . Account can be selected inside there since self.username is available
• make sure that rule is removed once data is recalled so that user regains their quota. Modify TapeRecall recurring action. Now data will be locked as soon as task is submitted, letting rule to expire in 1h is fine.

[belforte]

@dynamic-entropy
a first implementation is ready in https://github.com/belforte/CRABServer/tree/add-user-policy-to-tape-recall-8354

currently it uses crab_tape_recall account and Analysis Input activity with AskForApproval=False. Waiting for Analysis Tape Recall activity to be accepted by Rucio with automatic approval. So that I can verify that nothing is broken.

if you care to have a look, relevant code changes are below. Any comment is always welcome

https://github.com/belforte/CRABServer/blob/72c66a157e1263637108afa0ac5fbfcf5467cf06/src/python/RucioUtils.py#L98-L120

https://github.com/belforte/CRABServer/blob/72c66a157e1263637108afa0ac5fbfcf5467cf06/src/python/TaskWorker/Actions/RucioActions.py#L62-L85

https://github.com/belforte/CRABServer/blob/72c66a157e1263637108afa0ac5fbfcf5467cf06/src/python/TaskWorker/Actions/DBSDataDiscovery.py#L529-L550

[belforte]

on hold. wating for at least the new activity to be allowed on Rucio side.
@dynamic-entropy I think that's safe enough that you can put it in production w/o me testing on integration, right ?

Then we can put in production and test still using account crab_tape_recall and askforapproval=False and switch those once code is tested and the permission part in Rucio is also true.

[dynamic-entropy]

Hello Stefano
I have created a PR to add the activity to the schema as well as check for the crab_tape_recall account, after which the rule shall created.
The approve part will need yet another PR. But now you can create rules without askApproval.
dmwm/CMSRucio#802

[belforte]

thank you Rahul. Kindly let me know when it is deployed so I can try it and verify that I correctly compute the used quota

[dynamic-entropy]

Sure, I have added commits for auto-approve too. I will let you know when it's deployed.
Cheers

[dynamic-entropy]

Hello @belforte
It's in production now. Apologies, for the delay.
Kindly test if everything is okay and let me know if more things need plumbing.
Cheers

[belforte]

many thanks @dynamic-entropy
I am awfully busy atm with VOMS-to-IAM transition and removing deprecated HTCondor calls from CRAB.
But this is top of the list after that.
Hopefully sometimes next week.

[belforte]

Hi @dynamic-entropy
I finally got to test and have stumbled on a permission issue with this code
https://github.com/belforte/CRABServer/blob/1b400d1d3a739024960c2c0980e349b53d66d28f/src/python/TaskWorker/Actions/RucioActions.py#L79-L84

            ruleIds = self.rucioClient.add_replication_rule(  # N.B. returns a list
                dids=[did], copies=copies, rse_expression=rseExpression,
                grouping=grouping, weight=weight, lifetime=lifetime,
                account=account, activity=activity,
                comment=comment,
                ask_approval=askApproval, asynchronous=True)

it appears that no matter what I put as account, Rucio will try to create the use using the user of the current client and I get

Rucio exception creating rule: Access to the requested resource denied.
Details: Account crab_tape_recall can not add replication rule

I have askApprova=True and activity="Analysis TapeRecall"

Any idea ? Your PR has the same crab_tape_recall username so it does not look a simple typo.

[belforte]

maybe scope matters ?
This is a test where I have this did '{scope': 'user.crab_tape_recall', 'name': '/TapeRecall/pippo1/USER'}

[dynamic-entropy]

Hello @belforte

I am unable to reproduce this.

rucio add-rule user.crab_tape_recall:/TapeRecall/pippo1/USER 1 T2_RC_MOCK --lifetime 86400 --activity "Analysis TapeRecall" --ask-approval --account rchauhan 

d135f23c65764f9f9bf0b5a3600ecae3

https://cms-rucio-webui.cern.ch/rule?rule_id=d135f23c65764f9f9bf0b5a3600ecae3#locks

Can you please provide me with the values of those variables in the test?

[belforte]

@dynamic-entropy how could your command in #8354 (comment) work w/o number of copies and RSE expression ?

[belforte]

this worked for me as well from CLI

belforte@lxplus806/Rucio> rucio add-rule $did  --activity "Analysis TapeRecall" --ask-approval --account belforte 1 T1_FR_CCIN2P3_Disk
0e615779a04940199cd7e9f218243b04
belforte@lxplus806/Rucio> rucio whoami
status     : ACTIVE
deleted_at : None
updated_at : 2020-10-15T20:57:44
suspended_at : None
account    : crab_tape_recall
account_type : SERVICE
email      : stefano.belforte@cern.ch
created_at : 2020-10-15T20:57:44
belforte@lxplus806/Rucio> rucio list-content $did
+------------------------------------------------------------------+--------------+
| SCOPE:NAME                                                       | [DID TYPE]   |
|------------------------------------------------------------------+--------------|
| cms:/MuonEG/Run2016B-v1/RAW#86bc5e3e-1519-11e6-a3f4-001e67ac06a0 | DATASET      |
+------------------------------------------------------------------+--------------+
belforte@lxplus806/Rucio> echo $did
user.crab_tape_recall:/TapeRecall/testtask-1720020665/USER
belforte@lxplus806/Rucio> 

I will try to find what is different when doing it in python

[dynamic-entropy]

how could your command in #8354 (comment) work w/o number of copies and RSE expression ?

Hello @belforte ,
I do specify those

rucio add-rule user.crab_tape_recall:/TapeRecall/pippo1/USER 1 T2_RC_MOCK [NAMES ARGUMENTS]

You created the rule for FR_CCIN2P3 this will trigger transfers.
Its okay if you also wanted this to be the case. However, for testing you can use the T2_RC_MOCK rse. this does not have any protocols specified.

[belforte]

thanks for the tip about MOCK. I will cleanup

[belforte]

when trying a minimal python, I get a new error, w/o any useful detail :-(

belforte@lxplus806/Rucio> python3
Python 3.6.8 (default, Apr 25 2024, 09:54:46) 
[GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from rucio.client import Client
>>> rucio=Client(account='crab_tape_recall')
>>> rucio.whoami()
{'status': 'ACTIVE', 'deleted_at': None, 'updated_at': '2020-10-15T20:57:44', 'suspended_at': None, 'account': 'crab_tape_recall', 'email': 'stefano.belforte@cern.ch', 'account_type': 'SERVICE', 'created_at': '2020-10-15T20:57:44'}
>>> did='user.crab_tape_recall:/TapeRecall/testtask-1720020665/USER'
>>> ruleIds = rucio.add_replication_rule(
...     dids=[did], copies=1,
...     grouping='ALL', weight='ddm_quota', lifetime=86400,
...     account='belforte', activity='Analysis TapeRecall',
...     comment='this is a test', ask_approval=True,
...     rse_expression='T2_RC_MOCK',
...     asynchronous=True)
Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "/cvmfs/cms.cern.ch/rucio/x86_64/rhel8/py3/current/lib/python3.6/site-packages/rucio/client/ruleclient.py", line 91, in add_replication_rule
    raise exc_cls(exc_msg)
rucio.common.exception.RucioException: An unknown exception occurred.
Details: no error information passed (http status code: 500)
>>> 

what is different from CLI which worked above ?

[belforte]

user client version (user.user_agent) is the same in the code and the example in lxplus (1.31.7). Why http 500 vs. Access to the requested resource denied.\nDetails: Account crab_tape_recall can not add replication rule ?

I am quite confused and unsure what to do :-(

[belforte]

so fare rucio.add_replication_rule worked for me, in this case I only change value of arguments.

[belforte]

explicitly adding auth_type='x509' to Rucio client instantiation resulted in same http 500 error. I am quite curious why I can't reproduce the resource denied in the simple test.

[dynamic-entropy]

when trying a minimal python, I get a new error, w/o any useful detail :-(

Hello @belforte , did should be a dict of the form
{'scope':'user.crab_tape_recall', name:'<lfn>}

[dynamic-entropy]

I didn't check the logs yet, but it seems like this is not handled properly in the rucio code.
Can you give it a try?

Cheers

[belforte]

🤦

[belforte]

thanks @dynamic-entropy . Using the correct DID format makes the standalone python script work. I should be abel to find out what makes it fail in the code.

[belforte]

The error appears when I use this rse_expression

T1_DE_KIT_Disk|T1_ES_PIC_Disk|T1_FR_CCIN2P3_Disk|T1_IT_CNAF_Disk|T1_RU_JINR_Disk|T1_UK_RAL_Disk|T1_US_FNAL_Disk|T2_BE_IIHE|T2_BE_UCL|T2_BR_SPRACE|T2_CH_CERN|T2_CH_CSCS|T2_DE_DESY|T2_DE_RWTH|T2_EE_Estonia|T2_ES_CIEMAT|T2_FI_HIP|T2_FR_GRIF|T2_FR_GRIF_IRFU|T2_FR_GRIF_LLR|T2_FR_IPHC|T2_HU_Budapest|T2_IT_Bari|T2_IT_Legnaro|T2_IT_Pisa|T2_IT_Rome|T2_KR_KISTI|T2_RU_JINR|T2_UK_London_IC|T2_UK_SGrid_RALPP|T2_US_Caltech|T2_US_Florida|T2_US_MIT|T2_US_Nebraska|T2_US_Purdue|T2_US_UCSD|T2_US_Vanderbilt|T2_US_Wisconsin

if I simply put T1_FR_CCIN2P3 it works.

puzzled.

[dynamic-entropy]

Hmm, so the reason is,
https://github.com/dmwm/CMSRucio/blob/26e67bead81417e59fd2811569b842db2fddd7f6/src/policy/CMSRucioPolicy/permission.py#L318-L320

Let me fix it asap

[dynamic-entropy]

Hello @belforte
Should be good now. Can you please try again?
Cheers

[belforte]

Works finely now. Thanks a lot @dynamic-entropy 🙏

[dynamic-entropy]

I see you created rules under my account 😅

[belforte]

I was testing copypasting your command line before I read your update.
your, mine... the command is issued by crab_tape_recall, makes no difference.
I am deleting all those rules anyhow

[belforte]

mechanics is working now, I will work next week on making sure that also checking user limits and enforcing quota works before putting in production.
Thanks again @dynamic-entropy

[ericvaandering]

Closing

[ericvaandering]

Sorry, this is your issue not mine. It was in my board. :-)