Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove un-needed dependencies from crabtaskworker-pypi #8518

Closed
belforte opened this issue Jun 19, 2024 · 2 comments
Closed

remove un-needed dependencies from crabtaskworker-pypi #8518

belforte opened this issue Jun 19, 2024 · 2 comments
Assignees

Comments

@belforte
Copy link
Member

it currently takes O(10) min to build TW container in the pipeline.
I think it is because we still have a list of dependencies from "old rpm times" with all possible things used by all possible WMCore application in https://github.com/dmwm/CRABServer/blob/master/cicd/crabtaskworker_pypi/requirements.txt
Already https://github.com/dmwm/CRABServer/blob/master/cicd/crabserver_pypi/requirements.txt is much more trimmed.

@novicecpp
Copy link
Contributor

Thanks Stefano.
From what I saw in CI machine, most CPU is gone due to I/O wait. Reducing the deps will help cutting time by a lot.

@belforte
Copy link
Member Author

yeah, I had been looking at pipeline log in realtime and noticed that all those pip install were taking a lot.
I built pypi-test2-reducetwdep with that and at least TW is built in same time as server. A lot of time goes in conda/gfal as you know. I guess the only way to cut time now is to create a base image with gfal and start from that. Not sure it is worth

belforte added a commit to belforte/CRABServer that referenced this issue Jun 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants