Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up examples CI #279

Merged
merged 6 commits into from
Mar 10, 2023
Merged

Speed up examples CI #279

merged 6 commits into from
Mar 10, 2023

Conversation

hmellor
Copy link
Member

@hmellor hmellor commented Mar 9, 2023

What does this PR do?

  • Prevents the venvs created for each example from clashing
  • Uses a FileLock to stop concurrent pip installs of optimum-graphcore
  • Add POPTORCH_WAIT_FOR_IPU=1 to test-examples.yml, this allows:
    • Multiple examples to compile concurrently
    • Maximum IPU utilisation because the IPUs won't be idle while waiting for the next example to compile
  • Increase the number of workers in CI to 4

@hmellor hmellor changed the title Speed up CI Speed up examples CI Mar 9, 2023
@regisss
Copy link
Contributor

regisss commented Mar 9, 2023

This PR should fix the doc build: #280

@hmellor
Copy link
Member Author

hmellor commented Mar 10, 2023

Great, thanks for the fix. Will I be ok to land this without the docs build passing? (assuming an approving review)

Also, re my most recent commit, I've reduced the worker count from 16 down to 4 because we likely don't have the RAM/swap to compile 16 models at once.

@jimypbr jimypbr merged commit 054b4a2 into huggingface:main Mar 10, 2023
ncouro-gc pushed a commit to graphcore/optimum-graphcore-fork that referenced this pull request Mar 17, 2023
* Add `POPTORCH_WAIT_FOR_IPU=1` to CI workflow
* Run with 4 parallel workers to overlap compute and compilation of different examples
ncouro-gc pushed a commit to graphcore/optimum-graphcore-fork that referenced this pull request Mar 17, 2023
* Add `POPTORCH_WAIT_FOR_IPU=1` to CI workflow
* Run with 4 parallel workers to overlap compute and compilation of different examples
@hmellor hmellor deleted the parallelise-ci branch March 22, 2023 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants