Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MPS test jobs are failing #8433

Closed
NicolasHug opened this issue May 20, 2024 · 3 comments
Closed

MPS test jobs are failing #8433

NicolasHug opened this issue May 20, 2024 · 3 comments

Comments

@NicolasHug
Copy link
Member

with

Traceback (most recent call last):
  File "/Users/runner/work/vision/vision/pytorch/vision/test/smoke_test.py", line 103, in <module>
    main()
  File "/Users/runner/work/vision/vision/pytorch/vision/test/smoke_test.py", line 99, in main
    smoke_test_torchvision_resnet50_classify("mps")
  File "/Users/runner/work/vision/vision/pytorch/vision/test/smoke_test.py", line 52, in smoke_test_torchvision_resnet50_classify
    img = read_image(str(SCRIPT_DIR / ".." / "gallery" / "assets" / "dog2.jpg")).to(device)
RuntimeError: MPS backend out of memory (MPS allocated: 0 bytes, other allocations: 0 bytes, max allowed: 1.70 GB). Tried to allocate 0 bytes on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).

example: https://github.com/pytorch/vision/actions/runs/9157662717/job/25174485539?pr=7990

@NicolasHug
Copy link
Member Author

Seems that all the failures are related to MPS - pinging @qqaatw as you've been our POC on this so far :)
Do you have any idea what might be causing this? Has there been any change on the PyTorch core side related to MPS that could be the source? Thanks!

@qqaatw
Copy link
Contributor

qqaatw commented May 20, 2024

Hmm, it looks related to MPSAllocator. cc @kulinseth

@NicolasHug NicolasHug changed the title MacOS unit test jobs are red MPS test jobs are failing May 28, 2024
@NicolasHug
Copy link
Member Author

Looks like it wasn't directly related to MPS but rather related to the recent drop of macos x86 support from pytorch. After #8446 I see the MPS tests passing, so I'll close this issue.

Thank you for your prompt response @qqaatw !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants