Rebuild for CUDA 11.8 support #70

regro-cf-autotick-bot · 2023-09-02T09:40:20Z

This PR has been triggered in an effort to update cuda118.

Notes and instructions for merging this PR:

Please merge the PR only after the tests have passed.
Feel free to push to the bot's branch to update this PR if needed.

Please note that if you close this PR we presume that the feedstock has been rebuilt, so if you are going to perform the rebuild yourself don't close this PR until the your rebuild has been merged.

If this PR was opened in error or needs to be updated please add the bot-rerun label to this PR. The bot will close this PR and schedule another one. If you do not have permissions to add this label, you can use the phrase @conda-forge-admin, please rerun bot in a PR comment to have the conda-forge-admin add it for you.

_{This PR was created by the regro-cf-autotick-bot. The regro-cf-autotick-bot is a service to automatically track the dependency graph, migrate packages, and propose package version updates for conda-forge. Feel free to drop us a line if there are any issues! This PR was generated by https://github.com/regro/cf-scripts/actions/runs/6057259647, please use this URL for debugging.}

Closes #64

…nda-forge-pinning 2023.09.02.03.58.03

conda-forge-webservices · 2023-09-02T09:40:24Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

jakirkham · 2023-11-28T04:02:19Z

@conda-forge-admin , please re-render

…nda-forge-pinning 2023.11.27.19.54.17

conda-forge.yml

jakirkham · 2023-11-28T04:07:15Z

@conda-forge-admin , please re-render

…nda-forge-pinning 2023.11.27.19.54.17

jakirkham · 2023-11-28T04:12:47Z

Think we will want this logic as well ( c2edc96 )

Edit: Done in commit ( 805825f ) below

recipe/meta.yaml

jakirkham · 2023-11-28T04:43:49Z

@conda-forge-admin , please re-render

github-actions · 2023-11-28T04:46:22Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/faiss-split-feedstock/actions/runs/7014307041.

jakirkham · 2023-11-28T04:48:03Z

recipe/build-lib.sh

+    if [ $(version2int $cuda_compiler_version) -ge $(version2int "11.8") ]; then
+        # Hopper support for H100 (sm_90) needs cuda >= 11.8
+        LATEST_ARCH=90
+        # ARCHES does not contain LATEST_ARCH; see usage below
+        ARCHES=( "${ARCHES[@]}" 75 80 86)


Added a CUDA 11.8 branch here that includes sm_90

Please let me know if anything else is needed here

jakirkham · 2023-11-28T05:43:39Z

recipe/build-lib.sh

+        # Hopper support for H100 (sm_90) needs cuda >= 11.8
+        LATEST_ARCH=90
+        # ARCHES does not contain LATEST_ARCH; see usage below
+        ARCHES=( "${ARCHES[@]}" 75 80 86)


Depending on how we want to handle sm_89 (if at all)

We could potentially add sm_89 here

Suggested change

ARCHES=( "${ARCHES[@]}" 75 80 86)

ARCHES=( "${ARCHES[@]}" 75 80 86 89)

Alternatively we could replace sm_86 with sm_89

Suggested change

ARCHES=( "${ARCHES[@]}" 75 80 86)

ARCHES=( "${ARCHES[@]}" 75 80 89)

Some relevant background

Given we are already running into CI time limits on Azure, think the answer needs to be picking 86 or 89, but not both (at least not without dropping others)

jakirkham · 2023-11-28T06:53:15Z

It looks like TestClustering.test_ivf_train_2level failed on the CPU only build. Not sure what is going on there

Punting on the arch builds where we are running into CI limits. Likely need to trim GPU architectures there. Had tried pushing on cross-compiling as an alternative in the past without much luck ( #62 )

h-vetinari · 2023-11-28T07:22:08Z

Something we can do (finally) is split off the AVX2 builds into separate jobs. It's on my backlog.

Currently this feedstock doesn't have CUDA arch builds. This PR adds them as CUDA arch builds as they are handled as part of enabling arch builds. Since these timeout currently and may require a bit more work to figure out (like adding cross-compilation), disable CUDA arch builds for now. Instead focus on OSes and architectures already supported here.

jakirkham · 2023-12-19T07:57:12Z

@conda-forge-admin , please re-render

…nda-forge-pinning 2023.12.19.06.41.24

jakirkham · 2023-12-19T08:10:04Z

@conda-forge-admin , please re-render

…nda-forge-pinning 2023.12.19.06.41.24

jakirkham · 2023-12-19T08:16:30Z

@conda-forge-admin , please re-render

…nda-forge-pinning 2023.12.19.08.11.07

jakirkham · 2023-12-19T08:21:54Z

@conda-forge-admin , please re-render

…nda-forge-pinning 2023.12.19.08.11.07

jakirkham · 2023-12-20T01:01:47Z

It looks like TestClustering.test_ivf_train_2level failed on the CPU only build. Not sure what is going on there

Looks like upstream relaxed the threshold of this test ( facebookresearch/faiss#2927 ). Have added a patch doing the same

Punting on the arch builds where we are running into CI limits. Likely need to trim GPU architectures there. Had tried pushing on cross-compiling as an alternative in the past without much luck ( #62 )

Have disabled these above

jakirkham · 2023-12-20T01:07:35Z

On Windows CUDA 11.8, it appears the job hung waiting for input in the midst of running the tests

tests\test_fast_scan_ivf.py ............................................ [ 34%]
Traceback (most recent call last):
  File "C:\Miniforge\Scripts\conda-mambabuild-script.py", line 9, in <module>
    sys.exit(main())
  File "C:\Miniforge\lib\site-packages\boa\cli\mambabuild.py", line 256, in main
    call_conda_build(action, config)
  File "C:\Miniforge\lib\site-packages\boa\cli\mambabuild.py", line 228, in call_conda_build
    result = api.build(
  File "C:\Miniforge\lib\site-packages\conda_build\api.py", line 253, in build
    return build_tree(
  File "C:\Miniforge\lib\site-packages\conda_build\build.py", line 3819, in build_tree
    test(pkg, config=metadata.config.copy(), stats=stats)
  File "C:\Miniforge\lib\site-packages\conda_build\build.py", line 3625, in test
    utils.check_call_env(
  File "C:\Miniforge\lib\site-packages\conda_build\utils.py", line 445, in check_call_env
    return _func_defaulting_env_to_os_environ("call", *popenargs, **kwargs)
  File "C:\Miniforge\lib\site-packages\conda_build\utils.py", line 416, in _func_defaulting_env_to_os_environ
    proc = PopenWrapper(_args, **kwargs)
  File "C:\Miniforge\lib\site-packages\conda_build\utils.py", line 286, in __init__
    self.out, self.err = self._execute(*args, **kwargs)
  File "C:\Miniforge\lib\site-packages\conda_build\utils.py", line 360, in _execute
    self.disk = max(directory_size(disk_usage_dir), self.disk)
  File "C:\Miniforge\lib\site-packages\conda_build\utils.py", line 186, in directory_size
    out = subprocess.check_output(command.format(path), shell=True)
  File "C:\Miniforge\lib\subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\Miniforge\lib\subprocess.py", line 505, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "C:\Miniforge\lib\subprocess.py", line 1141, in communicate
    stdout = self.stdout.read()
KeyboardInterrupt
.............................................................

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
%PREFIX%\lib\site-packages\faiss\swigfaiss.py:5811: KeyboardInterrupt
(to show a full traceback on KeyboardInterrupt use --full-trace)
========== 334 passed, 1 skipped, 4 deselected in 116.91s (0:01:56) ===========
Terminate batch job (Y/N)? 
Entering debug mode. Use h or ? for help. 

At D:\a\_tasks\CmdLine_d9bafed4-0b18-4f58-968d-86655b4d2ce9\2.231.0\ps_modules\VstsTaskSdk\ToolFunctions.ps1:113
char:13

+         if ($originalEncoding) {

+             ~~~~~~~~~~~~~~~~~
^C
[DBG]: PS D:\a\1\s>>

Are we running pytest with pdb enabled? If so, that might explain this behavior (and we should disable using pdb in CI to fix this)

jakirkham · 2024-01-06T03:37:06Z

It looks like TestClustering.test_ivf_train_2level failed on the CPU only build. Not sure what is going on there

Looks like upstream relaxed the threshold of this test ( facebookresearch/faiss#2927 ). Have added a patch doing the same

Appears that was not enough

>       self.assertLess(ndiff, 51)
E       AssertionError: 52 not less than 51

On Windows CUDA 11.8, it appears the job hung waiting for input in the midst of running the tests

Sorry think I misread. It looks like CUDA 11.2 jobs for Windows & Linux exceed the CI time limit ( 6hrs )

jakirkham · 2024-03-07T21:01:22Z

@conda-forge-admin , please re-render

…nda-forge-pinning 2024.03.07.20.43.08

jakirkham · 2024-03-12T02:41:04Z

JFYI we are planning to drop CUDA 11.2 support in conda-forge soon

Please see this announcement and this issue ( conda-forge/conda-forge-pinning-feedstock#5339 ) for more details

The smoothest upgrade path would be to add CUDA 11.8. IOW completing this migration PR

Please let us know if you have any questions on next steps

jakirkham · 2024-05-02T03:03:56Z

JFYI CUDA 11.2 is now officially dropped

jakirkham · 2024-05-02T03:08:22Z

@conda-forge-admin , please re-render

…nda-forge-pinning 2024.05.01.20.49.02

conda-forge-webservices · 2024-05-02T03:10:50Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

This recipe is using a compiler, which now requires adding a build dependence on {{ stdlib("c") }} as well. For further details, please see META: {{ stdlib("c") }} migration conda-forge.github.io#2102.

Also fix lint

conda-forge-webservices · 2024-05-02T03:28:12Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

jakirkham · 2024-05-02T03:28:23Z

@conda-forge-admin , please re-render

…nda-forge-pinning 2024.05.01.20.49.02

regro-cf-autotick-bot added 2 commits September 2, 2023 09:38

Rebuild for CUDA 11.8 w/arch support

d939fff

MNT: Re-rendered with conda-build 3.26.1, conda-smithy 3.25.1, and co…

e818b7a

…nda-forge-pinning 2023.09.02.03.58.03

regro-cf-autotick-bot requested a review from h-vetinari as a code owner September 2, 2023 09:40

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.30.0, and co…

25b7d7f

…nda-forge-pinning 2023.11.27.19.54.17

jakirkham reviewed Nov 28, 2023

View reviewed changes

conda-forge.yml Outdated Show resolved Hide resolved

Drop cross-compilation of Linux arch builds

8603347

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.30.0, and co…

05e4465

…nda-forge-pinning 2023.11.27.19.54.17

Add CUDA 11.8 branch for sm_90

805825f

jakirkham reviewed Nov 28, 2023

View reviewed changes

recipe/meta.yaml Outdated Show resolved Hide resolved

Drop outdated skips

6596e1b

jakirkham reviewed Nov 28, 2023

View reviewed changes

jakirkham mentioned this pull request Nov 28, 2023

Update CUDA arches for CUDA 11.8 / sm_90 support. #64

Closed

5 tasks

jakirkham reviewed Nov 28, 2023

View reviewed changes

conda-forge-webservices[bot] and others added 2 commits December 19, 2023 07:59

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.30.1, and co…

edb7bcb

…nda-forge-pinning 2023.12.19.06.41.24

Treat cuda_compiler as "None" if undefined

baf1206

conda-forge-webservices[bot] and others added 2 commits December 19, 2023 08:12

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.30.1, and co…

8e3c0e1

…nda-forge-pinning 2023.12.19.06.41.24

Check cuda_compiler first in skip

bc7fd40

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.30.1, and co…

ae30d84

…nda-forge-pinning 2023.12.19.08.11.07

Only do CUDA builds on x86_64

6be560e

conda-forge-webservices[bot] and others added 2 commits December 19, 2023 08:24

MNT: Re-rendered with conda-build 3.27.0, conda-smithy 3.30.1, and co…

4e92681

…nda-forge-pinning 2023.12.19.08.11.07

Drop extraneous blank space

c5baf50

jakirkham changed the title ~~Rebuild for CUDA 11.8 w/arch support~~ Rebuild for CUDA 11.8 support Dec 19, 2023

Apply upstream for test_ivf_train_2level

c921455

MNT: Re-rendered with conda-build 24.1.2, conda-smithy 3.31.1, and co…

ae6f2a9

…nda-forge-pinning 2024.03.07.20.43.08

jakirkham mentioned this pull request Mar 7, 2024

Dropping CUDA 11.2 conda-forge/conda-forge-pinning-feedstock#5339

Closed

MNT: Re-rendered with conda-build 24.3.0, conda-smithy 3.35.0, and co…

2694d7e

…nda-forge-pinning 2024.05.01.20.49.02

Add {{ stdlib('c') }}

7c467a7

Also fix lint

MNT: Re-rendered with conda-build 24.3.0, conda-smithy 3.35.0, and co…

47a385f

…nda-forge-pinning 2024.05.01.20.49.02

h-vetinari mentioned this pull request Aug 8, 2024

faiss v1.8.0 #75

Merged

h-vetinari closed this in #75 Aug 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rebuild for CUDA 11.8 support #70

Rebuild for CUDA 11.8 support #70

regro-cf-autotick-bot commented Sep 2, 2023 •

edited by jakirkham

Loading

conda-forge-webservices bot commented Sep 2, 2023

jakirkham commented Nov 28, 2023

jakirkham commented Nov 28, 2023

jakirkham commented Nov 28, 2023 •

edited

Loading

jakirkham commented Nov 28, 2023

github-actions bot commented Nov 28, 2023

jakirkham Nov 28, 2023

jakirkham Nov 28, 2023 •

edited

Loading

jakirkham Nov 29, 2023

jakirkham commented Nov 28, 2023

h-vetinari commented Nov 28, 2023

jakirkham commented Dec 19, 2023

jakirkham commented Dec 19, 2023

jakirkham commented Dec 19, 2023

jakirkham commented Dec 19, 2023

jakirkham commented Dec 20, 2023

jakirkham commented Dec 20, 2023

jakirkham commented Jan 6, 2024

jakirkham commented Mar 7, 2024

jakirkham commented Mar 12, 2024

jakirkham commented May 2, 2024

jakirkham commented May 2, 2024

conda-forge-webservices bot commented May 2, 2024

conda-forge-webservices bot commented May 2, 2024

jakirkham commented May 2, 2024

	ARCHES=( "${ARCHES[@]}" 75 80 86)
	ARCHES=( "${ARCHES[@]}" 75 80 86 89)

	ARCHES=( "${ARCHES[@]}" 75 80 86)
	ARCHES=( "${ARCHES[@]}" 75 80 89)

Rebuild for CUDA 11.8 support #70

Rebuild for CUDA 11.8 support #70

Conversation

regro-cf-autotick-bot commented Sep 2, 2023 • edited by jakirkham Loading

conda-forge-webservices bot commented Sep 2, 2023

jakirkham commented Nov 28, 2023

jakirkham commented Nov 28, 2023

jakirkham commented Nov 28, 2023 • edited Loading

jakirkham commented Nov 28, 2023

github-actions bot commented Nov 28, 2023

jakirkham Nov 28, 2023

Choose a reason for hiding this comment

jakirkham Nov 28, 2023 • edited Loading

Choose a reason for hiding this comment

jakirkham Nov 29, 2023

Choose a reason for hiding this comment

jakirkham commented Nov 28, 2023

h-vetinari commented Nov 28, 2023

jakirkham commented Dec 19, 2023

jakirkham commented Dec 19, 2023

jakirkham commented Dec 19, 2023

jakirkham commented Dec 19, 2023

jakirkham commented Dec 20, 2023

jakirkham commented Dec 20, 2023

jakirkham commented Jan 6, 2024

jakirkham commented Mar 7, 2024

jakirkham commented Mar 12, 2024

jakirkham commented May 2, 2024

jakirkham commented May 2, 2024

conda-forge-webservices bot commented May 2, 2024

conda-forge-webservices bot commented May 2, 2024

jakirkham commented May 2, 2024

regro-cf-autotick-bot commented Sep 2, 2023 •

edited by jakirkham

Loading

jakirkham commented Nov 28, 2023 •

edited

Loading

jakirkham Nov 28, 2023 •

edited

Loading