Remove OSX support, add cudadevrt #5

gmarkall · 2020-03-24T17:48:55Z

The primary aim of this PR is to add the cudadevrt static library, which is needed for dynamic parallelism, grid sync, etc. For example, it is required for this Numba PR: numba/numba#4551

It seems that the OSX packages aren't built anymore:

$ conda search cudatoolkit[subdir=osx-64]
Loading channels: done
# Name                       Version           Build  Channel     
cudatoolkit                      9.0      h41a26b3_0  pkgs/main

and the recipe doesn't even point to a correct URL for the toolkit, so rather than fixing that up, I've removed the OSX builder instead.

OSX CUDA Toolkit packages have not been published since 9.1, and the present build.py implementation does not correctly build the package.

cudadevrt is a static library, so we add an additional platform-specific key and related logic to handle copying static libraries.

gmarkall · 2020-03-24T17:49:54Z

I forgot to mention, example packages built with the recipe in this PR for Linux and Windows are at: https://anaconda.org/gmarkall/cudatoolkit/files

jakirkham · 2020-03-24T17:55:22Z

@jjhelmus, would it be possible to get a review here? 🙂

leofang · 2020-03-24T17:58:39Z

Holy xxxx! This is not in cudatoolkit yet?! CuPy also needs it for the very same reasons...

leofang · 2020-03-24T18:04:26Z

It seems even with this addition, CuPy would still be unable to locate it...This is our searching strategy: https://github.com/cupy/cupy/blob/775cb7d9ccfbee8aebb46ff7c310dddc3a2b72af/cupy/cuda/compiler.py#L78-L100
Basically, we look for

CUDA_PATH (@jakirkham, is this set by cudatoolkit??)
the parent directory of nvcc (on CF this does not come with cudatoolkit)
/usr/local/cuda (not applicable to CF)

Am I right that none of these is met by conda-forge's packaging?

jakirkham · 2020-03-24T18:07:04Z

Interesting did not realize this was needed by CuPy too. Should we open an issue on the cupy-feedstock to discuss further? 😉

gmarkall · 2020-03-24T18:11:25Z

With the cudatoolkit package installed (and also an installation of the toolkit at /usr/local/cuda, I get:

In [1]: import cupy                                                                                                                                          

In [2]: from cupy.cuda import get_cuda_path                                                                                                                  

In [3]: get_cuda_path()                                                                                                                                      
Out[3]: '/usr/local/cuda'

In [4]: from numba.cuda.cudadrv.libs import test                                                                                                             

In [5]: test()                                                                                                                                               
Finding cublas from Conda environment
	located at /raid/gmarkall/miniconda3/envs/numbaenv/lib/libcublas.so.10.0.130
	trying to open library...	ok
Finding cusparse from Conda environment
	located at /raid/gmarkall/miniconda3/envs/numbaenv/lib/libcusparse.so.10.0.130
	trying to open library...	ok
Finding cufft from Conda environment
	located at /raid/gmarkall/miniconda3/envs/numbaenv/lib/libcufft.so.10.0.145
	trying to open library...	ok
Finding curand from Conda environment
	located at /raid/gmarkall/miniconda3/envs/numbaenv/lib/libcurand.so.10.0.130
	trying to open library...	ok
Finding nvvm from Conda environment
	located at /raid/gmarkall/miniconda3/envs/numbaenv/lib/libnvvm.so.3.3.0
	trying to open library...	ok
Finding libdevice from Conda environment
	searching for compute_20...	ok
	searching for compute_30...	ok
	searching for compute_35...	ok
	searching for compute_50...	ok
Out[5]: True

It looks like CuPy doesn't find the cudatoolkit binaries in general? (c.f. Numba, which prioritises the conda package - e.g. https://github.com/numba/numba/blob/master/numba/cuda/cuda_paths.py#L85 in order of search)

leofang · 2020-03-24T18:16:26Z

Right, CuPy's searching strategy (aka get_cuda_path()) would not be aware of cudatoolkit, but since the CF CuPy is correctly linked against it, nothing would go wrong in production...

@gmarkall Let me know if I am wrong: Numba can find cudatoolkit in runtime because it needs to dynamically load the CUDA libraries? In CuPy the linking is done at compile time (and for CF, package installation time), so they don't need this.

Sounds like the simplest solution is to set CUDA_PATH in cupy-feedstock? No code change is needed. @jakirkham Is this possible?

jakirkham · 2020-03-24T18:19:40Z

@gmarkall, how did you install cupy? Here is what I see with the conda-forge package:

In [1]: from cupy.cuda import get_cuda_path                                     

In [2]: get_cuda_path()                                                         
Out[2]: '/datasets/jkirkham/miniconda/envs/rapids13dev'
``

leofang · 2020-03-24T18:22:38Z

@jakirkham I think it's because you have nvcc somewhere in your conda env. Do you happen to build nvcc-feedstock locally? This is what I get from a fresh CF cupy (conda create -n CF_cupy_test -c conda-forge python=3.7 cupy cudatoolkit=10.0):

$ conda activate CF_cupy_test
$ python
Python 3.7.6 | packaged by conda-forge | (default, Mar 23 2020, 23:03:20) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cupy.cuda import get_cuda_path 
>>> get_cuda_path()
'/usr/local/cuda'

jakirkham · 2020-03-24T18:24:21Z

Ah ok. That might be. Yeah I installed nvcc_linux-64 into this environment.

jakirkham · 2020-03-24T18:31:14Z

Thanks for identifying this issue. 🙂

It seems like an easy fix upstream. Have raised issue ( cupy/cupy#3222 ).

As to setting CUDA_PATH, I'm guessing you mean setting this at runtime? We could add an activation script to the recipe. This would also work. Though generally is not preferred when there is another option like fixing upstream's path detection logic.

gmarkall · 2020-03-24T18:57:09Z

@gmarkall, how did you install cupy? Here is what I see with the conda-forge package:

In [1]: from cupy.cuda import get_cuda_path                                     

In [2]: get_cuda_path()                                                         
Out[2]: '/datasets/jkirkham/miniconda/envs/rapids13dev'
``

I did:

$ conda install cupy
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.2
  latest version: 4.8.3

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /raid/gmarkall/miniconda3/envs/numbaenv

  added / updated specs:
    - cupy


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cudnn-7.6.5                |       cuda10.0_0       165.0 MB
    cupy-6.0.0                 |   py37hc0ce245_0        10.2 MB
    fastrlock-0.4              |   py37he6710b0_0          29 KB
    nccl-1.3.5                 |       cuda10.0_0         1.3 MB
    ------------------------------------------------------------
                                           Total:       176.6 MB

The following NEW packages will be INSTALLED:

  cudnn              pkgs/main/linux-64::cudnn-7.6.5-cuda10.0_0
  cupy               pkgs/main/linux-64::cupy-6.0.0-py37hc0ce245_0
  fastrlock          pkgs/main/linux-64::fastrlock-0.4-py37he6710b0_0
  nccl               pkgs/main/linux-64::nccl-1.3.5-cuda10.0_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
nccl-1.3.5           | 1.3 MB    | ################################################################################################################## | 100% 
cudnn-7.6.5          | 165.0 MB  | ################################################################################################################## | 100% 
fastrlock-0.4        | 29 KB     | ################################################################################################################## | 100% 
cupy-6.0.0           | 10.2 MB   | ################################################################################################################## | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

The cudatoolkit package is a 10.0 one I self-built (basically this PR backported to 10.0 because I'm on a machine with an older driver that can't support 10.1 or 10.2):

$ conda list
# packages in environment at /raid/gmarkall/miniconda3/envs/numbaenv:
#
# Name                    Version                   Build  Channel
...
cudatoolkit               10.0.130             h6bb024c_0    <unknown>
...

leofang · 2020-03-24T19:15:45Z

We could add an activation script to the recipe. This would also work.

Yes this is exactly what I meant!

Though generally is not preferred when there is another option like fixing upstream's path detection logic.

I feel this is our own issue (providing a customized CUDA Toolkit that is split apart in a highly non-trivial way), and if possible we should just fix it with our own (conda) tools. It's a bit unfair to bother upstream folks for this (even if the final PR comes from one of us) as this CUDA usage was not what was originally intended (by NVIDIA or any sensible people) at all.

jakirkham · 2020-03-24T19:20:19Z

Thanks Graham! I'm also able to reproduce without nvcc_linux-64 installed.

jakirkham · 2020-03-24T19:30:18Z

Leo, let's find a new forum for this discussion. I don't want us to derail Graham's PR 😉

leofang · 2020-03-24T20:15:55Z

Sounds good, redirecting to conda-forge/cupy-feedstock#46.

leofang · 2020-03-24T20:21:48Z

@gmarkall Do you know if cudadevrt is picky on the CUDA driver version (which is beyond control of conda)?

gmarkall · 2020-03-24T20:49:18Z

@gmarkall Do you know if cudadevrt is picky on the CUDA driver version (which is beyond control of conda)?

I'm afraid I don't know, but I would guess it's probably no more picky than the other components (e.g. cudart).

leofang · 2020-04-08T18:14:55Z

Just curious, what's blocking this PR? LGTM.

jakirkham · 2020-04-08T18:18:16Z

cc @jjhelmus

leofang · 2020-04-28T03:55:51Z

Ping @jjhelmus again. Without including cudadevrt it could make downstream packages fail if dynamic parallelism or cooperative group is required, see, e.g.:
https://dev.azure.com/nsls2forge/nsls2forge/_build/results?buildId=2455&view=logs&j=d0d954b5-f111-5dc4-4d76-03b6c9d0cf7e&t=6d4b912b-175d-51da-0fd9-4d30fe1eb4e7&l=2832

jjhelmus · 2020-04-29T13:59:48Z

This is great, sorry for taking so long to review.
I'll bump the build number and work on getting new packages out. I will post an update when they are available.

gmarkall · 2020-04-29T15:31:15Z

Many thanks for the review and merge!

jakirkham · 2020-04-29T16:11:47Z

Awesome! Thanks Jonathan! 😄

jjhelmus · 2020-04-29T16:38:23Z

cudatoolkit 10.2.89 build 1 packages which include cudadevrt are now available for linux-64 and win-64 in defaults.

The specific build strings files are:
linux-64: cudatoolkit-10.2.89-hfd86e86_1
win-64: cudatoolkit-10.2.89-h74a9793_1

leofang · 2020-04-30T03:41:39Z

Thanks Jonathan! I confirm this works.

leofang · 2020-06-09T13:49:09Z

Hi @jjhelmus Is it possible to add cudadevrt to all cudatoolkit, not just to 10.2?

jakirkham · 2020-06-11T17:57:53Z

I raised issue ( #8 ) about how we might manage/maintain multiple cudatoolkit versions. Had a couple ideas on how we might do this. Thoughts welcome 🙂

gmarkall added 2 commits March 24, 2020 11:21

Remove OSX support

5d1abc4

OSX CUDA Toolkit packages have not been published since 9.1, and the present build.py implementation does not correctly build the package.

Add cudadevrt

6882aa3

cudadevrt is a static library, so we add an additional platform-specific key and related logic to handle copying static libraries.

gmarkall mentioned this pull request Mar 24, 2020

Add Basic CUDA Coordination Group Support numba/numba#4551

Closed

jakirkham mentioned this pull request Mar 24, 2020

Update CUDA search path to pick up cudatoolkit in Conda installs cupy/cupy#3222

Open

leofang mentioned this pull request Mar 24, 2020

Add new activation script and/or run dependency? conda-forge/cupy-feedstock#46

Closed

leofang mentioned this pull request Apr 7, 2020

Set/unset CUDA_PATH in activate/deactivate conda-forge/cupy-feedstock#49

Merged

5 tasks

jjhelmus merged commit d14a6dd into AnacondaRecipes:master Apr 29, 2020

leofang mentioned this pull request Jun 9, 2020

Import fails although CUDA is present cupy/cupy#3403

Open

gmarkall mentioned this pull request Jun 12, 2020

CUDA 11 RC support #7

Closed

4 tasks

leofang mentioned this pull request Jul 30, 2020

Managing multiple cudatoolkit versions #8

Open

gmarkall mentioned this pull request Sep 17, 2020

CUDA Cooperative grid groups numba/numba#6245

Merged

gmarkall mentioned this pull request Oct 13, 2020

Cudatoolkit conda-forge/staged-recipes#12882

Merged

8 tasks

jakirkham mentioned this pull request Nov 6, 2020

Feedback on cudatoolkit packages conda-forge/cudatoolkit-feedstock#15

Closed

leofang mentioned this pull request Nov 6, 2020

Should there be a cudatoolkit-static package? conda-forge/cudatoolkit-feedstock#27

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove OSX support, add cudadevrt #5

Remove OSX support, add cudadevrt #5

gmarkall commented Mar 24, 2020

gmarkall commented Mar 24, 2020

jakirkham commented Mar 24, 2020

leofang commented Mar 24, 2020

leofang commented Mar 24, 2020 •

edited

Loading

jakirkham commented Mar 24, 2020

gmarkall commented Mar 24, 2020

leofang commented Mar 24, 2020 •

edited

Loading

jakirkham commented Mar 24, 2020

leofang commented Mar 24, 2020 •

edited

Loading

jakirkham commented Mar 24, 2020

jakirkham commented Mar 24, 2020

gmarkall commented Mar 24, 2020

leofang commented Mar 24, 2020 •

edited

Loading

jakirkham commented Mar 24, 2020

jakirkham commented Mar 24, 2020

leofang commented Mar 24, 2020

leofang commented Mar 24, 2020

gmarkall commented Mar 24, 2020

leofang commented Apr 8, 2020

jakirkham commented Apr 8, 2020

leofang commented Apr 28, 2020

jjhelmus commented Apr 29, 2020

gmarkall commented Apr 29, 2020

jakirkham commented Apr 29, 2020

jjhelmus commented Apr 29, 2020

leofang commented Apr 30, 2020

leofang commented Jun 9, 2020

jakirkham commented Jun 11, 2020

Remove OSX support, add cudadevrt #5

Remove OSX support, add cudadevrt #5

Conversation

gmarkall commented Mar 24, 2020

gmarkall commented Mar 24, 2020

jakirkham commented Mar 24, 2020

leofang commented Mar 24, 2020

leofang commented Mar 24, 2020 • edited Loading

jakirkham commented Mar 24, 2020

gmarkall commented Mar 24, 2020

leofang commented Mar 24, 2020 • edited Loading

jakirkham commented Mar 24, 2020

leofang commented Mar 24, 2020 • edited Loading

jakirkham commented Mar 24, 2020

jakirkham commented Mar 24, 2020

gmarkall commented Mar 24, 2020

leofang commented Mar 24, 2020 • edited Loading

jakirkham commented Mar 24, 2020

jakirkham commented Mar 24, 2020

leofang commented Mar 24, 2020

leofang commented Mar 24, 2020

gmarkall commented Mar 24, 2020

leofang commented Apr 8, 2020

jakirkham commented Apr 8, 2020

leofang commented Apr 28, 2020

jjhelmus commented Apr 29, 2020

gmarkall commented Apr 29, 2020

jakirkham commented Apr 29, 2020

jjhelmus commented Apr 29, 2020

leofang commented Apr 30, 2020

leofang commented Jun 9, 2020

jakirkham commented Jun 11, 2020

leofang commented Mar 24, 2020 •

edited

Loading

leofang commented Mar 24, 2020 •

edited

Loading

leofang commented Mar 24, 2020 •

edited

Loading

leofang commented Mar 24, 2020 •

edited

Loading