Drop support for RedisAI 1.2.5 (#383)

Drops support for RedisAI 1.2.5. The only RedisAI version is now 1.2.7. Since the officially released RedisAI 1.2.7 has a bug which breaks the build process on Mac OSX, it was decided to use commit RedisAI/redis-inference-optimization@634916c from RedisAI's GitHub repository, where such bug has been fixed. This applies to all operating systems. [ committed by @al-rigazzi ] [ reviewed by @MattToast @mellis13 ]
CrayLabs · Oct 4, 2023 · aaeee3b · aaeee3b
1 parent 933e7bd
commit aaeee3b
Show file tree

Hide file tree

Showing 11 changed files with 71 additions and 456 deletions.
diff --git a/.github/workflows/run_tests.yml b/.github/workflows/run_tests.yml
@@ -51,16 +51,9 @@ jobs:
       matrix:
         os: [macos-12, ubuntu-20.04] # Operating systems
         compiler: [8] # GNU compiler version
-        rai: [1.2.5, 1.2.7] # Redis AI versions
+        rai: [1.2.7] # Redis AI versions
         py_v: [3.8, 3.9, '3.10'] # Python versions
-        exclude:
-          # Do not build with Redis AI 1.2.5 on MacOS
-          - os: macos-12
-            rai: 1.2.5
-          # Do not build Redis AI 1.2.5 with py3.10
-          # as wheels for dependecies are not availble
-          - py_v: '3.10'
-            rai: 1.2.5
+
 
     env:
       SMARTSIM_REDISAI: ${{ matrix.rai }}
@@ -86,7 +79,7 @@ jobs:
           echo "$(brew --prefix)/opt/make/libexec/gnubin" >> $GITHUB_PATH
 
       - name: Build Singularity from source
-        if: contains( matrix.os, 'ubuntu' ) && matrix.py_v == 3.9 && matrix.rai == '1.2.5'
+        if: contains( matrix.os, 'ubuntu' ) && matrix.py_v == 3.9
         run: |
           sudo apt-get install -y libseccomp-dev pkg-config squashfs-tools cryptsetup curl git # wget build-essential
           echo 'export PATH=/usr/local/go/bin:$PATH' >> ~/.bashrc
@@ -100,7 +93,7 @@ jobs:
           sudo make -C builddir install
 
       - name: singularity pull test container # This lets us time how long the pull takes
-        if: contains( matrix.os, 'ubuntu' ) && matrix.py_v == 3.9 && matrix.rai == '1.2.5'
+        if: contains( matrix.os, 'ubuntu' ) && matrix.py_v == 3.9
         run: singularity pull docker://alrigazzi/smartsim-testing
 
       # Note: The develop branch of smartredis is installed first to ensure that any tests that depend
@@ -119,8 +112,6 @@ jobs:
         run: smart build --device cpu -v
 
       - name: Run mypy
-        # TF 2.6.2 has a dep conflict with new mypy versions
-        if: (matrix.rai != '1.2.5')
         run: |
           python -m pip install .[mypy]
           make check-mypy

diff --git a/README.md b/README.md
@@ -649,18 +649,6 @@ from C, C++, Fortran and Python with the SmartRedis Clients:
       <td>ONNX</td>
       <td>1.11.x</td>
     </tr>
-      <td rowspan="3">1.2.5</td>
-      <td>PyTorch</td>
-      <td>1.9.x</td>
-    </tr>
-    <tr>
-      <td>TensorFlow\Keras</td>
-      <td>2.6.x</td>
-    </tr>
-    <tr>
-      <td>ONNX</td>
-      <td>1.9.x</td>
-    </tr>
   </tbody>
 </table>
 

diff --git a/doc/changelog.rst b/doc/changelog.rst
@@ -12,6 +12,26 @@ SmartSim
 ========
 
 
+Development branch
+------------------
+
+To be released at some future point in time
+
+Description
+
+- Remove support for RedisAI 1.2.5, use RedisAI 1.2.7 commit
+
+Detailed Notes
+
+- RedisAI 1.2.5 is not supported anymore. The only RedisAI version
+  is now 1.2.7. Since the officially released RedisAI 1.2.7 has a
+  bug which breaks the build process on Mac OSX, it was decided to
+  use commit 634916c_ from RedisAI's GitHub repository, where such
+  bug has been fixed. This applies to all operating systems. (PR383_)
+
+  .. _PR383: https://github.com/CrayLabs/SmartSim/pull/383
+  .. _634916c: https://github.com/RedisAI/RedisAI/commit/634916c722e718cc6ea3fad46e63f7d798f9adc2
+
 0.5.1
 -----
 

diff --git a/doc/installation_instructions/basic.rst b/doc/installation_instructions/basic.rst
@@ -80,17 +80,13 @@ Supported Versions
 
 
 Native support for various machine learning libraries and their
-versions is dictated by our dependency on RedisAI_ 1.2.7. Users
-can also select RedisAI 1.2.5 (though that also limits
-the version of the ML libraries).
+versions is dictated by our dependency on RedisAI_ 1.2.7.
 
 +------------------+----------+-------------+---------------+
 | RedisAI          | PyTorch  | Tensorflow  | ONNX Runtime  |
 +==================+==========+=============+===============+
 | 1.2.7 (default)  | 1.11.0   | 2.8.0       | 1.11.1        |
 +------------------+----------+-------------+---------------+
-| 1.2.5            | 1.9.0    | 2.6.0       | 1.9.0         |
-+------------------+----------+-------------+---------------+
 
 TensorFlow_ 2.0 and Keras_ are supported through `graph freezing`_.
 
@@ -304,5 +300,3 @@ Build the SmartRedis library
 ============================
 
 .. include:: ../../smartredis/doc/install/lib.rst
-
-
diff --git a/doc/installation_instructions/platform/olcf-summit.rst b/doc/installation_instructions/platform/olcf-summit.rst
@@ -11,23 +11,28 @@ instructions to get a working SmartSim build with PyTorch and TensorFlow for GPU
 on Summit.  Note that SmartSim and SmartRedis will be downloaded to the working
 directory from which these instructions are executed.
 
+Note that the available PyTorch version (1.10.2) does not match
+the one expected by RedisAI 1.2.7 (1.11): it is still compatible and should
+work, but please open an issue on SmartSim's GitHub repo if you run
+into problems.
+
 .. code-block:: bash
 
   # setup Python and build environment
   export ENV_NAME=smartsim-0.5.1
   git clone https://github.com/CrayLabs/SmartRedis.git smartredis
   git clone https://github.com/CrayLabs/SmartSim.git smartsim
-  conda config --prepend channels https://ftp.osuosl.org/pub/open-ce/1.4.1/
+  conda config --prepend channels https://ftp.osuosl.org/pub/open-ce/1.6.1/
   conda create --name $ENV_NAME -y  python=3.9 \
                                     git-lfs \
                                     cmake \
                                     make \
                                     cudnn=8.1.1_11.2 \
                                     cudatoolkit=11.2.2 \
-                                    tensorflow=2.6.2 \
-                                    libtensorflow=2.6.2 \
-                                    pytorch=1.9.0 \
-                                    torchvision=0.10.0
+                                    tensorflow=2.8.1 \
+                                    libtensorflow \
+                                    pytorch=1.10.2 \
+                                    torchvision=0.11.3
   conda activate $ENV_NAME
   export CC=$(which gcc)
   export CXX=$(which g++)
@@ -50,7 +55,7 @@ directory from which these instructions are executed.
   # install PyTorch and TensorFlow backend for the Orchestrator database.
   export Torch_DIR=/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/python3.9/site-packages/torch/share/cmake/Torch/
   export CFLAGS="$CFLAGS -I/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/python3.9/site-packages/tensorflow/include"
-  export SMARTSIM_REDISAI=1.2.5
+  export SMARTSIM_REDISAI=1.2.7
   export Tensorflow_BUILD_DIR=/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/python3.9/site-packages/tensorflow/
   smart build --device=gpu --torch_dir $Torch_DIR --libtensorflow_dir $Tensorflow_BUILD_DIR -v
 

diff --git a/smartsim/_core/_cli/build.py b/smartsim/_core/_cli/build.py
@@ -65,11 +65,8 @@ def check_py_onnx_version(versions: Versioner) -> None:
         msg = (
             "An onnx wheel is not available for "
             f"Python {py_version.major}.{py_version.minor}. "
-            "Instead consider using Python 3.8 or 3.9 with Onnx "
+            "Instead consider using Python 3.8 or 3.9 for ONNX 1.11 support"
         )
-        if sys.platform == "linux":
-            msg += "1.2.5 or "
-        msg += "1.2.7."
         raise SetupError(msg)
     _check_packages_in_python_env(
         {

diff --git a/smartsim/_core/_install/buildenv.py b/smartsim/_core/_install/buildenv.py
@@ -189,17 +189,6 @@ class RedisAIVersion(Version_):
     """
 
     defaults = {
-        "1.2.5": {
-            "tensorflow": "2.6.2",
-            "onnx": "1.9.0",
-            "skl2onnx": "1.10.3",
-            "onnxmltools": "1.10.0",
-            "scikit-learn": "1.0.2",
-            "torch": "1.9.1",
-            "torch_cpu_suffix": "+cpu",
-            "torch_cuda_suffix": "+cu111",
-            "torchvision": "0.10.1",
-        },
         "1.2.7": {
             "tensorflow": "2.8.0",
             "onnx": "1.11.0",
@@ -214,14 +203,10 @@ class RedisAIVersion(Version_):
     }
     # Remove options with unsported wheels for python>=3.10
     if sys.version_info >= (3, 10):
-        defaults.pop("1.2.5")
         defaults["1.2.7"].pop("onnx")
         defaults["1.2.7"].pop("skl2onnx")
         defaults["1.2.7"].pop("onnxmltools")
         defaults["1.2.7"].pop("scikit-learn")
-    # Remove incompatible RAI versions for OSX
-    if sys.platform == "darwin":
-        defaults.pop("1.2.5", None)
 
     def __init__(self, vers: str) -> None:  # pylint: disable=super-init-not-called
         min_rai_version = min(Version_(ver) for ver in self.defaults)

diff --git a/smartsim/_core/_install/builder.py b/smartsim/_core/_install/builder.py
@@ -351,21 +351,6 @@ def build_onnx(self) -> bool:
     def fetch_onnx(self) -> bool:
         return self.build_onnx
 
-    def copy_tf_cmake(self) -> None:
-        """Copy the FindTensorFlow.cmake file to the build directory
-        as the version included in RedisAI is out of date for us.
-        Note: opt/cmake/modules removed in RedisAI v1.2.5
-        """
-        # remove the previous version
-        tf_cmake = self.rai_build_path / "opt/cmake/modules/FindTensorFlow.cmake"
-        tf_cmake.resolve()
-        if tf_cmake.is_file():
-            tf_cmake.unlink()
-            # copy ours in
-            self.copy_file(
-                self.bin_path / "modules/FindTensorFlow.cmake", tf_cmake, set_exe=False
-            )
-
     def symlink_libtf(self, device: str) -> None:
         """Add symbolic link to available libtensorflow in RedisAI deps.
 
@@ -453,7 +438,7 @@ def build_from_git(self, git_url: str, branch: str, device: str = "cpu") -> None
 
         # Circumvent a bad `get_deps.sh` script from RAI on 1.2.7 with ONNX
         # TODO: Look for a better way to do this or wait for RAI patch
-        if sys.platform == "darwin" and branch == "v1.2.7" and self.build_onnx:
+        if branch == "v1.2.7":
             # Clone RAI patch commit for OSX
             clone_cmd += ["RedisAI"]
             checkout_osx_fix = [
@@ -462,7 +447,7 @@ def build_from_git(self, git_url: str, branch: str, device: str = "cpu") -> None
                 "634916c722e718cc6ea3fad46e63f7d798f9adc2",
             ]
         else:
-            # Clone RAI release commit
+            # Clone RAI release commit for versions > 1.2.7
             clone_cmd += [
                 "--branch",
                 branch,
@@ -476,9 +461,6 @@ def build_from_git(self, git_url: str, branch: str, device: str = "cpu") -> None
                 checkout_osx_fix, out=subprocess.DEVNULL, cwd=self.rai_build_path
             )
 
-        # copy FindTensorFlow.cmake to RAI cmake dir
-        self.copy_tf_cmake()
-
         # get RedisAI dependencies
         dep_cmd = self._rai_build_env_prefix(
             with_pt=self.build_torch,