Trac #33740: Add conda dev environment

We add a conda enviroment that is meant for development, i.e. includes tools that one probably needs while working on sage development such as linters, ssh (for contributing to trac) etc. Moreover, we extract the conda-related things from `docs/bootstrap` to a new `bootstrap-conda` script in order to simplify the conda workflow (see updated documentation). Github workflow run: https://github.com/sagemath/sagetrac- mirror/actions/runs/2268985510 URL: https://trac.sagemath.org/33740 Reported by: gh-tobiasdiez Ticket author(s): Tobias Diez Reviewer(s): Matthias Koeppe
sagemath · May 22, 2022 · 773ec37 · 773ec37
2 parents 5e65c16 + 87f360c
commit 773ec37
Show file tree

Hide file tree

Showing 9 changed files with 126 additions and 83 deletions.
diff --git a/.github/workflows/ci-conda.yml b/.github/workflows/ci-conda.yml
@@ -43,15 +43,8 @@ jobs:
           bash ~/miniconda.sh -b -p $HOME/miniconda
           echo "CONDA=$HOME/miniconda" >> $GITHUB_ENV
 
-      - name: Install bootstrap prerequisites
-        run: |
-          export PATH="$(pwd)/build/bin:$PATH"
-          SYSTEM=$(sage-guess-package-system)
-          eval $(sage-print-system-package-command $SYSTEM --sudo install $(sage-get-system-packages $SYSTEM _bootstrap))
-
-      # Create conda environment file
-      - name: Bootstrap
-        run: ./bootstrap
+      - name: Create conda environment files
+        run: ./bootstrap-conda
 
       - name: Cache conda packages
         uses: actions/cache@v2
@@ -80,6 +73,7 @@ jobs:
         shell: bash -l {0}
         continue-on-error: true
         run: |
+          ./bootstrap
           echo "::add-matcher::.github/workflows/configure-systempackage-problem-matcher.json"
           ./configure --enable-build-as-root --with-python=$CONDA_PREFIX/bin/python --prefix=$CONDA_PREFIX $(for pkg in $(./sage -package list :standard: --has-file spkg-configure.m4 --has-file distros/conda.txt); do echo --with-system-$pkg=force; done)
           echo "::remove-matcher owner=configure-system-package-warning::"
@@ -88,8 +82,8 @@ jobs:
       - name: Build
         shell: bash -l {0}
         run: |
-          pip install --no-build-isolation -v -v -e pkgs/sage-conf pkgs/sage-setup 
-          pip install --no-build-isolation -v -v -e src
+          pip install --no-build-isolation -v -v -e ./pkgs/sage-conf ./pkgs/sage-setup
+          pip install --no-build-isolation -v -v -e ./src
         env:
           SAGE_NUM_THREADS: 2
 

diff --git a/.gitignore b/.gitignore
@@ -23,6 +23,7 @@
 /environment.yml
 /environment-optional.yml
 /src/environment.yml
+/src/environment-dev.yml
 /src/environment-optional.yml
 
 /src/setup.cfg

diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -21,4 +21,8 @@
         "--doctest-modules",
     ],
     "python.testing.unittestEnabled": false,
+    "cSpell.words": [
+        "Conda",
+        "Cython"
+    ],
 }
diff --git a/Makefile b/Makefile
@@ -171,6 +171,7 @@ bootstrap-clean:
 	rm -f src/doc/en/reference/repl/*.txt
 	rm -f environment.yml
 	rm -f src/environment.yml
+	rm -f src/environment-dev.yml
 	rm -f environment-optional.yml
 	rm -f src/environment-optional.yml
 	rm -f src/Pipfile

diff --git a/bootstrap b/bootstrap
@@ -158,6 +158,7 @@ SAGE_SPKG_FINALIZE([$pkgname], [$pkgtype], [$SPKG_SOURCE], [$SPKG_TREE_VAR])"
     # ONLY stderr, and to re-output the results back to stderr leaving
     # stdout alone. Basically we swap the two descriptors using a
     # third, filter, and then swap them back.
+    ./bootstrap-conda && \
     src/doc/bootstrap && \
     install_config_rpath && \
     aclocal -I m4 && \

diff --git a/bootstrap-conda b/bootstrap-conda
@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+
+########################################################################
+# Generate auto-generated conda environment files
+#########################################################################
+
+STRIP_COMMENTS="sed s/#.*//;"
+RECOMMENDED_SPKG_PATTERN="@(_recommended$(for a in $(head -n 1 build/pkgs/_recommended/dependencies); do echo -n "|"$a; done))"
+
+BOOTSTRAP_PACKAGES=$(echo $(${STRIP_COMMENTS} build/pkgs/_bootstrap/distros/conda.txt))
+SYSTEM_PACKAGES=
+OPTIONAL_SYSTEM_PACKAGES=
+SAGELIB_SYSTEM_PACKAGES=
+SAGELIB_OPTIONAL_SYSTEM_PACKAGES=
+RECOMMENDED_SYSTEM_PACKAGES=
+for PKG_BASE in $(./sage --package list --has-file distros/conda.txt); do
+    PKG_SCRIPTS=build/pkgs/$PKG_BASE
+    SYSTEM_PACKAGES_FILE=$PKG_SCRIPTS/distros/conda.txt
+    PKG_TYPE=$(cat $PKG_SCRIPTS/type)
+    PKG_SYSTEM_PACKAGES=$(echo $(${STRIP_COMMENTS} $SYSTEM_PACKAGES_FILE))
+    if [ -n "PKG_SYSTEM_PACKAGES" ]; then
+        if [ -f $PKG_SCRIPTS/spkg-configure.m4 ]; then
+            case "$PKG_BASE:$PKG_TYPE" in
+                *:standard)
+                    SYSTEM_PACKAGES+=" $PKG_SYSTEM_PACKAGES"
+                    ;;
+                $RECOMMENDED_SPKG_PATTERN:*)
+                    RECOMMENDED_SYSTEM_PACKAGES+=" $PKG_SYSTEM_PACKAGES"
+                    ;;
+                *)
+                    OPTIONAL_SYSTEM_PACKAGES+=" $PKG_SYSTEM_PACKAGES"
+                    ;;
+            esac
+        else
+            case "$PKG_TYPE" in
+                standard)
+                    SAGELIB_SYSTEM_PACKAGES+=" $PKG_SYSTEM_PACKAGES"
+                    ;;
+                *)
+                    SAGELIB_OPTIONAL_SYSTEM_PACKAGES+=" $PKG_SYSTEM_PACKAGES"
+                    ;;
+            esac
+        fi
+    fi
+done
+echo >&2 $0:$LINENO: generate conda enviroment files
+echo "name: sage-build" >  environment.yml
+echo "channels:"        >> environment.yml
+echo "  - conda-forge"  >> environment.yml
+echo "  - nodefaults"   >> environment.yml
+echo "dependencies:"    >> environment.yml
+for pkg in $SYSTEM_PACKAGES; do
+    echo "  - $pkg" >> environment.yml
+done
+echo "  # Packages needed for ./bootstrap" >> environment.yml
+for pkg in $BOOTSTRAP_PACKAGES; do
+    echo "  - $pkg" >> environment.yml
+done
+sed 's/name: sage-build/name: sage/' environment.yml > src/environment.yml
+for pkg in $SAGELIB_SYSTEM_PACKAGES; do
+    echo "  - $pkg" >> src/environment.yml
+done
+sed 's/name: sage/name: sage-dev/' src/environment.yml > src/environment-dev.yml
+echo "  # Additional dev tools" >> src/environment-dev.yml
+echo "  - openssh" >> src/environment-dev.yml
+echo "  - pycodestyle" >> src/environment-dev.yml
+echo "  - pytest" >> src/environment-dev.yml
+
+cp environment.yml environment-optional.yml
+    echo "  # optional packages" >> environment-optional.yml
+for pkg in $OPTIONAL_SYSTEM_PACKAGES; do
+    echo "  - $pkg" >> environment-optional.yml
+    done
+cp src/environment.yml src/environment-optional.yml
+    echo "  # optional packages" >> src/environment-optional.yml
+for pkg in $OPTIONAL_SYSTEM_PACKAGES $SAGELIB_OPTIONAL_SYSTEM_PACKAGES; do
+    echo "  - $pkg" >> src/environment-optional.yml
+done
diff --git a/build/pkgs/jupyter_sphinx/distros/conda.txt b/build/pkgs/jupyter_sphinx/distros/conda.txt
@@ -0,0 +1 @@
+jupyter_sphinx
diff --git a/src/doc/bootstrap b/src/doc/bootstrap
@@ -27,7 +27,7 @@ shopt -s extglob
 
 RECOMMENDED_SPKG_PATTERN="@(_recommended$(for a in $(head -n 1 build/pkgs/_recommended/dependencies); do echo -n "|"$a; done))"
 
-for SYSTEM in arch debian fedora cygwin homebrew conda; do
+for SYSTEM in arch debian fedora cygwin homebrew; do
     SYSTEM_PACKAGES=
     OPTIONAL_SYSTEM_PACKAGES=
     SAGELIB_SYSTEM_PACKAGES=
@@ -63,40 +63,13 @@ for SYSTEM in arch debian fedora cygwin homebrew conda; do
                     fi
                 fi
     done
-    if [ "${SYSTEM}" = "conda" ]; then
-        if [ "${BOOTSTRAP_QUIET}" = "no" ]; then
-            echo >&2 $0:$LINENO: installing environment.yml, environment-optional.yml, src/environment.yml and src/environment-optional.yml
-        fi
-        echo "name: sage-build" >  environment.yml
-        echo "channels:"        >> environment.yml
-        echo "  - conda-forge"  >> environment.yml
-        echo "  - nodefaults"   >> environment.yml
-        echo "dependencies:"    >> environment.yml
-        for pkg in $SYSTEM_PACKAGES; do
-            echo "  - $pkg" >> environment.yml
-        done
-        sed 's/name: sage-build/name: sage/' environment.yml > src/environment.yml
-        for pkg in $SAGELIB_SYSTEM_PACKAGES; do
-            echo "  - $pkg" >> src/environment.yml
-        done
-        cp environment.yml environment-optional.yml
-            echo "  # optional packages" >> environment-optional.yml
-        for pkg in $OPTIONAL_SYSTEM_PACKAGES; do
-            echo "  - $pkg" >> environment-optional.yml
-            done
-        cp src/environment.yml src/environment-optional.yml
-            echo "  # optional packages" >> src/environment-optional.yml
-        for pkg in $OPTIONAL_SYSTEM_PACKAGES $SAGELIB_OPTIONAL_SYSTEM_PACKAGES; do
-            echo "  - $pkg" >> src/environment-optional.yml
-        done
-    else
-        if [ "${BOOTSTRAP_QUIET}" = "no" ]; then
-            echo >&2 $0:$LINENO: installing "$OUTPUT_DIR"/$SYSTEM"*.txt"
-        fi
-        echo "$(sage-print-system-package-command $SYSTEM --prompt --sudo install $(echo $(echo $SYSTEM_PACKAGES | xargs -n 1 echo | sort | uniq)))" > "$OUTPUT_DIR"/$SYSTEM.txt
-        echo "$(sage-print-system-package-command $SYSTEM --prompt --sudo install $(echo $(echo $OPTIONAL_SYSTEM_PACKAGES | xargs -n 1 echo | sort | uniq)))" > "$OUTPUT_DIR"/$SYSTEM-optional.txt
-        echo "$(sage-print-system-package-command $SYSTEM --prompt --sudo install $(echo $(echo $RECOMMENDED_SYSTEM_PACKAGES | xargs -n 1 echo | sort | uniq)))" > "$OUTPUT_DIR"/$SYSTEM-recommended.txt
+
+    if [ "${BOOTSTRAP_QUIET}" = "no" ]; then
+        echo >&2 $0:$LINENO: installing "$OUTPUT_DIR"/$SYSTEM"*.txt"
     fi
+    echo "$(sage-print-system-package-command $SYSTEM --prompt --sudo install $(echo $(echo $SYSTEM_PACKAGES | xargs -n 1 echo | sort | uniq)))" > "$OUTPUT_DIR"/$SYSTEM.txt
+    echo "$(sage-print-system-package-command $SYSTEM --prompt --sudo install $(echo $(echo $OPTIONAL_SYSTEM_PACKAGES | xargs -n 1 echo | sort | uniq)))" > "$OUTPUT_DIR"/$SYSTEM-optional.txt
+    echo "$(sage-print-system-package-command $SYSTEM --prompt --sudo install $(echo $(echo $RECOMMENDED_SYSTEM_PACKAGES | xargs -n 1 echo | sort | uniq)))" > "$OUTPUT_DIR"/$SYSTEM-recommended.txt
 done
 
 OUTPUT_DIR="src/doc/en/reference/spkg"

diff --git a/src/doc/en/installation/conda.rst b/src/doc/en/installation/conda.rst
@@ -65,25 +65,22 @@ conda-forge/sage-feedstock <https://github.com/conda-forge/sage-feedstock/issues
 Using conda to provide system packages for the Sage distribution
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-If Conda is installed (check by typing ``conda info``), there are two ways to
-prepare for installing SageMath from source:
+If Conda is installed (check by typing ``conda info``), one can install SageMath
+from source as follows:
 
   - If you are using a git checkout::
 
-      $ ./bootstrap
+      $ ./bootstrap-conda
 
-  - Create a new empty environment and activate::
+  - Create a new conda environment including all standard packages
+    recognized by sage, and activate it::
 
-      $ conda create -n sage-build
+      $ conda env create --file environment.yml --name sage-build
       $ conda activate sage-build
 
-  - Install standard packages recognized by sage's ``spkg-configure`` mechanism::
-
-      $ conda env update --file environment.yml -n sage-build
-
-  - Or install all standard and optional packages recognized by sage::
-
-      $ conda env update --file environment-optional.yml -n sage-build
+    Alternatively, use ``environment-optional.yml`` in place of
+    ``environment.yml`` to create an environment with all standard and optional
+    packages recognized by sage.
 
   - Then the SageMath distribution will be built using the compilers provided by Conda
     and using many packages installed by Conda::
@@ -114,38 +111,34 @@ Here we assume that you are using a git checkout.
 
       $ export SAGE_NUM_THREADS=24
 
-  - As a recommended step, install the ``mamba`` package manager.  If
+  - As a recommended step, install the ``mamba`` package manager. If
     you skip this step, replace ``mamba`` by ``conda`` in the
     following steps::
 
       $ conda install mamba
 
-  - Create and activate a new conda environment that provides the
-    bootstrapping prerequisites. You can replace 3.9 by another Python
-    version::
-
-      $ mamba create -n sage-build python=3.9 \
-            gettext autoconf automake libtool pkg-config
-      $ conda activate sage-build
-
-  - Run ``bootstrap``; this generates the files ``src/environment*.yml`` used
+  - Generate the conda environment files ``src/environment*.yml`` used
     in the next step::
 
-      $ ./bootstrap
-
-  - Populate the conda environment with the dependencies of Sage::
+      $ ./bootstrap-conda
 
-      $ mamba env update -n sage-build -f src/environment.yml  # alternatively, use
+  - Create and activate a new conda environment with the dependencies of Sage
+    and a few additional developer tools::
 
-    Alternatively, you can use ``src/environment-optional.yml``, which will
-    install some additional packages.
+      $ mamba env create --file src/environment-dev.yml --name sage-dev
+      $ conda activate sage-dev
 
-  - Activate the conda environment again::
+    Alternatively, you can use ``src/environment.yml`` or
+    ``src/environment-optional.yml``, which will only install standard
+    (and optional) packages without any additional developer tools.
 
-      $ conda activate sage-build
+    By default, the most recent version of Python supported by Sage is
+    installed. You can use the additional option ``python=3.9`` in the above
+    ``env create`` command to select another Python version (here 3.9). 
 
   - Run the ``configure`` script::
 
+      $ ./bootstrap
       $ ./configure --with-python=$CONDA_PREFIX/bin/python             \
                     --prefix=$CONDA_PREFIX                             \
                     $(for pkg in $(./sage -package list :standard:     \
@@ -154,20 +147,17 @@ Here we assume that you are using a git checkout.
                           echo --with-system-$pkg=force;               \
                       done)
 
-  - Install the build prerequisites of the Sage library::
-
-      $ pip install --no-build-isolation -v -v --editable pkgs/sage-conf pkgs/sage-setup
-
-  - Install the Sage library::
+  - Install the build prerequisites and the Sage library::
 
-      $ pip install --no-build-isolation -v -v --editable src
+      $ pip install --no-build-isolation -v -v --editable ./pkgs/sage-conf ./pkgs/sage-setup
+      $ pip install --no-build-isolation -v -v --editable ./src
 
   - Verify that Sage has been installed::
 
       $ sage -c 'print(version())'
       SageMath version 9.6.beta5, Release Date: 2022-03-12
 
-Note that ``make`` is not used at all.  All dependencies
+Note that ``make`` is not used at all. All dependencies
 (including all Python packages) are provided by conda.
 
 Thus, you will get a working version of Sage much faster.  However,
@@ -179,6 +169,6 @@ library is installed in editable mode.  This means that when you only
 edit Python files, there is no need to rebuild the library; it
 suffices to restart Sage.
 
-After editing any Cython files, rebuild by repeating the command::
+After editing any Cython files, rebuild the Sage library using::
 
   $ pip install --no-build-isolation -v -v --editable src