Modularization of sagelib: Break out separate packages sagemath-environment, sagemath-repl

[mkoeppe]

The distribution sagemath-environment has no dependencies and contains:

• sage.env
• sage.features
• sage.misc.package
• sage.misc.package_dir (from Prepare sage.doctest for namespace packages #33033)
• sage.misc.temporary_file (dependency on SAGE_TMP/lazy_string removed in sage.misc.temporary_file: Remove use of SAGE_TMP #33797)
• sage.misc.viewer
  (The basic distributions sagemath-objects and sagemath-categories do not depend on these!)

It also includes the top-level scripts sage and sage-env, and the helper script sage-venv-config. This will allow users to run sage -sh even if sagelib is not installed.

To test:

$ ./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-environment && SAGE_NUM_THREADS=16 tox -r -v -v -v -e py39)'

The distribution sagemath-repl depends on (install-requires) sagemath-environment and sagemath-objects and contains at least the following:

• sage.repl
• sage.doctest (which depends on sage.repl)
• sage.misc.sage_input
• sage.misc.sage_eval

It also includes scripts such as sage-runtests (which implements sage -t), sage-ipython (which is called when sage is invoked with no arguments), etc.

Of course, with nothing else installed, neither sage -t nor sage will work fully because sage.all is not available. But sage -t already has the option --environment for specifying a different top-level module in which to run the doctester; we use this in the test suite of sagemath-objects and sagemath-polyhedra (#32432).

To test:

$ ./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-repl && SAGE_NUM_THREADS=16 tox -r -v -v -v -e py39)'

Afterwards:

$ source pkgs/sagemath-repl/.tox/py39/bin/activate
(py39) $ sage                                                                                        
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.6.rc3, Release Date: 2022-04-29                 │
│ Using Python 3.9.10. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┃ Warning: sage.all is not available; this is a limited REPL.        ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage:                                                                                                                                                                                                                       

Note the new warning about sage.all.

We also change all pkgs/*/VERSION.txt files from symlinks to regular files, updated by sage-update-version when a new release is made. This makes it easier to cut post-versions such as 9.6.rc3.post4 or dev versions such as 9.6.rc4.dev1, as was necessary during the development of this ticket. See documentation added in this ticket.

Follow-ups (see Meta-ticket #29705):

• We rework sagemath-objects, sagemath-categories as namespace packages (Modify find_python_sources to support modularization of sagelib by native namespace packages (PEP 420) #28925).
• Change sage-setup so that it install-requires sagemath-environment; then remove the load-path manipulations in setup.py. (Refactor distributions sagemath-{objects,categories} through sagemath-{environment,repl} #33812)
• We change the distributions sagemath-objects and sagemath-categories (from Modularization of sagelib: Break out separate packages sagemath-objects, sagemath-categories #29865) so that they no longer includes the above modules but instead declare an extras_require "test" on sagemath-repl. (Refactor distributions sagemath-{objects,categories} through sagemath-{environment,repl} #33812)
• Make sage -help, sage -advance show what to install to make options work. Also show what to install on error. (#33795 comment:7)
• Add to sagemath-environment: sage.misc.sage_ostools (need to split out Cython stuff), sage.misc.cython (?)
• Add an option to make sage_eval work with a smaller module than sage.all, and use this option in most uses of sage_eval (Meta-ticket: Remove runtime dependency on everything via sage_eval #33802)

Depends on #33797
Depends on #33016
Depends on #33799
Depends on #33801
Depends on #33803

CC: @kwankyu @tobiasdiez @kliem @orlitzky @dimpase @jhpalmieri @kiwifb

Component: refactoring

Keywords: sd111

Author: Matthias Koeppe

Branch/Commit: 3884373

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/29941

[mkoeppe]

comment:2

Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111

[mkoeppe]

Changed keywords from none to sd111

[mkoeppe]

comment:4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[mkoeppe]

Dependencies: #29865

[mkoeppe]

Changed dependencies from #29865 to #29865, #32899

[mkoeppe]

Branch: u/mkoeppe/modularization_of_sagelib__break_out_separate_packages_sagemath_environment__sagemath_repl

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

23ecefc

pkgs/sagemath-environment: Edit metadata

[sagetrac-git]

Commit: 23ecefc

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

c23ffa0

pkgs/sagemath-environment: Edit metadata

[sagetrac-git]

Changed commit from 23ecefc to c23ffa0

[sagetrac-git]

Changed commit from c23ffa0 to d2c3a93

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

d2c3a93

src/doc/en/developer/packaging_sage_library.rst: Explain build/pkgs

[mkoeppe]

Author: Matthias Koeppe, ...

[mkoeppe]

Changed dependencies from #29865, #32899 to #29865, #32899, #32977

[mkoeppe]

comment:131

trivial merge

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

0b74a68

Merge tag '9.6' into t/29941/modularization_of_sagelib__break_out_separate_packages_sagemath_environment__sagemath_repl

[sagetrac-git]

Changed commit from 15a2894 to 0b74a68

[sagetrac-git]

Changed commit from 0b74a68 to ffcbc79

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

ffcbc79

Merge tag '9.7.beta1' into t/29941/modularization_of_sagelib__break_out_separate_packages_sagemath_environment__sagemath_repl

[vbraun]

comment:136

Not the only fail but e.g.:

**********************************************************************
File "src/sage/misc/lazy_attribute.pyx", line 89, in sage.misc.lazy_attribute._lazy_attribute._sage_src_lines_
Failed example:
    lines
Expected:
    81
Got:
    89
**********************************************************************
1 item had failures:
   1 of   6 in sage.misc.lazy_attribute._lazy_attribute._sage_src_lines_
    [100 tests, 1 failure, 2.55 s]
----------------------------------------------------------------------
sage -t --long --warn-long 46.9 --random-seed=123 src/sage/misc/lazy_attribute.pyx  # 1 doctest failed
----------------------------------------------------------------------

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

a5da84a	src/sage/misc/{abstract_method.py,lazy_attribute.pyx}: Update doctest output
3884373	src/sage/tests/cmdline.py: Update doctest for changed help message

[sagetrac-git]

Changed commit from ffcbc79 to 3884373

[mkoeppe]

comment:139

The failure in sage/doctest/forker.py is unrelated, fixed in #33917.

[dimpase]

comment:140

OK

[vbraun]

Changed branch from u/mkoeppe/modularization_of_sagelib__break_out_separate_packages_sagemath_environment__sagemath_repl to 3884373