Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tox.ini: Add centos-6, docker-nobootstrap, manylinux-* #30195

Closed
mkoeppe opened this issue Jul 22, 2020 · 35 comments
Closed

tox.ini: Add centos-6, docker-nobootstrap, manylinux-* #30195

mkoeppe opened this issue Jul 22, 2020 · 35 comments
Labels
c: porting p: major / 3 t: enhancement
Milestone
sage-9.2

Comments

@mkoeppe
Copy link
Contributor

mkoeppe commented Jul 22, 2020

The docker-nobootstrap option uses bootstrap -D; thus, autotools are not required on the host for the build.

This enables building with centos-6 and the manylinux images based on centos-6 (https://github.com/pypa/manylinux).

Examples:

tox -e docker-manylinux-2014-standard
tox -e docker-manylinux-2014-standard-python3.9
tox -e docker-manylinux-2010-standard-i686-python3.8

Depends on #29929

CC: @kliem @slel

Component: porting

Author: Matthias Koeppe

Branch/Commit: 14da64d

Reviewer: Jonathan Kliem

Issue created by migration from https://trac.sagemath.org/ticket/30195
@mkoeppe mkoeppe added this to the sage-9.2 milestone Jul 22, 2020
@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 22, 2020

Dependencies: #29929

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 22, 2020

Author: Matthias Koeppe

@mkoeppe mkoeppe changed the title tox.ini: Add manylinux tox.ini: Add centos-6, docker-nobootstrap, manylinux-* Jul 22, 2020
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 22, 2020

Branch: u/mkoeppe/tox_ini__add_manylinux

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 22, 2020

Commit: 585bdac

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 22, 2020

comment:3

Branch is 1 commit on top of #29929, which also needs review

Last 10 new commits:

25393b0Handle SAGE_CHECK_PACKAGES in build/make/Makefile.in, not sage-spkg
5aca238Merge branch 't/30118/handle_sage_check_packages_in_build_make_makefile_in__not_sage_spkg' into t/29929/tox_ini__add_a_macos_environment_without_homebrew__conda
a17c655m4/sage_spkg_collect.m4: Do not include script packages in SAGE_SDIST_PACKAGES
827923abootstrap: Accept 2020s in config.guess version check
c30ac54tox.ini: Add local-nobootstrap
de6464d.github/workflows/tox.yml: Add jobs "dist", "local-macos-nohomebrew
775b7f7Trac #30088: add a few explanatory comments to m4/sage_spkg_collect.m4.
160862fm4/sage_spkg_collect.m4: Mention sagelib in the comment
d1a2cbfMerge branch 't/30088/fix__make_sdist___add_test_run_to_github_actions' into t/29929/tox_ini__add_a_macos_environment_without_homebrew__conda
585bdactox.ini: Add centos-6, docker-nobootstrap, manylinux-*

@kliem
Copy link
Contributor

kliem commented Jul 22, 2020

comment:4

Can you provide an example of this at work?

According to the readme, only CPython up to 3.8 are provided with manylinux. Why have you configured python3.9?

@kliem
Copy link
Contributor

kliem commented Jul 22, 2020

Reviewer: Jonathan Kliem

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 22, 2020

comment:5

The readme does not seem to be up to date. On an actual docker image:

[root@4185c8d9310b /]# /opt/python/cp3 TAB
cp35-cp35m/ cp36-cp36m/ cp37-cp37m/ cp38-cp38/  cp39-cp39/

@kliem
Copy link
Contributor

kliem commented Jul 22, 2020

comment:6

That was one of my guesses.

Do the new environments work with github actions? If you haven't tested it yet, I have free capacities and can just test this ticket.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 22, 2020

comment:7

They should work, just add them to tox.yml for testing. I haven't tested them widely yet, only one run of docker-manylinux-2014-standard on my computer. Best to test together with #30173

@kliem
Copy link
Contributor

kliem commented Jul 22, 2020

comment:8
 File "/usr/lib/python3/dist-packages/tox/session.py", line 38, in main
    config = prepare(args)
  File "/usr/lib/python3/dist-packages/tox/session.py", line 26, in prepare
    config = parseconfig(args)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 239, in parseconfig
    parseini(config, inipath)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 760, in __init__
    self.make_envconfig(name, section, reader._subs, config)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 791, in make_envconfig
    res = meth(env_attr.name, env_attr.default)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 964, in getargvlist
    return _ArgvlistReader.getargvlist(self, s)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1134, in getargvlist
    commands.append(cls.processcommand(reader, current_command))
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1164, in processcommand
    new_word = reader._replace(word)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1008, in _replace
    return Replacer(self, crossonly=crossonly).do_replace(value)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1028, in do_replace
    return self.RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1057, in _replace_match
    return self._replace_env(match)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1077, in _replace_env
    (envkey, envkey))
tox.ConfigError: ConfigError: substitution env:'BOOTSTRAP': unknown environment variable 'BOOTSTRAP'  or recursive definition.
##[error]Process completed with exit code 1.

https://github.com/kliem/sage/runs/897566509?check_suite_focus=true

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jul 22, 2020

Changed commit from 585bdac to 756f310

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jul 22, 2020

Branch pushed to git repo; I updated commit sha1. New commits:

756f310tox.ini: Initialize the BOOTSTRAP variable

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 22, 2020

comment:10

Thanks for catching this

@kliem
Copy link
Contributor

kliem commented Jul 23, 2020

comment:11

Centos 6 seems to work in the way that it fails to buil mpir.

There are many errors with manylinux like

   File "/usr/lib/python3/dist-packages/tox/config.py", line 1008, in _replace
    return Replacer(self, crossonly=crossonly).do_replace(value)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1028, in do_replace
    return self.RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1057, in _replace_match
    return self._replace_env(match)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1071, in _replace_env
    envvalue = self.reader.get_environ_value(envkey)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 903, in get_environ_value
    return self._setenv.get(name)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 286, in get
    self.resolved[name] = res = self.reader._replace(val)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1008, in _replace
    return Replacer(self, crossonly=crossonly).do_replace(value)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1028, in do_replace
    return self.RE_ITEM_REF.sub(self._replace_match, x)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1057, in _replace_match
    return self._replace_env(match)
  File "/usr/lib/python3/dist-packages/tox/config.py", line 1077, in _replace_env
    (envkey, envkey))
tox.ConfigError: ConfigError: substitution env:'BASE_IMAGE': unknown environment variable 'BASE_IMAGE'  or recursive definition.

https://github.com/kliem/sage/runs/899346008?check_suite_focus=true

and

 Step 14/18 : RUN ${BOOTSTRAP}
 ---> Running in d5c9ccb6ceb3
rm -rf config configure build/make/Makefile-auto.in
rm -f src/doc/en/installation/*.txt
rm -rf src/doc/en/reference/spkg/*.rst
rm -f src/doc/en/reference/repl/*.txt
Traceback (most recent call last):
  File "/sage/build/bin/sage-download-file", line 27, in ?
    from sage_bootstrap.download.cmdline import run_safe
  File "/sage/build/bin/../sage_bootstrap/download/__init__.py", line 7, in ?
    from sage_bootstrap.download.transfer import Download
  File "/sage/build/bin/../sage_bootstrap/download/transfer.py", line 43
    n = 0 if chunks_so_far == 0 else self.length // 2
           ^
SyntaxError: invalid syntax
Error: downloading configure-339b4599590f3dc3acec5eace4b7a82f210d3219.tar.gz failed

https://github.com/kliem/sage/runs/899346035?check_suite_focus=true

Should they be fixed here?

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 23, 2020

comment:12

I'll look into the tox config error first.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 23, 2020

comment:13

Replying to @kliem:

Centos 6 seems to work in the way that it fails to buil mpir.

Right... errors from an ancient assembler version.

tmp-mul_1.s: Assembler messages:
tmp-mul_1.s:94: Error: no such instruction: `mulx (%rsi),%r9,%r8'
tmp-mul_1.s:95: Error: no such instruction: `mulx 8(%rsi),%r11,%r10'
tmp-mul_1.s:96: Error: no such instruction: `mulx 16(%rsi),%rcx,%r12'

We'll definitely not try to support centos-6 and shouldn't be adding it to tox.yml.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 23, 2020

comment:14

Replying to @kliem:

Centos 6 seems to work in the way that it fails to buil mpir.

There are many errors with manylinux like

tox.ConfigError: ConfigError: substitution env:'BASE_IMAGE': unknown environment variable 'BASE_IMAGE'  or recursive definition.

This is just caused by me not adding documentation and leaving you to guess what is supported. manylinux is not a configuration - only manylinux-1, manylinux-2010, manylinux-2014 are.

I'll make manylinux an alias for manylinux-2014 and add some documentation.

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jul 23, 2020

Changed commit from 756f310 to 8c13ac7

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jul 23, 2020

Branch pushed to git repo; I updated commit sha1. New commits:

8c13ac7tox.ini: Make manylinux default to manylinux-2014, add documentation

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 23, 2020

comment:16

Replying to @kliem:

  File "/sage/build/bin/../sage_bootstrap/download/transfer.py", line 43
    n = 0 if chunks_so_far == 0 else self.length // 2
           ^
SyntaxError: invalid syntax
Error: downloading configure-339b4599590f3dc3acec5eace4b7a82f210d3219.tar.gz failed

https://github.com/kliem/sage/runs/899346035?check_suite_focus=true

The added comments also explain this one -- manylinux-1 is too old, after all, and sage cannot build on it

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 23, 2020

comment:17

On manylinux-2010-minimal (https://github.com/kliem/sage/runs/899346056), building python3 fails with ./python: error while loading shared libraries: libcrypt.so.2: cannot open shared object file: No such file or directory (related to #29012?)
... but actually it was intended to use the installed python3 in /opt. Looks like I need to make adjustments for -2010.

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jul 24, 2020

Changed commit from 8c13ac7 to 2747c37

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jul 24, 2020

Branch pushed to git repo; I updated commit sha1. New commits:

4c670d3build/pkgs/xz/spkg-configure.m4: Slightly lower the xz version bound set in #26286, for manylinux-2010
2f43164tox.ini [manylinux]: Use --with-system-python3=force
2747c37tox.ini [manylinux]: Patch the downloaded configure file so that system xz is accepted

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 24, 2020

comment:20

When you have a chance, please test this version again. See above on my comments on the supported tox environments. Also note the images for non-x86 archs can't work on Linux Docker (and thus on GH Actions) because they are not multi-arch images. They work on macOS Docker Desktop because they are running inside of an emulator.

@kliem
Copy link
Contributor

kliem commented Jul 24, 2020

comment:21

https://github.com/kliem/sage/actions/runs/180807074

That is what I'm testing for now:

 tox_system_factor: [ubuntu-focal, debian-buster, centos-8, manylinux-1, manylinux-2010, manylinux-2014, manylinux-2014-python3.7]

manylinux-1 will of course fail.

@kliem
Copy link
Contributor

kliem commented Jul 24, 2020

comment:22

I'm getting configure failures with minimal. Will this be a subject of this ticket or of another one?

Otherwise it's looking fine.

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jul 24, 2020

Changed commit from 2747c37 to 14da64d

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Jul 24, 2020

Branch pushed to git repo; I updated commit sha1. New commits:

14da64dtox.ini [manylinux]: Do not use --with-system-python3=force for -minimal

@kliem
Copy link
Contributor

kliem commented Jul 24, 2020

comment:24

manylinux-2014-python3.7, minimal still fails at configuration.

https://github.com/kliem/sage/runs/908012032?check_suite_focus=true

Maybe I just got the name scheme wrong.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 24, 2020

comment:25

minimal with python3.7 cannot work because python3 dependencies such as sqlite3 are not available from the system and therefore it cannot use system python3.

@kliem
Copy link
Contributor

kliem commented Jul 24, 2020

comment:26

https://github.com/kliem/sage/actions/runs/181590963

This seems to work well. At least according to the timings we are well in the build process. Any failures are probably subject to different tickets now.

If the run turns out to your satisfaction, you can put it on positive review on my behalf.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jul 25, 2020

comment:27

Thank you!

@vbraun
Copy link
Member

vbraun commented Jul 28, 2020

Changed branch from u/mkoeppe/tox_ini__add_manylinux to 14da64d

@vbraun vbraun closed this as completed in 95d311d Jul 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
c: porting p: major / 3 t: enhancement
Projects
None yet
Milestone
sage-9.2
Development

No branches or pull requests
3 participants
@vbraun @mkoeppe @kliem