Python 3.8: Migrate connected formulae #47326

bayandin · 2019-11-29T15:19:41Z

Have you followed the guidelines for contributing?
Have you checked that there aren't other open pull requests for the same formula update/change?
Have you built your formula locally with brew install --build-from-source <formula>, where <formula> is the name of the formula you're submitting?
Is your test running fine brew test <formula>, where <formula> is the name of the formula you're submitting?
Does your build pass brew audit --strict <formula> (after doing brew install <formula>)?

This PR is a rough approach to migration connected formulae to python@3.8.

Formula/anjuta.rb

bayandin · 2019-11-29T18:49:32Z

For me it didn't work properly, maybe because `python@3.8` is keg_only or because I have `python` installed (and the formula picked it first).

…

On Fri, 29 Nov 2019, 18:45 Bo Anderson, ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In Formula/anjuta.rb <#47326 (comment)> : > depends_on "shared-mime-info" depends_on "vala" depends_on "vte3" def install + # Required for migration period (3.7 -> 3.8) + ENV.prepend_path "PATH", ***@***.***"].opt_bin Does brew not already do this? It seems to for library paths at least. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#47326?email_source=notifications&email_token=AAGS7VO7X2JVBONEJDXGJGDQWFPMFA5CNFSM4JTAZQNKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCNOIM3Q#pullrequestreview-324830830>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGS7VNYENKQ7VUHIG5F5A3QWFPMFANCNFSM4JTAZQNA> .

Bo98 · 2019-11-29T18:59:04Z

The superenv should put dependencies at the front of the PATH list, including keg-only dependencies.

What might be happening is regular python is being pulled in somewhere, perhaps as a build-only dependency.

Bo98 · 2019-11-30T00:06:22Z

Progress so far

boost-python3 doesn't work - last 15 lines of logs are insufficient to tell why.
csound needs migrating.
itstool has the wrong PYTHON environment variable.
libpeas might be picking up the wrong Python.
vtk has the wrong include path (m suffix). Also is picking up 3.7 as an interpreter but not overly critical.

There's perhaps other cases of picking up the wrong Python that might be going unnoticed. Really, I should see where the hole is that it is picking up 3.7.

bayandin · 2019-12-02T12:30:09Z

The superenv should put dependencies at the front of the PATH list, including keg-only dependencies.

What might be happening is regular python is being pulled in somewhere, perhaps as a build-only dependency.

Good to know, will remove redundant PATH prepending

There's perhaps other cases of picking up the wrong Python that might be going unnoticed. Really, I should see where the hole is that it is picking up 3.7.

For vtk I've missed python@3.8 here:

-DPYTHON_EXECUTABLE=#{Formula["python"].opt_bin}/python3

Thanks for the help!

bayandin · 2019-12-04T14:12:09Z

There's perhaps other cases of picking up the wrong Python that might be going unnoticed. Really, I should see where the hole is that it is picking up 3.7.

Looks like even if we are able to migrate all the connected formulae at once, there still be a case when user rebuilds a formula from source and they has both python and python@3.8 the formula could pick the "wrong" python.
Probably we need to set parameters for make/cmake for all the python@3.8 converted formulae..

Bo98 · 2019-12-04T14:35:39Z

when user rebuilds a formula from source and they has both python and python@3.8 the formula could pick the "wrong" python

Apart from the test stage, that shouldn’t happen unless the build scripts are hardcoded to look at /usr/local over any paths which are set. Is there a particular example of this happening?

bayandin · 2019-12-04T14:54:00Z

Is there a particular example of this happening?

I suspect adios2:

cmake reports:

-- Found PythonLibs: /usr/local/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7m.dylib (found version "3.7.5")

Here is the failed installation: https://jenkins.brew.sh/job/Homebrew%20Core%20Pull%20Requests/52862/version=highsierra/testReport/junit/brew-test-bot/high_sierra/install___build_bottle_adios2/

Bo98 · 2019-12-04T15:54:10Z

It's because cmake_find_frameworks does not use CMAKE_FRAMEWORK_PATH for some reason. I might consider that a bug worth fixing in CMake...

-DCMAKE_FIND_FRAMEWORK_EXTRA_LOCATIONS=/usr/local/opt/python@3.8/Frameworks/Python.framework works but this is very problematic as a call to, say, cmake_find_framework(Tk) would also return the Python framework...

If you can find a way to get -DCMAKE_FIND_FRAMEWORK_EXTRA_LOCATIONS=/usr/local/opt/python@3.8/Frameworks/\${fwk}.framework to work that would be nice.

https://gitlab.kitware.com/cmake/cmake/blob/f7ca8fc24ccb4e7724d2228d9997b2df99e71133/Modules/CMakeFindFrameworks.cmake#L26

Bo98 · 2019-12-04T16:15:29Z

Another possibility is to symlink libpython3.8.dylib in lib if that is acceptable and doesn't cause problems elsewhere.

bayandin · 2019-12-04T17:02:48Z

This cmake issue looks relevant: https://gitlab.kitware.com/cmake/cmake/issues/18204 (Closed)

Bo98 · 2019-12-05T14:11:41Z

Just to add that if we do this:

Another possibility is to symlink libpython3.8.dylib in lib if that is acceptable and doesn't cause problems elsewhere.

then we also need to symlink the contents of include as well. The reason it works is because CMake prioritises the lib and include directories over frameworks in FindPythonLibs.

I'll look further into suggesting a patch for framework searching upstream at the weekend.

bayandin · 2019-12-05T14:41:47Z

Just to add that if we do this:

Another possibility is to symlink libpython3.8.dylib in lib if that is acceptable and doesn't cause problems elsewhere.

then we also need to symlink the contents of include as well. The reason it works is because CMake prioritises the lib and include directories over frameworks in FindPythonLibs.

I think we should consider @iMichka's suggestion:

We could explicitly set the right python path, like we are doing for VTK: https://github.com/Homebrew/homebrew-core/blob/master/Formula/vtk.rb#L59-L61
      -DPYTHON_EXECUTABLE=#{Formula["python"].opt_bin}/python3
      -DPYTHON_INCLUDE_DIR=#{py_prefix}/include/python#{pyver}m
      -DPYTHON_LIBRARY=#{py_prefix}/lib/libpython#{pyver}.dylib

It's explicit but will require to change all the formulae depending on python and cmake

Bo98 · 2019-12-05T15:13:50Z

Note that those which use the newer FindPython/FindPython3 won't work with those variables, but do have a better framework check anyway. So it's not quite everything depending on CMake.

basictheprogram · 2019-12-17T19:51:19Z

12 days since the last comment.

How migration going?

I've asked in other tickets how I can help, been directed to this ticket, so I'll post here, that I am available to help.

Was told needed to "wait" on this pull to be merge before I can help.

Can the work not be done in parallel or distributed?

Seem like a lot of work and responsibility for just 1 person.

iMichka · 2019-12-17T21:00:20Z

It's not just one person, we are a few maintainers working on this.
As you have seen, we do struggle with this update.
We have somewhat hit some problems, mostly due to too many formulae to update at once.

Some of these formula were not compatible with Python 3.8 (backward compatibility?). Others failed to build (reproducible builds?). Build times are getting huge, and we our disk space on our CI has been exhausted .... With issues piling up, this is not an easy move.

I think that we will discuss all these issues in February during our annual homebrew meeting.
Until then, we need to get this PR going.

As a first step, we created a pyhton@3.8 formula, which is now merged: #47273

This python version will not be in your path, and can be found at:
/usr/local/Cellar/python@3.8/3.8.0/bin/python3

iMichka · 2019-12-17T21:29:51Z

I would propose to rebase this and:

Remove the python@3.8 change, as this has now been merged and we have the formula (this should also slightly speed up the build).
Use my suggestion for adios2.

Let the build run and have a look at the logs, to see what needs to be still fixed.

We may still fill the hard drive of our bot. In that case I'll have a look at the brew test-bot code to add more aggressive brew cleanup calls ...

bayandin · 2019-12-19T12:45:29Z

Rebased the branch and applied fixes for some formulae (stashed since the last run). Let's see how it goes

bayandin · 2019-12-19T14:11:47Z

sphinx-doc fails because of virtualenv_install_with_resources. Will be fixed in Homebrew/brew#6858

Downloading the test file takes ages: it was mirrored on bintray to speed up the test, which would sometimes hang forever.

iMichka · 2020-04-04T16:38:42Z

pytouhou source repository is currently down

It is back up. At this point if it fails again we will pull this PR anyway. The same holds for any other not-so-important package, we had enough rebuilds now.

caffe fails style

Fixed

C: 69: 6: end at 69, 5 is not aligned with resource("test_model_weights").stage do at 64, 4.
astrometry-net still not playing ball on the Catalina CI for some reason. I wonder if there is a verbose mode or something that we could enable.

Fixed, I wrote a patch for it.

I triggered a new rebuild. If this is green we can pull the PR right away.

Fixes: fitsio/hdu/table.py:234: SyntaxWarning: "is" with a literal. Did you mean "=="? if data.shape is (): fitsio/hdu/table.py:234: SyntaxWarning: "is" with a literal. Did you mean "=="? if data.shape is (): Error found in Homebrew/homebrew-core#47326

bayandin · 2020-04-05T09:58:07Z

Great work @iMichka and @Bo98!

iMichka · 2020-04-05T09:59:16Z

Thanks for the initial PR @bayandin !

Follow up PR: #52590
And @Bo98 is building the missing pytouhou Mojave bottle right now.

Bo98 reviewed Nov 29, 2019

View reviewed changes

Formula/anjuta.rb Outdated Show resolved Hide resolved

bayandin force-pushed the python-3.8.0-p2 branch 2 times, most recently from da871a9 to a9549da Compare December 3, 2019 01:17

This was referenced Dec 3, 2019

Watchman python3 revision bump #47419

Closed

Python 3.8 migration #47274

Closed

python 3.8.0 #45337

Closed

Python 3.8: glib #47447

Closed

bayandin force-pushed the python-3.8.0-p2 branch from a9549da to 4c213a2 Compare December 19, 2019 12:41

bayandin mentioned this pull request Dec 19, 2019

Add python@3.8 handling to virtualenv_install_with_resources Homebrew/brew#6858

Merged

6 tasks

iMichka added the python Python use is a significant feature of the PR or issue label Dec 19, 2019

jonchang mentioned this pull request Dec 20, 2019

python 3.8.1 #48097

Closed

bayandin force-pushed the python-3.8.0-p2 branch from 4c213a2 to 2d6f1fe Compare December 20, 2019 14:00

bayandin and others added 20 commits April 4, 2020 18:34

pyqt: revision bump for python@3.8

fbec344

pytouhou: revision bump for python@3.8

78bf81f

qscintilla2: revision bump for python@3.8

75f4133

root: revision bump for python@3.8

84f36b2

scipy: revision bump for python@3.8

8f2fa1e

sip: revision bump for python@3.8

4721749

tbb: revision bump for python@3.8

c7ae169

template-glib: revision bump for python@3.8

5451e80

urh: revision bump for python@3.8

a9ee4c4

vtk: revision bump for python@3.8

fbcfc56

wxpython: revision bump for python@3.8

753bf13

xdot: revision bump for python@3.8

642754f

caffe: improve test

1ac3be2

Downloading the test file takes ages: it was mirrored on bintray to speed up the test, which would sometimes hang forever.

pcl: revision bump for python@3.8

41cde55

glib: revision bump for python@3.8

64dbb34

meson: revision bump for python@3.8

407d373

cairo: revision bump for python@3.8

96cdb20

zim: revision bump for python@3.8

ee5c808

pympress: revision bump for python@3.8

7f06170

visp: revision bump for python@3.8

9f2dc97

iMichka force-pushed the python-3.8.0-p2 branch from 082736d to 9f2dc97 Compare April 4, 2020 16:37

iMichka closed this in 987c9e1 Apr 5, 2020

bayandin deleted the python-3.8.0-p2 branch April 5, 2020 11:20

jodal mentioned this pull request Apr 11, 2020

Can't install via brew as of today mopidy/homebrew-mopidy#36

Closed

lock bot added the outdated PR was locked due to age label May 5, 2020

lock bot locked as resolved and limited conversation to collaborators May 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3.8: Migrate connected formulae #47326

Python 3.8: Migrate connected formulae #47326

bayandin commented Nov 29, 2019 •

edited

Loading

bayandin commented Nov 29, 2019 via email •

edited

Loading

Bo98 commented Nov 29, 2019

Bo98 commented Nov 30, 2019 •

edited

Loading

bayandin commented Dec 2, 2019

bayandin commented Dec 4, 2019

Bo98 commented Dec 4, 2019 •

edited

Loading

bayandin commented Dec 4, 2019

Bo98 commented Dec 4, 2019

Bo98 commented Dec 4, 2019

bayandin commented Dec 4, 2019 •

edited

Loading

Bo98 commented Dec 5, 2019 •

edited

Loading

bayandin commented Dec 5, 2019

Bo98 commented Dec 5, 2019

basictheprogram commented Dec 17, 2019

iMichka commented Dec 17, 2019

iMichka commented Dec 17, 2019

bayandin commented Dec 19, 2019

bayandin commented Dec 19, 2019 •

edited

Loading

iMichka commented Apr 4, 2020

bayandin commented Apr 5, 2020

iMichka commented Apr 5, 2020

Python 3.8: Migrate connected formulae #47326

Python 3.8: Migrate connected formulae #47326

Conversation

bayandin commented Nov 29, 2019 • edited Loading

bayandin commented Nov 29, 2019 via email • edited Loading

Bo98 commented Nov 29, 2019

Bo98 commented Nov 30, 2019 • edited Loading

bayandin commented Dec 2, 2019

bayandin commented Dec 4, 2019

Bo98 commented Dec 4, 2019 • edited Loading

bayandin commented Dec 4, 2019

Bo98 commented Dec 4, 2019

Bo98 commented Dec 4, 2019

bayandin commented Dec 4, 2019 • edited Loading

Bo98 commented Dec 5, 2019 • edited Loading

bayandin commented Dec 5, 2019

Bo98 commented Dec 5, 2019

basictheprogram commented Dec 17, 2019

iMichka commented Dec 17, 2019

iMichka commented Dec 17, 2019

bayandin commented Dec 19, 2019

bayandin commented Dec 19, 2019 • edited Loading

iMichka commented Apr 4, 2020

bayandin commented Apr 5, 2020

iMichka commented Apr 5, 2020

bayandin commented Nov 29, 2019 •

edited

Loading

bayandin commented Nov 29, 2019 via email •

edited

Loading

Bo98 commented Nov 30, 2019 •

edited

Loading

Bo98 commented Dec 4, 2019 •

edited

Loading

bayandin commented Dec 4, 2019 •

edited

Loading

Bo98 commented Dec 5, 2019 •

edited

Loading

bayandin commented Dec 19, 2019 •

edited

Loading