Only build x86_64 / Intel 10.9+ on macOS #220

mayeut · 2019-11-19T23:56:37Z

Only build x86_64 / Intel 10.9+ on macOS

mayeut · 2019-11-21T00:26:17Z

It probably should be the default. intel can even be dropped I think, that would simplify things greatly as well as going 10.9+. Then this PR would become "fix python 3.5 on macOS to produce a macos_10_9_x86_64 wheel" and changes will be dropped when python 3.5 goes EOL. We can also close #202 / #203 if we go that way.

SELECT
  COUNT(*) AS num_downloads,
  REGEXP_EXTRACT(details.distro.version, r"^(\d+.\d+)") as system_version,
  details.cpu
FROM `the-psf.pypi.downloads*`
WHERE
  details.system.name = 'Darwin' AND
  file.filename like '%macos%' AND
  _TABLE_SUFFIX
    BETWEEN FORMAT_DATE(
      '%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
    AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
GROUP BY system_version, details.cpu
ORDER BY num_downloads DESC

PS: I'm not sure that i686/x86_64 actually reflect the flavor of python running (I download one of my wheel from a 10.14 system from python3.7-32 and I can't see it in the stats while I'm sure it wasn't cached). Given that python runs x86_64 as a default when possible, it shouldn't change much the outcome.

joerick · 2019-11-21T09:10:15Z

Tbh I'm on board with 64bit only default on Mac too, if there are no major objections. I can't remember my last 32bit mac, but it was a loong time ago! The only thing I don't know (agreeing with your suspicion with those figures) is if people are still running python-32 on a 64bit machine (for binary compat reasons), but I'd say it was an edge case.

I think CIBW_PREFERRED_ARCHITECTURE_MACOS might be a good name? Keeping the platform as a suffix for conformity.

Also, I'm thinking this PR could remove the arch from build identifier, so it'd be just cp38-macosx_10_14.

What do you think @mayeut?

joerick · 2019-11-21T09:22:53Z

That build identifier would actually become even simpler cp37-macosx when #156 lands, because the specific version of macOS depends on an environment variable that's set when building. I should probably have made them that simple from the start! Hindsight 20/20 I suppose 🙄

EDIT: see #220 (comment) for more discussion on this - I've changed my mind a bit

mayeut · 2019-11-21T21:18:51Z

I think CIBW_PREFERRED_ARCHITECTURE_MACOS might be a good name? Keeping the platform as a suffix for conformity.

I think your proposal is much clearer for the end user (and even for me) but for consistency sake, I went with MACHINE which is used throughout CPython when building platform tags. I guess my explanation doesn't change your preference ?

Tbh I'm on board with 64bit only default on Mac too, if there are no major objections. I can't remember my last 32bit mac, but it was a loong time ago! The only thing I don't know (agreeing with your suspicion with those figures) is if people are still running python-32 on a 64bit machine (for binary compat reasons), but I'd say it was an edge case.

Ok, so default to x86_64 but still allow intel builds on demand ? You don't want to just drop 32-bit and go to 10.9 as a minimum ?

YannickJadoul · 2019-11-23T22:40:08Z

You don't want to just drop 32-bit and go to 10.9 as a minimum ?

In combination with #156, aren't we leaving the minimum version up to the user to decide and to be checked by wheel?

mayeut · 2019-11-26T21:29:26Z

Let's wait for #156 to be merged-in, I'll rework this after.

joerick · 2019-12-26T12:29:04Z

#156 is waiting on a pypa/wheel release, which might be a while yet...

@mayeut:

it probably should be the default. intel can even be dropped I think, that would simplify things greatly as well as going 10.9+. Then this PR would become "fix python 3.5 on macOS to produce a macos_10_9_x86_64 wheel" and changes will be dropped when python 3.5 goes EOL. We can also close #202 / #203 if we go that way.

For the sake of simplicity, I'm happy to just do the above - drop intel and go to 10_9 where we can. 32-bit isn't something our users have asked for, and it'll be less code to maintain. Would you be able to go ahead with that @mayeut ?

joerick

Feeling good about how this simplifies things :) let me know what you think about the build identifier change!

joerick · 2019-12-26T16:22:27Z

cibuildwheel/macos.py

+        PythonConfiguration(version='2.7', identifier='cp27-macosx_10_9_x86_64', url='https://www.python.org/ftp/python/2.7.17/python-2.7.17-macosx10.9.pkg'),
+        PythonConfiguration(version='3.5', identifier='cp35-macosx_10_9_x86_64', url='https://www.python.org/ftp/python/3.5.4/python-3.5.4-macosx10.6.pkg'),
+        PythonConfiguration(version='3.6', identifier='cp36-macosx_10_9_x86_64', url='https://www.python.org/ftp/python/3.6.8/python-3.6.8-macosx10.9.pkg'),
+        PythonConfiguration(version='3.7', identifier='cp37-macosx_10_9_x86_64', url='https://www.python.org/ftp/python/3.7.6/python-3.7.6-macosx10.9.pkg'),


Aha, so we have a little snag here - the build identifiers are changing - both the machine and the macos system.

Previously I've mused about renaming these e.g. cp37-macosx_10_9_x86_64 → cp37-macosx. Thinking about this fresh...

The architecture isn't needed any longer, since we only build x86_64. Should we remove it? Will we ever need an architecture switch on macos again? Hm, well, ARM Macs might exist one day! And, all other platforms contain an architecture flag, so keeping it retains consistency.

Then there's the 10_9 bit - we expect to remove this part because #156 should be making this conditional on the MACOSX_DEPLOYMENT_TARGET var instead.

So here's what I'm thinking... rather than making users update their settings from cp37-macosx_10_6_intel to cp37-macosx_10_9_x86_64 and then from cp37-macosx_10_9_x86_64 to cp37-macosx_x86_64 when #156 lands, let's pull in the build selector migration and go to the cp37-macosx_x86_64 format in this PR.

Thoughts @YannickJadoul @Czaki ?

Depends a bit on when we can merge #156, i.e., when a new wheel version will be released.

If that's before you plan a new release, I'd argue to keep this as it is now and not mix different PRs. If wheel won't release too soon, then yeah, maybe. But maybe that's still another clean PR, then?
(I've asked, btw: pypa/wheel#314 (comment); let's see what comes out of that.)

On keeping the architecture: I don't think that's really bothering? It even keeps things clear, that wheels will only be built for 64-bit, I'd say. And with the wildcard matching on *-macosx_*, most users might not even need to change too much (I'd hope users don't depend on this 10_6/10_9 too much?). And indeed: keeping it would be good for the sake of consistency with manylinux_x86_64/manylinux_i686/ win32/win_amd64, where also now always have architecture, but not the platform's version :-)

Afterthought: is there any reason to use the 10.9 packages from Python.org? We should be able to build 64-bit-only wheels that are still 10.6-compatible, or not, @mayeut? In that case, we can still keep our old build identifiers for a little while, until #156.

Thanks for getting in touch with the wheel folks! I had been trying to unblock a few PRs but hopefully that's moving now.

Afterthought: is there any reason to use the 10.9 packages from Python.org?

I think this is more convenient because the 10.9 packages build a x86_64 wheel by default, but I'll let @mayeut correct me if I'm wrong...

joerick · 2019-12-26T16:24:13Z

README.md

+| Python 3.5 | ✅ | ✅ | ✅ | ✅  | ✅  |
+| Python 3.6 | ✅ | ✅ | ✅ | ✅  | ✅  |
+| Python 3.7 | ✅ | ✅ | ✅ | ✅  | ✅  |
+| Python 3.8 | ✅ | ✅ | ✅ | ✅  | ✅  |


Don't merge #185, then :-/

joerick · 2020-01-28T10:35:45Z

I think this PR can get moving again! @mayeut, are you able to rebase/merge master?

joerick · 2020-01-31T17:36:11Z

I've made the changes you suggested @Czaki @YannickJadoul, and I think I'm happy with it. Any further comments?

YannickJadoul · 2020-01-31T18:17:15Z

Looks good to me!

One thing I can still think of: now that macosx_intel do not work anymore, should we add a warning/error/something else to __main__.py, just like we did before when we replaced old build tags?

If not, go ahead and merge, as far as I'm concerned :-)

Czaki · 2020-01-31T18:27:24Z

I try to build quite big python 3.5 depend project. I takes about an hour to finish build. In few hour i will give information if it pass.

Czaki · 2020-01-31T19:38:44Z

It works.

YannickJadoul · 2020-01-31T19:49:19Z

Hurray! Thanks for testing, @Czaki!

One thing I can still think of: now that macosx_intel do not work anymore, should we add a warning/error/something else to __main__.py, just like we did before when we replaced old build tags?

Then it's just this. Any thoughts, @Czaki? Because maybe this should have already be done in #156, actually.
If not, let's just leave it to @joerick's judgement :-)

joerick · 2020-01-31T19:57:48Z

One thing I can still think of: now that macosx_intel do not work anymore, should we add a warning/error/something else to main.py, just like we did before when we replaced old build tags?

Ah! Yes, good catch, thank you @YannickJadoul !

YannickJadoul · 2020-01-31T20:00:29Z

Ah! Yes, good catch, thank you @YannickJadoul !

But so, worth taking into account that it also already changed in #156. So you might want to check for macosx_10_6 or macosx_10_9 as well!

joerick · 2020-01-31T20:01:30Z

So you might want to check for macosx_10_6 or macosx_10_9 as well!

Indeed. We haven't released that migration, so they can maybe be merged together.

Czaki · 2020-01-31T21:55:16Z

in #156 system version is removed only. I do not do any changes with intel/x86_64 suffixes.

YannickJadoul

Yesss, nice! :-)

YannickJadoul · 2020-02-02T18:59:51Z

@joerick @Czaki Just found that the docs still say

To get C++11 and C++14 support, set MACOSX_DEPLOYMENT_TARGET to (at least) "10.9".

In cpp_standards.md

Since we've now bumped everything to at least 10.9, we don't need that anymore?

Czaki · 2020-02-02T19:03:07Z

Yes

…GET >= 10.9 (see #220)

YannickJadoul · 2020-02-02T19:08:10Z

Fixed; quickly pushed ffec9ae to master.

McSinyx · 2020-03-01T04:08:14Z

Hi, is it possible to have a release with this in a near future? Our project needs 10.9 for C++11 support and the previous release only have it for Python 3.8.

Czaki · 2020-03-01T08:19:38Z

Hi @McSinyx. Only thing you need is to define MACOSX_DEPLOYMENT_TARGET to 10.9. See more: https://cibuildwheel.readthedocs.io/en/latest/cpp_standards/

McSinyx · 2020-03-01T08:48:28Z

Thank you @Czaki for the tip. How do I specify the Python version then? Is the following OK?

CIBW_BUILD=cp37-macosx_10_6_intel

I see the linked documentation is built on GitHub master instead of the released one though. I've just checked out v1.1.0 and MACOSX_DEPLOYMENT_TARGET is not handled anywhere.

Czaki · 2020-03-01T08:55:19Z

I see the linked documentation is built on GitHub master instead of the released one though. I've just checked out v1.1.0 and MACOSX_DEPLOYMENT_TARGET is not handled anywhere.

Handling of MACOSX_DEPLOYMENT_TARGET is in wheel package form release 0.34.0

Thank you @Czaki for the tip. How do I specify the Python version then? Is the following OK?

because next release should be in short time I suggest usage

CIBW_BUILD="cp37-macosx*"

McSinyx · 2020-03-01T09:12:20Z

Handling of MACOSX_DEPLOYMENT_TARGET is in wheel package form release 0.34.0

Thanks, I was wondering why it isn't prefixed with CIBW. Your suggestion works very well too!

Czaki · 2020-03-01T09:28:28Z

Thanks, I was wondering why it isn't prefixed with CIBW.

Because this is MacOs system variable used to control compatibility .

Your suggestion works very well too!

Why you use separate job for each build? Maybe try: CIBW_BUILD="cp3[6-8]-macosx*"

McSinyx · 2020-03-01T09:33:17Z

Why you use separate job for each build?

We want to make use of some parallelism, but mainly it's for better Python compatibility test (e.g. if 3.6 fails we would still want to know if 3.7 passes).

mayeut mentioned this pull request Nov 20, 2019

Run 32-bit tests on macOS #202

Closed

mayeut force-pushed the macos-platform branch from 8b02d38 to a754b1d Compare November 20, 2019 06:26

mayeut mentioned this pull request Nov 20, 2019

Add support for GitHub Actions #194

Merged

4 tasks

mayeut force-pushed the macos-platform branch 3 times, most recently from bfbc981 to 8414d90 Compare November 20, 2019 23:38

mayeut force-pushed the macos-platform branch from 8414d90 to cd026d8 Compare November 21, 2019 21:47

mayeut force-pushed the macos-platform branch 2 times, most recently from 0db3757 to 58d6212 Compare December 2, 2019 10:10

mayeut force-pushed the macos-platform branch from 58d6212 to 983cbd9 Compare December 12, 2019 20:24

mayeut force-pushed the macos-platform branch from 983cbd9 to dc7c55c Compare December 26, 2019 13:43

mayeut changed the title ~~Add option to build x86_64 only on macOS~~ Only build x86_64 / Intel 10.9+ on macOS Dec 26, 2019

mayeut force-pushed the macos-platform branch 3 times, most recently from da835c5 to 0e0478a Compare December 26, 2019 15:08

mayeut marked this pull request as ready for review December 26, 2019 15:11

joerick reviewed Dec 26, 2019

View reviewed changes

joerick mentioned this pull request Jan 7, 2020

Fix all PEP-008 issues #212

Merged

mayeut force-pushed the macos-platform branch 2 times, most recently from c4c9eba to 7aff399 Compare January 10, 2020 21:23

joerick mentioned this pull request Jan 27, 2020

upgrade macos version for support c++11 #156

Merged

YannickJadoul mentioned this pull request Jan 28, 2020

Adding PyPy support #185

Merged

Fix test protection reference

a97b6a1

Add migration for build identifier change, and test for migration

c3f628d

YannickJadoul approved these changes Feb 1, 2020

View reviewed changes

Czaki approved these changes Feb 1, 2020

View reviewed changes

joerick merged commit 8236877 into pypa:master Feb 1, 2020

joerick mentioned this pull request Feb 2, 2020

32-bit tests are not run on macOS when creating an intel wheel #203

Closed

YannickJadoul mentioned this pull request Feb 2, 2020

drop support for python 2.7 #245

Closed

YannickJadoul added a commit that referenced this pull request Feb 2, 2020

Update line in docs/cpp_standards.md on setting MACOSX_DEPLOYMENT_TAR…

ffec9ae

…GET >= 10.9 (see #220)

mayeut deleted the macos-platform branch February 6, 2020 17:15

McSinyx mentioned this pull request Mar 1, 2020

Packaging for Windows McSinyx/palace#1

Closed

YannickJadoul mentioned this pull request Apr 28, 2020

macOS target version and C++17 #325

Closed

Only build x86_64 / Intel 10.9+ on macOS #220

Only build x86_64 / Intel 10.9+ on macOS #220

Conversation

mayeut commented Nov 19, 2019 • edited Loading

mayeut commented Nov 21, 2019

joerick commented Nov 21, 2019

joerick commented Nov 21, 2019 • edited Loading

mayeut commented Nov 21, 2019

YannickJadoul commented Nov 23, 2019

mayeut commented Nov 26, 2019

joerick commented Dec 26, 2019

joerick left a comment

Choose a reason for hiding this comment

joerick Dec 26, 2019

Choose a reason for hiding this comment

YannickJadoul Dec 26, 2019

Choose a reason for hiding this comment

YannickJadoul Dec 26, 2019

Choose a reason for hiding this comment

joerick Dec 27, 2019

Choose a reason for hiding this comment

joerick Dec 26, 2019

Choose a reason for hiding this comment

YannickJadoul Dec 26, 2019

Choose a reason for hiding this comment

joerick commented Jan 28, 2020

joerick commented Jan 31, 2020

YannickJadoul commented Jan 31, 2020

Czaki commented Jan 31, 2020

Czaki commented Jan 31, 2020

YannickJadoul commented Jan 31, 2020

joerick commented Jan 31, 2020

YannickJadoul commented Jan 31, 2020

joerick commented Jan 31, 2020 • edited Loading

Czaki commented Jan 31, 2020

YannickJadoul left a comment

Choose a reason for hiding this comment

YannickJadoul commented Feb 2, 2020

Czaki commented Feb 2, 2020

YannickJadoul commented Feb 2, 2020

McSinyx commented Mar 1, 2020

Czaki commented Mar 1, 2020

McSinyx commented Mar 1, 2020

Czaki commented Mar 1, 2020

McSinyx commented Mar 1, 2020

Czaki commented Mar 1, 2020

McSinyx commented Mar 1, 2020

mayeut commented Nov 19, 2019 •

edited

Loading

joerick commented Nov 21, 2019 •

edited

Loading

joerick commented Jan 31, 2020 •

edited

Loading