Skip to content

Commit

Permalink
feat: add musllinux support (PEP 656) doc
Browse files Browse the repository at this point in the history
  • Loading branch information
@mayeut
mayeut committed Sep 19, 2021
1 parent 45fc85e commit d328132
Show file tree
Hide file tree
Showing 3 changed files with 44 additions and 23 deletions.
21 changes: 11 additions & 10 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,17 +22,17 @@ Python wheels are great. Building them across **Mac, Linux, Windows**, on **mult
What does it do?
----------------

| | macOS Intel | macOS Apple Silicon | Windows 64bit | Windows 32bit | manylinux x86_64 | manylinux i686 | manylinux aarch64 | manylinux ppc64le | manylinux s390x |
|---------------|----|-----|-----|-----|----|-----|----|-----|-----|
| CPython 3.6 | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.7 | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.8 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.9 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.10 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| PyPy 3.7 v7.3 | ✅ | N/A | ✅ | N/A | ✅ | ✅ | ✅ | N/A | N/A |
| | macOS Intel | macOS Apple Silicon | Windows 64bit | Windows 32bit | manylinux x86_64 | manylinux i686 | manylinux aarch64 | manylinux ppc64le | manylinux s390x | musllinux x86_64 | musllinux i686 | musllinux aarch64 | musllinux ppc64le | musllinux s390x |
|---------------|----|-----|-----|-----|----|-----|----|-----|-----|----|-----|----|-----|-----|
| CPython 3.6 | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.7 | ✅ | N/A | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.8 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.9 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CPython 3.10 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| PyPy 3.7 v7.3 | ✅ | N/A | ✅ | N/A | ✅ | ✅ | ✅ | N/A | N/A | N/A | N/A | N/A | N/A | N/A |


- Builds manylinux, macOS 10.9+, and Windows wheels for CPython and PyPy
- Builds manylinux, musllinux, macOS 10.9+, and Windows wheels for CPython and PyPy
- Works on GitHub Actions, Azure Pipelines, Travis CI, AppVeyor, CircleCI, and GitLab CI
- Bundles shared library dependencies on Linux and macOS through [auditwheel](https://github.com/pypa/auditwheel) and [delocate](https://github.com/matthew-brett/delocate)
- Runs your library's tests against the wheel-installed version of your library
Expand Down Expand Up @@ -62,7 +62,7 @@ Usage
Example setup
-------------

To build manylinux, macOS, and Windows wheels on GitHub Actions, you could use this `.github/workflows/wheels.yml`:
To build manylinux, musllinux, macOS, and Windows wheels on GitHub Actions, you could use this `.github/workflows/wheels.yml`:

```yaml
name: Build
Expand Down Expand Up @@ -115,6 +115,7 @@ Options
| | [`CIBW_BEFORE_BUILD`](https://cibuildwheel.readthedocs.io/en/stable/options/#before-build) | Execute a shell command preparing each wheel's build |
| | [`CIBW_REPAIR_WHEEL_COMMAND`](https://cibuildwheel.readthedocs.io/en/stable/options/#repair-wheel-command) | Execute a shell command to repair each (non-pure Python) built wheel |
| | [`CIBW_MANYLINUX_*_IMAGE`](https://cibuildwheel.readthedocs.io/en/stable/options/#manylinux-image) | Specify alternative manylinux Docker images |
| | [`CIBW_MUSLLINUX_*_IMAGE`](https://cibuildwheel.readthedocs.io/en/stable/options/#musllinux-image) | Specify alternative musllinux Docker images |
| | [`CIBW_DEPENDENCY_VERSIONS`](https://cibuildwheel.readthedocs.io/en/stable/options/#dependency-versions) | Specify how cibuildwheel controls the versions of the tools it uses |
| **Testing** | [`CIBW_TEST_COMMAND`](https://cibuildwheel.readthedocs.io/en/stable/options/#test-command) | Execute a shell command to test each built wheel |
| | [`CIBW_BEFORE_TEST`](https://cibuildwheel.readthedocs.io/en/stable/options/#before-test) | Execute a shell command before testing each wheel |
Expand Down
6 changes: 3 additions & 3 deletions docs/faq.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@ title: Tips and tricks

### Linux builds on Docker

Linux wheels are built in the [`manylinux` docker images](https://github.com/pypa/manylinux) to provide binary compatible wheels on Linux, according to [PEP 571](https://www.python.org/dev/peps/pep-0571/). Because of this, when building with `cibuildwheel` on Linux, a few things should be taken into account:
Linux wheels are built in the [`manylinux`/`musllinux` docker images](https://github.com/pypa/manylinux) to provide binary compatible wheels on Linux, according to [PEP 600](https://www.python.org/dev/peps/pep-0600/) / [PEP 656](https://www.python.org/dev/peps/pep-0656/). Because of this, when building with `cibuildwheel` on Linux, a few things should be taken into account:

- Programs and libraries are not installed on the CI runner host, but rather should be installed inside of the Docker image - using `yum` for `manylinux2010` or `manylinux2014`, and `apt-get` for `manylinux_2_24`, or manually. The same goes for environment variables that are potentially needed to customize the wheel building.
- Programs and libraries are not installed on the CI runner host, but rather should be installed inside of the Docker image - using `yum` for `manylinux2010` or `manylinux2014`, `apt-get` for `manylinux_2_24` and `apk` for `musllinux_1_1`, or manually. The same goes for environment variables that are potentially needed to customize the wheel building.

`cibuildwheel` supports this by providing the [`CIBW_ENVIRONMENT`](options.md#environment) and [`CIBW_BEFORE_ALL`](options.md#before-all) options to setup the build environment inside the running Docker image.

- The project directory is mounted in the running Docker instance as `/project`, the output directory for the wheels as `/output`. In general, this is handled transparently by `cibuildwheel`. For a more finegrained level of control however, the root of the host file system is mounted as `/host`, allowing for example to access shared files, caches, etc. on the host file system. Note that `/host` is not available on CircleCI due to their Docker policies.

- Alternative Docker images can be specified with the `CIBW_MANYLINUX_*_IMAGE` options to allow for a custom, preconfigured build environment for the Linux builds. See [options](options.md#manylinux-image) for more details.
- Alternative Docker images can be specified with the `CIBW_MANYLINUX_*_IMAGE`/`CIBW_MUSLLINUX_*_IMAGE` options to allow for a custom, preconfigured build environment for the Linux builds. See [options](options.md#manylinux-image) for more details.

### Building macOS wheels for Apple Silicon {: #apple-silicon}

Expand Down
40 changes: 30 additions & 10 deletions docs/options.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -151,12 +151,12 @@ When setting the options, you can use shell-style globbing syntax, as per [fnmat

<div class="build-id-table-marker"></div>

| | macOS | Windows | Manylinux Intel | Manylinux Other |
|--------------|---------------------------------------------------------------------|-------------------------------|-----------------------------------------------|----------------------------------------------------------------------------|
| Python 3.6 | cp36-macosx_x86_64 | cp36-win_amd64<br/>cp36-win32 | cp36-manylinux_x86_64<br/>cp36-manylinux_i686 | cp36-manylinux_aarch64<br/>cp36-manylinux_ppc64le<br/>cp36-manylinux_s390x |
| Python 3.7 | cp37-macosx_x86_64 | cp37-win_amd64<br/>cp37-win32 | cp37-manylinux_x86_64<br/>cp37-manylinux_i686 | cp37-manylinux_aarch64<br/>cp37-manylinux_ppc64le<br/>cp37-manylinux_s390x |
| Python 3.8 | cp38-macosx_x86_64<br/>cp38-macosx_universal2<br/>cp38-macosx_arm64 | cp38-win_amd64<br/>cp38-win32 | cp38-manylinux_x86_64<br/>cp38-manylinux_i686 | cp38-manylinux_aarch64<br/>cp38-manylinux_ppc64le<br/>cp38-manylinux_s390x |
| Python 3.9 | cp39-macosx_x86_64<br/>cp39-macosx_universal2<br/>cp39-macosx_arm64 | cp39-win_amd64<br/>cp39-win32 | cp39-manylinux_x86_64<br/>cp39-manylinux_i686 | cp39-manylinux_aarch64<br/>cp39-manylinux_ppc64le<br/>cp39-manylinux_s390x |
| | macOS | Windows | Manylinux Intel | Manylinux Other | Musllinux Intel | Musllinux Other |
|--------------|---------------------------------------------------------------------|-------------------------------|-----------------------------------------------|----------------------------------------------------------------------------|-----------------------------------------------|----------------------------------------------------------------------------|
| Python 3.6 | cp36-macosx_x86_64 | cp36-win_amd64<br/>cp36-win32 | cp36-manylinux_x86_64<br/>cp36-manylinux_i686 | cp36-manylinux_aarch64<br/>cp36-manylinux_ppc64le<br/>cp36-manylinux_s390x | cp36-musllinux_x86_64<br/>cp36-musllinux_i686 | cp36-musllinux_aarch64<br/>cp36-musllinux_ppc64le<br/>cp36-musllinux_s390x |
| Python 3.7 | cp37-macosx_x86_64 | cp37-win_amd64<br/>cp37-win32 | cp37-manylinux_x86_64<br/>cp37-manylinux_i686 | cp37-manylinux_aarch64<br/>cp37-manylinux_ppc64le<br/>cp37-manylinux_s390x | cp37-musllinux_x86_64<br/>cp37-musllinux_i686 | cp37-musllinux_aarch64<br/>cp37-musllinux_ppc64le<br/>cp37-musllinux_s390x |
| Python 3.8 | cp38-macosx_x86_64<br/>cp38-macosx_universal2<br/>cp38-macosx_arm64 | cp38-win_amd64<br/>cp38-win32 | cp38-manylinux_x86_64<br/>cp38-manylinux_i686 | cp38-manylinux_aarch64<br/>cp38-manylinux_ppc64le<br/>cp38-manylinux_s390x | cp38-musllinux_x86_64<br/>cp38-musllinux_i686 | cp38-musllinux_aarch64<br/>cp38-musllinux_ppc64le<br/>cp38-musllinux_s390x |
| Python 3.9 | cp39-macosx_x86_64<br/>cp39-macosx_universal2<br/>cp39-macosx_arm64 | cp39-win_amd64<br/>cp39-win32 | cp39-manylinux_x86_64<br/>cp39-manylinux_i686 | cp39-manylinux_aarch64<br/>cp39-manylinux_ppc64le<br/>cp39-manylinux_s390x | cp39-musllinux_x86_64<br/>cp39-musllinux_i686 | cp39-musllinux_aarch64<br/>cp39-musllinux_ppc64le<br/>cp39-musllinux_s390x |
| PyPy3.7 v7.3 | pp37-macosx_x86_64 | pp37-win_amd64 | pp37-manylinux_x86_64 | |


Expand Down Expand Up @@ -862,6 +862,26 @@ Auditwheel detects the version of the manylinux standard in the Docker image thr
Like any other option, these can be placed in `[tool.cibuildwheel.linux]`
if you prefer; they have no effect on `macos` and `windows`.

### CIBW_MUSLLINUX_*_IMAGE {: #musllinux-image}
> Specify alternative musllinux Docker images
The available options are:

- `CIBW_MUSLLINUX_X86_64_IMAGE`
- `CIBW_MUSLLINUX_I686_IMAGE`
- `CIBW_MUSLLINUX_AARCH64_IMAGE`
- `CIBW_MUSLLINUX_PPC64LE_IMAGE`
- `CIBW_MUSLLINUX_S390X_IMAGE`

Set an alternative Docker image to be used for building [musllinux](https://github.com/pypa/manylinux) wheels. cibuildwheel will then pull these instead of the default images, [`quay.io/pypa/musllinux_1_1_x86_64`](https://quay.io/pypa/musllinux_1_1_x86_64), [`quay.io/pypa/musllinux_1_1_i686`](https://quay.io/pypa/musllinux_1_1_i686), [`quay.io/pypa/musllinux_1_1_aarch64`](https://quay.io/pypa/musllinux_1_1_aarch64), [`quay.io/pypa/musllinux_1_1_ppc64le`](https://quay.io/pypa/musllinux_1_1_ppc64le), and [`quay.io/pypa/musllinux_1_1_s390x`](https://quay.io/pypa/musllinux_1_1_s390x).

The value of this option can either be set to `musllinux_1_1` to use a pinned version of the [official musllinux images](https://github.com/pypa/musllinux). Alternatively, set these options to any other valid Docker image name.
If this option is blank, it will fall though to the next available definition (environment variable -> pyproject.toml -> default).

If setting a custom Docker image, you'll need to make sure it can be used in the same way as the official, default Docker images: all necessary Python and pip versions need to be present in `/opt/python/`, and the auditwheel tool needs to be present for cibuildwheel to work. Apart from that, the architecture and relevant shared system libraries need to be compatible to the relevant standard to produce valid musllinux_1_1 wheels (see [pypa/manylinux on GitHub](https://github.com/pypa/manylinux) and [PEP 656](https://www.python.org/dev/peps/pep-0656/) for more details).

Auditwheel detects the version of the musllinux standard in the Docker image through the `AUDITWHEEL_PLAT` environment variable, as cibuildwheel has no way of detecting the correct `--plat` command line argument to pass to auditwheel for a custom image. If a Docker image does not correctly set this `AUDITWHEEL_PLAT` environment variable, the `CIBW_ENVIRONMENT` option can be used to do so (e.g., `CIBW_ENVIRONMENT='AUDITWHEEL_PLAT="musllinux_1_1_$(uname -m)"'`).

### `CIBW_DEPENDENCY_VERSIONS` {: #dependency-versions}
> Specify how cibuildwheel controls the versions of the tools it uses
Expand Down Expand Up @@ -894,8 +914,8 @@ Platform-specific environment variables are also available:<br/>

!!! note
This option does not affect the tools used on the Linux build - those versions
are bundled with the manylinux image that cibuildwheel uses. To change
dependency versions on Linux, use the [CIBW_MANYLINUX_*](#manylinux-image)
are bundled with the manylinux/musllinux image that cibuildwheel uses. To change
dependency versions on Linux, use the [CIBW_MANYLINUX_*](#manylinux-image) / [CIBW_MUSLLINUX_*](#musllinux-image)
options.

#### Examples
Expand Down Expand Up @@ -1139,7 +1159,7 @@ With macOS `universal2` wheels, you can also skip the individual archs inside th

```yaml
# Will avoid testing on emulated architectures
CIBW_TEST_SKIP: "*-manylinux_{aarch64,ppc64le,s390x}"
CIBW_TEST_SKIP: "*-*linux_{aarch64,ppc64le,s390x}"

# Skip trying to test arm64 builds on Intel Macs
CIBW_TEST_SKIP: "*-macosx_arm64 *-macosx_universal2:arm64"
Expand All @@ -1150,7 +1170,7 @@ With macOS `universal2` wheels, you can also skip the individual archs inside th
```toml
[tool.cibuildwheel]
# Will avoid testing on emulated architectures
test-skip = "*-manylinux_{aarch64,ppc64le,s390x}"
test-skip = "*-*linux_{aarch64,ppc64le,s390x}"

# Skip trying to test arm64 builds on Intel Macs
test-skip = "*-macosx_arm64 *-macosx_universal2:arm64"
Expand Down

0 comments on commit d328132

Please sign in to comment.