From aed999dc9b1ce054ba75b656b0d1d6af74e0c029 Mon Sep 17 00:00:00 2001 From: Will Dietz Date: Mon, 19 Feb 2018 15:11:44 -0600 Subject: [PATCH 1/8] copy rfc template, unnumbered as of yet --- rfcs/0000-musl-libc.md | 47 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 rfcs/0000-musl-libc.md diff --git a/rfcs/0000-musl-libc.md b/rfcs/0000-musl-libc.md new file mode 100644 index 000000000..fa5e28498 --- /dev/null +++ b/rfcs/0000-musl-libc.md @@ -0,0 +1,47 @@ +--- +feature: (fill me in with a unique ident, my_awesome_feature) +start-date: (fill me in with today's date, YYYY-MM-DD) +author: (name of the main author) +co-authors: (find a buddy later to help our with the RFC) +related-issues: (will contain links to implementation PRs) +--- + +# Summary +[summary]: #summary + +One paragraph explanation of the feature. + +# Motivation +[motivation]: #motivation + +Why are we doing this? What use cases does it support? What is the expected +outcome? + +# Detailed design +[design]: #detailed-design + +This is the bulk of the RFC. Explain the design in enough detail for somebody +familiar with the ecosystem to understand, and implement. This should get +into specifics and corner-cases, and include examples of how the feature is +used. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +# Alternatives +[alternatives]: #alternatives + +What other designs have been considered? What is the impact of not doing this? + +# Unresolved questions +[unresolved]: #unresolved-questions + +What parts of the design are still TBD or unknowns? + +# Future work +[future]: #future-work + +What future work, if any, would be implied or impacted by this feature +without being directly part of the work? From 9bec84d3ddce178356eb2d95eff9625c70a5654e Mon Sep 17 00:00:00 2001 From: Will Dietz Date: Mon, 19 Feb 2018 18:49:36 -0600 Subject: [PATCH 2/8] musl-libc RFC: first draft --- rfcs/0000-musl-libc.md | 128 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 116 insertions(+), 12 deletions(-) diff --git a/rfcs/0000-musl-libc.md b/rfcs/0000-musl-libc.md index fa5e28498..520d3c22b 100644 --- a/rfcs/0000-musl-libc.md +++ b/rfcs/0000-musl-libc.md @@ -1,39 +1,143 @@ --- -feature: (fill me in with a unique ident, my_awesome_feature) -start-date: (fill me in with today's date, YYYY-MM-DD) -author: (name of the main author) +feature: musl-libc +start-date: 2018-02-19 +author: Will Dietz co-authors: (find a buddy later to help our with the RFC) -related-issues: (will contain links to implementation PRs) +related-issues: [big musl PR](https://github.com/NixOS/nixpkgs/pull/34645), [issues matching "musl", newest first](https://github.com/NixOS/nixpkgs/search?o=desc&q=musl&s=created&type=Issues&utf8=%E2%9C%93), [2015 libc discussion](https://github.com/NixOS/nixpkgs/issues/6221#issuecomment-116754223) --- # Summary [summary]: #summary -One paragraph explanation of the feature. +When targeting Linux platforms, Nixpkgs builds software against +the defacto standard of Linux libc implementations: +[glibc](https://www.gnu.org/software/libc/). + +This RFC proposes supporting the use of an alternative libc implementation, +[musl](https://www.musl-libc.org/), for the reasons outlined below. +Adding this support is similar to introducing support for an architecture, +and realistically will be limited in compatibility and features +compared to non-musl siblings such as `x86_64-unknown-linux-gnu`. + +Initial support is already available in nixpkgs master, capable +of building up through important packages such as nixUnstable itself +and LLVM. This is not to be taken as an endorsement: discussing +and ultimately deciding whether support for musl should be part of nixpkgs +is the subject of this RFC. That said, this initial support is +a reasonable foundation for evaluating technical details discussed below, +and a convenient way for interested parties to explore the work so far. + +To help ensure we're all on the same page, unless otherwise specified +assume references to musl support implementation are in reference +to this commit (latest master at time of writing): + +[bd7d5b365799a145717d31122ebfd24b51fea117](https://github.com/NixOS/nixpkgs/commit/bd7d5b365799a145717d31122ebfd24b51fea117) # Motivation [motivation]: #motivation -Why are we doing this? What use cases does it support? What is the expected -outcome? +## Why Musl? +There are many reasons to prefer the use of musl, +a somewhat outdated overview comparing musl against other +implementations is available [here](http://www.etalabs.net/compare_libcs.html). +Note this comparison is maintained by the (primary) author of musl, +(as indicated at the top of the page). + +musl is advertised as being: +* lightweight +* fast +* simple +* free +* correctness: standards-conforming +* correctness: safety + +Additionally it is very popular when statically linking software, +creating binaries capable of executing most anywhere. + +In fact it is for this reason that Nixpkgs itself builds +the bootstrap busybox using musl. + +I'm unable to find good numbers but musl is "arguably" the second +most popular libc implementation on Linux, and is used +by a number of important projects you may be familiar with +large userbases, including: +* Alpine Linux - [#70 on Distrowatch](https://distrowatch.com/table.php?distribution=alpine) but very popular amongst docker users for producing slim container images. +* [OpenWRT/LEDE](https://openwrt.org/) - #1 open-source Linux router firmware project; foundation of most other projects targetting routers. +More projects and details of how they use musl can be found here: + +https://wiki.musl-libc.org/projects-using-musl.html + +## Why Nixpkgs? + +The importance of musl is not the primary point of contention in this RFC, +instead perhaps the main question is whether such support belongs in Nixpkgs or not. + +The main arguments for inclusion are: +* convenience of use +* foundation for exciting future work: musl is widely used by high-level languages + as the libc implementation used to produce statically linked programs: + these are easy to deploy, launch quickly, and only include the code required. + (NOTE: currently musl support prefers dynamic linking and shared libraries + as is the strong preference in Nixpkgs) +* Software sometimes must be patched to compile or run with musl; in @dtzWill's experience, + these changes are largely fixes improving compliance and correctness resulting in + higher-quality programs. Recent versions of glibc have started taking stronger stances + on enforcing compliance (look at patch fallout folllowing any glibc upgrade in last year or so) + resulting in overlapping work from both sides. + +And to a large extent: +"Why not?" -- similar to including support for architectures such as Aarch64 or RISC-V, + and just like support for those architectures it's relatively clear that pushing them + into private forks would be detrimental to the nixpkgs project as well as all users + interested in using Nixpkgs on those platforms/architectures. + +musl is clearly useful for a variety of important use cases, +however including support has a few costs (see Drawbacks, below): +do folks believe the costs are too high? + + +## Additional Resources + +* [musl FAQ](https://www.musl-libc.org/faq.html) +* [projects using musl](https://wiki.musl-libc.org/projects-using-musl.html) +* [Slides from a talk discussing various libcs, 2014](http://events17.linuxfoundation.org/sites/events/files/slides/libc-talk.pdf) + # Detailed design [design]: #detailed-design -This is the bulk of the RFC. Explain the design in enough detail for somebody -familiar with the ecosystem to understand, and implement. This should get -into specifics and corner-cases, and include examples of how the feature is -used. +I believe the linked implementation from current nixpkgs master is perhaps +the best embodiment of "Detailed Design"? + +Linky: [bd7d5b365799a145717d31122ebfd24b51fea117](https://github.com/NixOS/nixpkgs/commit/bd7d5b365799a145717d31122ebfd24b51fea117) # Drawbacks [drawbacks]: #drawbacks Why should we *not* do this? +Potential maintenance burden, particularly regarding collections of patches, +seems to be the primary concern. + +This RFC was prompted by concerns about the drawbacks: +["I'm really not in favour of adding support to Nixpkgs"](https://github.com/NixOS/nixpkgs/pull/34645#issuecomment-366789321). +This comment echoes very similar concerns expressed [back in 2015](https://github.com/NixOS/nixpkgs/issues/6221#issuecomment-116754223). + +## Rebuttal + +I believe the burden is not nearly as high as believed, for a few reasons: + +* musl is a "real" libc in terms of features, with improved safety in resource-constrained environments. +* glibc has increasingly enforced compliance, resulting in many packages being updated quickly or being patched by many other distributions as well. +* Nixpkgs has grown, making this much easier: "libc" abstraction is needed to properly support platforms such as Darwin and particularly the work + providing support for cross-compilation greatly simplifies the changes needed to support musl. + (the "big musl PR" is largely fixing cross-compilation further) + # Alternatives [alternatives]: #alternatives -What other designs have been considered? What is the impact of not doing this? +* No musl libc support. +* Maintained in a separate fork # Unresolved questions [unresolved]: #unresolved-questions From 9dce27f4dcba2520bfc32a76b03675e3120bf088 Mon Sep 17 00:00:00 2001 From: Will Dietz Date: Mon, 19 Feb 2018 18:53:13 -0600 Subject: [PATCH 3/8] musl-libc RFC: rename to patch pull req number --- rfcs/{0000-musl-libc.md => 0023-musl-libc.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename rfcs/{0000-musl-libc.md => 0023-musl-libc.md} (100%) diff --git a/rfcs/0000-musl-libc.md b/rfcs/0023-musl-libc.md similarity index 100% rename from rfcs/0000-musl-libc.md rename to rfcs/0023-musl-libc.md From 1fa3fb3e8737576919bd307fb5bae80b41504d40 Mon Sep 17 00:00:00 2001 From: Will Dietz Date: Mon, 19 Feb 2018 19:01:50 -0600 Subject: [PATCH 4/8] musl-libc: whoops, no links in top bit, that's metadata. Move links to later. --- rfcs/0023-musl-libc.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/rfcs/0023-musl-libc.md b/rfcs/0023-musl-libc.md index 520d3c22b..8ebe888be 100644 --- a/rfcs/0023-musl-libc.md +++ b/rfcs/0023-musl-libc.md @@ -3,9 +3,11 @@ feature: musl-libc start-date: 2018-02-19 author: Will Dietz co-authors: (find a buddy later to help our with the RFC) -related-issues: [big musl PR](https://github.com/NixOS/nixpkgs/pull/34645), [issues matching "musl", newest first](https://github.com/NixOS/nixpkgs/search?o=desc&q=musl&s=created&type=Issues&utf8=%E2%9C%93), [2015 libc discussion](https://github.com/NixOS/nixpkgs/issues/6221#issuecomment-116754223) +related-issues: 34645, 6221, ... --- + + # Summary [summary]: #summary @@ -102,6 +104,11 @@ do folks believe the costs are too high? * [projects using musl](https://wiki.musl-libc.org/projects-using-musl.html) * [Slides from a talk discussing various libcs, 2014](http://events17.linuxfoundation.org/sites/events/files/slides/libc-talk.pdf) +## Related Isssues + +* [big musl PR](https://github.com/NixOS/nixpkgs/pull/34645) +* [issues matching "musl", newest first](https://github.com/NixOS/nixpkgs/search?o=desc&q=musl&s=created&type=Issues&utf8=%E2%9C%93) +* [2015 libc discussion](https://github.com/NixOS/nixpkgs/issues/6221#issuecomment-116754223) # Detailed design [design]: #detailed-design From ccc7a1f132ee51dcd49199f82cb6f9a62836e519 Mon Sep 17 00:00:00 2001 From: Will Dietz Date: Tue, 20 Feb 2018 20:42:34 -0600 Subject: [PATCH 5/8] record the name of a brave co-author --- rfcs/0023-musl-libc.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/0023-musl-libc.md b/rfcs/0023-musl-libc.md index 8ebe888be..25c119d66 100644 --- a/rfcs/0023-musl-libc.md +++ b/rfcs/0023-musl-libc.md @@ -1,8 +1,8 @@ --- feature: musl-libc start-date: 2018-02-19 -author: Will Dietz -co-authors: (find a buddy later to help our with the RFC) +author: Will Dietz (@dtzWill) +co-authors: Shea Levy (@shlevy) related-issues: 34645, 6221, ... --- From 433d0ab23599caddd3206374e2408448f14584fc Mon Sep 17 00:00:00 2001 From: Will Dietz Date: Wed, 21 Feb 2018 15:55:02 -0600 Subject: [PATCH 6/8] musl rfc: squash second batch of changes --- rfcs/0023-musl-libc.md | 215 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 190 insertions(+), 25 deletions(-) diff --git a/rfcs/0023-musl-libc.md b/rfcs/0023-musl-libc.md index 25c119d66..2d2c33aba 100644 --- a/rfcs/0023-musl-libc.md +++ b/rfcs/0023-musl-libc.md @@ -15,8 +15,9 @@ When targeting Linux platforms, Nixpkgs builds software against the defacto standard of Linux libc implementations: [glibc](https://www.gnu.org/software/libc/). -This RFC proposes supporting the use of an alternative libc implementation, -[musl](https://www.musl-libc.org/), for the reasons outlined below. +This RFC proposes adding **experimental** support in Nixpkgs for the use +of an alternative libc implementation, [musl](https://www.musl-libc.org/), +for the reasons outlined below. Adding this support is similar to introducing support for an architecture, and realistically will be limited in compatibility and features compared to non-musl siblings such as `x86_64-unknown-linux-gnu`. @@ -39,11 +40,7 @@ to this commit (latest master at time of writing): [motivation]: #motivation ## Why Musl? -There are many reasons to prefer the use of musl, -a somewhat outdated overview comparing musl against other -implementations is available [here](http://www.etalabs.net/compare_libcs.html). -Note this comparison is maintained by the (primary) author of musl, -(as indicated at the top of the page). +There are many reasons to prefer the use of musl. musl is advertised as being: * lightweight @@ -59,6 +56,11 @@ creating binaries capable of executing most anywhere. In fact it is for this reason that Nixpkgs itself builds the bootstrap busybox using musl. +A somewhat outdated overview comparing musl against other +implementations is available [here](http://www.etalabs.net/compare_libcs.html). +Note this comparison is maintained by the (primary) author of musl, +(as indicated at the top of the page). + I'm unable to find good numbers but musl is "arguably" the second most popular libc implementation on Linux, and is used by a number of important projects you may be familiar with @@ -75,8 +77,8 @@ The importance of musl is not the primary point of contention in this RFC, instead perhaps the main question is whether such support belongs in Nixpkgs or not. The main arguments for inclusion are: -* convenience of use -* foundation for exciting future work: musl is widely used by high-level languages +* **convenience of use** +* **foundation for exciting future work**: musl is widely used by high-level languages as the libc implementation used to produce statically linked programs: these are easy to deploy, launch quickly, and only include the code required. (NOTE: currently musl support prefers dynamic linking and shared libraries @@ -86,6 +88,7 @@ The main arguments for inclusion are: higher-quality programs. Recent versions of glibc have started taking stronger stances on enforcing compliance (look at patch fallout folllowing any glibc upgrade in last year or so) resulting in overlapping work from both sides. + (NOTE: use of glibc extensions or reliance on non-standard behavior is still common and unlikely to go away soon) And to a large extent: "Why not?" -- similar to including support for architectures such as Aarch64 or RISC-V, @@ -113,10 +116,109 @@ do folks believe the costs are too high? # Detailed design [design]: #detailed-design -I believe the linked implementation from current nixpkgs master is perhaps -the best embodiment of "Detailed Design"? +## Goals +### Laying the Foundation + +Implement the following in nixpkgs: + +* [x] musl-based bootstrap +* [x] stdenv for native musl building +* [x] cross-musl stdenv + +These are already implemented and are currently tested +to build and provide basic functionality as part +of release-cross.nix. + +These features would be very difficult to implement +or maintain externally, and near impossible as an overlay. + +## Package Compatibility + +For a variety of reasons many packages do not work out-of-the-box +in musl-based environments. + +### "Normalization" + +Vast majority of the problems here are "minor" and are the +sort of problem we regularly encounter and address when +bumping to a new glibc version, new gcc version, or using +a clang-based stdenv (such as on Darwin). + +I'm calling these fixes "normalization". +These are changes like "adding a missing include" or +"don't assume compiler is invoked 'gcc'". + +These changes usually can be safely applied on all platforms +(although sometimes they are not for rebuild reasons) +and are easy to check for correctness or at least "couldn't-possibly-hurt". + +### Big Incompatibilities + +Some packages are very much not portable and require significant +and invasive changes to work with environments they don't expect. + +In the context of this RFC's proposed musl support, +there are a number of packages that are known to be in this category: + +* systemd +* ... + +This RFC proposes ignoring those for the immediate future, +to be revisited later, and focuses on systemd. + +#### Systemd + +Currently many packages depend on systemd. +This dependency is indirect for all but a handful of packages, +with a few key pieces of software integrating with systemd. + +As far as I know this dependency is generally optional, +and so we could easily avoid its use when using musl. + +This makes it possible to build a great number of packages +(thousands) but more complicated software "ecosystems" +and "desktop environments" will not work without something +to tie them together with the various roles played by systemd. + +Addressing this in any way is not in the scope of this RFC. + +Similarly, NixOS itself (especially services) require systemd +and we do not propose altering this. + +An early version of the "musl PR" patched systemd so that it +would build successfully, using patches from OpenEmbedded.org. + +The result was never tested or reviewed in terms of providing +basic functionality or general suitability for Nixpkgs/NixOs. + +(OE folks do great work, but they may expect rather different +things from systemd or workaround introduced shortcomings elsewhere +in various capacities) + +## Scope + +Primarily non-GUI packages for now, due to systemd blocker. + +In the future these will be supported. -Linky: [bd7d5b365799a145717d31122ebfd24b51fea117](https://github.com/NixOS/nixpkgs/commit/bd7d5b365799a145717d31122ebfd24b51fea117) +This RFC is primarily concerned with the groundwork for using musl at all. + +## Testing and Maintenance + +"Ideally" the answer would be an infinite number of builders would constantly +build all the things on all the platforms. + +Unfortunately this is unrealistic due to capacity constraints and other reasons. + +### Responsibility + +"musl team" is reponsible, to be formed. + +### Infrastructure + +Build at least stdenv with more being added in the future. + +Jobs may be given lower priority/shares. # Drawbacks [drawbacks]: #drawbacks @@ -126,33 +228,96 @@ Why should we *not* do this? Potential maintenance burden, particularly regarding collections of patches, seems to be the primary concern. -This RFC was prompted by concerns about the drawbacks: -["I'm really not in favour of adding support to Nixpkgs"](https://github.com/NixOS/nixpkgs/pull/34645#issuecomment-366789321). -This comment echoes very similar concerns expressed [back in 2015](https://github.com/NixOS/nixpkgs/issues/6221#issuecomment-116754223). +## Additional Costs + +* Maintenance +* Infrastructure (build pressure, storage, ...) + +## Fractured Community -## Rebuttal +> Another issue: adding musl support fractures the Nixpkgs user/development community: some people will run musl-based packages and some will run glibc-based packages. As a result all of Nixpkgs/NixOS will end up being less tested. it doubles the test matrix on Linux, after all. -I believe the burden is not nearly as high as believed, for a few reasons: +## Previous Discussion of drawbacks and concerns -* musl is a "real" libc in terms of features, with improved safety in resource-constrained environments. -* glibc has increasingly enforced compliance, resulting in many packages being updated quickly or being patched by many other distributions as well. -* Nixpkgs has grown, making this much easier: "libc" abstraction is needed to properly support platforms such as Darwin and particularly the work - providing support for cross-compilation greatly simplifies the changes needed to support musl. - (the "big musl PR" is largely fixing cross-compilation further) +This RFC was prompted by concerns about the drawbacks: +["I'm really not in favour of adding support to Nixpkgs"](https://github.com/NixOS/nixpkgs/pull/34645#issuecomment-366789321). +This comment echoes very similar concerns expressed [back in 2015](https://github.com/NixOS/nixpkgs/issues/6221#issuecomment-116754223). # Alternatives [alternatives]: #alternatives +* Maintain in a separate fork + * [SLNOS project is willing to adopt](https://github.com/NixOS/nixpkgs/pull/34645#issuecomment-366845015) +* Maintained as an overlay * No musl libc support. -* Maintained in a separate fork + * Not really an "alternative" :). # Unresolved questions [unresolved]: #unresolved-questions What parts of the design are still TBD or unknowns? +## Support + +We need to work on defining: + +* What "Support" entails +* Responsibility +* Blame? + +## Impact + +### Infrastructure + +* Hydra +* ofborg + +### Complexity + +* evaluation complexity + * cost of behind-the-scenes "magic" required +* keeping expressions avoidable + * cyclomatic complexity + +## How to Remove? + +Is there a good way to move forward +without becoming impossibly intertwined? +Such that a future party could +* reduce nixpkgs to what it "would be" without musl support +* Do so confidently without worrying about subtle + breakages? + +Maintaining entirely as an overlay (or fork?) +is an obviously effective solution in this regard. +Clear separation and enforced use of carefully crafted +interfaces/abstractions may also help with this. + +To some extent the importance of this depends +on how likely the community expects to find itself +"regretting" or wanting to be "rid" of musl support. + +However the design and use of good abstractions +is valuable in all cases :). + # Future work [future]: #future-work -What future work, if any, would be implied or impacted by this feature -without being directly part of the work? +### Fetch, Unpack, Patch + +(TODO: Split to new RFC?) +It may be possible to leverage proper use of "phases" so that +we can provide reasonable coverage of the unpack and patch +phases for all "supported" configurations. + +As an example, this would make it possible for our x86_64 builders and users to +get feedback ensuring that changes didn't break hashes or patch application +elsewhere without requiring builders of each configuration. + +The benefit of this would be in avoiding most of the burden of building everything +while making it easy to catch the most common sort of problems +so they can be addressed ("oops I didn't update the hash for darwin") +or flagged for investigation. + +I believe there's a branch or PR trying this somewhere. +Regardless, out of scope for this RFC. From 8599d40c16335e9ac8bf1593af7c75e0462f316b Mon Sep 17 00:00:00 2001 From: Will Dietz Date: Wed, 28 Feb 2018 18:07:18 -0600 Subject: [PATCH 7/8] rfc: specify initial team \o/, add a small clarification. --- rfcs/0023-musl-libc.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/rfcs/0023-musl-libc.md b/rfcs/0023-musl-libc.md index 2d2c33aba..c5363033c 100644 --- a/rfcs/0023-musl-libc.md +++ b/rfcs/0023-musl-libc.md @@ -212,7 +212,14 @@ Unfortunately this is unrealistic due to capacity constraints and other reasons. ### Responsibility -"musl team" is reponsible, to be formed. +"musl team" is reponsible, initially consisting of + +* @dtzWill +* @shlevy +* @domenkozar + +A team handle will be created to track this +and to ping the team on musl-related discussion or issues. ### Infrastructure @@ -265,6 +272,9 @@ We need to work on defining: * Responsibility * Blame? +For now we leave it as an informal understanding +which we can improve on in the future. + ## Impact ### Infrastructure From 6daaaefa5842a4d2f076a427e0552a718403fb66 Mon Sep 17 00:00:00 2001 From: Alexey Shmalko Date: Thu, 1 Mar 2018 20:48:30 +0200 Subject: [PATCH 8/8] Add myself to the "musl team" --- rfcs/0023-musl-libc.md | 1 + 1 file changed, 1 insertion(+) diff --git a/rfcs/0023-musl-libc.md b/rfcs/0023-musl-libc.md index c5363033c..d7e68511d 100644 --- a/rfcs/0023-musl-libc.md +++ b/rfcs/0023-musl-libc.md @@ -217,6 +217,7 @@ Unfortunately this is unrealistic due to capacity constraints and other reasons. * @dtzWill * @shlevy * @domenkozar +* @rasendubi A team handle will be created to track this and to ping the team on musl-related discussion or issues.