Upload to Debian upstream

[McSinyx]

Bintray is sunsetting, and even if another hosting facility is found, it'll still be better if zig is available in Debian's repo. I understand that the uploading process to Debian is tedious, but I'd love to see it happens 🥺

[SmiVan]

Shouldn't Bintray sunsetting be a separate issue from this?

[dankamongmen]

I'm a Debian Developer interested in making this happen. I know there's an existing ITP for zig, let me find it...

https://lists.debian.org/debian-mentors/2022/06/msg00023.html here we go

it looks like current zig is 0.14.0, whereas most of this work was from the 0.9.1 era.

let me take a look and see if the copyright situation has improved since 2022.

[McSinyx]

it looks like current zig is 0.14.0, whereas most of this work was from the 0.9.1 era.

Source bootstrapping Zig requires a little bit more effort now as the upstream checks in the WASM build artifact as the bootstrapping mechinism which is updated rather regularly between releases. This indicates the Zig compiler has to be built roughly that number of times to reach the latest version.

Guix's demonstrated this boostrapping path is certainly possible though.

[dankamongmen]

i definitely wouldn't want to be doing the full bootstrapping procedure (despite claims from back in the day); that's just for if you're missing builddeps, right? we'd want to be pulling anything vendored out and using debian-packaged LLVM and glibc. i just built ziglang/zig.git on debian unstable just fine in a single go...am i missing something?

[McSinyx]

The stage1/zig1.wasm file (used in bootstrap.c) is the build result of some previous version of the source code. Using it would be akin to using a gcc binary included in its source tarball to build gcc.

[dankamongmen]

thanks, grokked

[dankamongmen]

[schwarzgerat](0) $ file stage1/zig1.wasm 
stage1/zig1.wasm: WebAssembly (wasm) binary module version 0x1 (MVP)
[schwarzgerat](0) $ 

yeah

[dankamongmen]

so (not having looked at this in any depth) i'm thinking the way to go is a zig-bootstrap package which builds stage1/zig1.wasm as described in the link you provided. that's the only file it installs. then there's a zig package which build-deps and deps on zig-bootstrap. it basically just runs cmake like any other cmake-based debian package.

we keep all the hairy multistage stuff in zig-bootstrap, which hopefully won't be changing very often (again, as suggested by @andrewrk's post). that way people can easily touch the zig package, which is presumably changing more often.

we could do it all as one package, but that would add load to the build servers in the case of frequent zig rebuilds, and possibly intimidate collaborators heh.

[dankamongmen]

so it looks like in zig-bootstrap we'd need yank out the vendored lld, llvm, zstd, and zlib. let's see what we can do with that.

[dankamongmen]

alright i've got zig-bootstrap building using the host toolchain/libs

[dankamongmen]

so it looks like guix is bootstrapping 0.13 by pulling down their 0.12 zig compiler, and using that to build things, if i understand correctly: https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/zig.scm#n1572

i'm stuck at the moment trying to build zig1.wasm. i didn't realize it's present as a blob in both zig-bootstrap and zig (at least as of the 0.13 releases i'm working with).

[dankamongmen]

so the main sticking point is building this zig1.wasm. i don't see from @andrewrk's blog post how this was done. looking again tonight...

[dankamongmen]

The wasm binary is produced with zig build update-zig1

hopefully this is the key to things. but to do that we need some initial zig, right?

We provide a minimal WASI interpreter implementation that is built from C source, and then used to translate the Zig self-hosted compiler source code into C code. The C code is then compiled and linked, again by the system C compiler, into a stage2 binary. The stage2 binary can then be used repeatedly with zig build to build from source from that point on.

possibly i can build a zig-12 or whatever package that just requires llvm to build. that would be a builddep of the zig-bootstrap package, which will (as described above) create zig1.wasm using this older zig.

[dankamongmen]

going backwards looking for a zig-bootstrap without an embedded zig1.wasm...

12.1 -- has it
11.0 -- has it
0.10.1 -- clean

[dankamongmen]

initial experiments suggest that 0.10.1 is pretty closely tied to LLVM15 (see Findllvm.cmake in zig/cmake, which tests that LLVM is > 14 and < 16). llvm-15 is not in debian unstable (it does have llvm-14 and llvm-17+). naive build with llvm-14 definitely breaks. gonna try with 17 and see what's up.

llvm-14:

[ 82%] Building CXX object CMakeFiles/zigcpp.dir/src/zig_llvm-ar.cpp.o
/home/dank/src/dsscaw/zig-10-bootstrap-0.10.1+dfsg.1/zig/src/zig_llvm-ar.cpp: In function ‘bool isValidInBitMode(const A&)’:
/home/dank/src/dsscaw/zig-10-bootstrap-0.10.1+dfsg.1/zig/src/zig_llvm-ar.cpp:697:24: error: ‘getDefaultKindForHost’ is not a member of ‘llvm::object::Archive’
  697 |   if (object::Archive::getDefaultKindForHost() != object::Archive::K_AIXBIG)
      |                        ^~~~~~~~~~~~~~~~~~~~~
/home/dank/src/dsscaw/zig-10-bootstrap-0.10.1+dfsg.1/zig/src/zig_llvm-ar.cpp: In function ‘void performWriteOperation(ArchiveOperation, llvm::object::Archive*, std::unique_ptr<llvm::MemoryBuffer>, std::vector<llvm::NewArchiveMember>*)’:
/home/dank/src/dsscaw/zig-10-bootstrap-0.10.1+dfsg.1/zig/src/zig_llvm-ar.cpp:1047:47: error: ‘__gnu_cxx::__alloc_traits<std::allocator<llvm::NewArchiveMember>, llvm::NewArchiveMember>::value_type’ {aka ‘struct llvm::NewArchiveMember’} has no member named ‘detectKindFromObject’
 1047 |           InferredKind = NewMembersP->front().detectKindFromObject();
      |                                               ^~~~~~~~~~~~~~~~~~~~
/home/dank/src/dsscaw/zig-10-bootstrap-0.10.1+dfsg.1/zig/src/zig_llvm-ar.cpp:1049:45: error: ‘__gnu_cxx::__alloc_traits<std::allocator<llvm::NewArchiveMember>, llvm::NewArchiveMember>::value_type’ {aka ‘struct llvm::NewArchiveMember’} has no member named ‘detectKindFromObject’
 1049 |           InferredKind = NewMembers.front().detectKindFromObject();
      |                                             ^~~~~~~~~~~~~~~~~~~~
/home/dank/src/dsscaw/zig-10-bootstrap-0.10.1+dfsg.1/zig/src/zig_llvm-ar.cpp:1054:59: error: ‘__gnu_cxx::__alloc_traits<std::allocator<llvm::NewArchiveMember>, llvm::NewArchiveMember>::value_type’ {aka ‘struct llvm::NewArchiveMember’} has no member named ‘detectKindFromObject’
 1054 |       Kind = !NewMembersP->empty() ? NewMembersP->front().detectKindFromObject()
      |                                                           ^~~~~~~~~~~~~~~~~~~~
/home/dank/src/dsscaw/zig-10-bootstrap-0.10.1+dfsg.1/zig/src/zig_llvm-ar.cpp:1055:55: error: ‘getDefaultKindForHost’ is not a member of ‘llvm::object::Archive’
 1055 |                                    : object::Archive::getDefaultKindForHost();
      |                                                       ^~~~~~~~~~~~~~~~~~~~~
/home/dank/src/dsscaw/zig-10-bootstrap-0.10.1+dfsg.1/zig/src/zig_llvm-ar.cpp:1057:55: error: ‘__gnu_cxx::__alloc_traits<std::allocator<llvm::NewArchiveMember>, llvm::NewArchiveMember>::value_type’ {aka ‘struct llvm::NewArchiveMember’} has no member named ‘detectKindFromObject’
 1057 |       Kind = !NewMembers.empty() ? NewMembers.front().detectKindFromObject()
      |                                                       ^~~~~~~~~~~~~~~~~~~~
/home/dank/src/dsscaw/zig-10-bootstrap-0.10.1+dfsg.1/zig/src/zig_llvm-ar.cpp:1058:53: error: ‘getDefaultKindForHost’ is not a member of ‘llvm::object::Archive’
 1058 |                                  : object::Archive::getDefaultKindForHost();
      |                                                     ^~~~~~~~~~~~~~~~~~~~~
/home/dank/src/dsscaw/zig-10-bootstrap-0.10.1+dfsg.1/zig/src/zig_llvm-ar.cpp: In function ‘int ar_main(int, char**)’:
/home/dank/src/dsscaw/zig-10-bootstrap-0.10.1+dfsg.1/zig/src/zig_llvm-ar.cpp:1330:24: error: ‘getDefaultKindForHost’ is not a member of ‘llvm::object::Archive’
 1330 |   if (object::Archive::getDefaultKindForHost() == object::Archive::K_AIXBIG) {
      |                        ^~~~~~~~~~~~~~~~~~~~~
/home/dank/src/dsscaw/zig-10-bootstrap-0.10.1+dfsg.1/zig/src/zig_llvm-ar.cpp:1391:28: error: ‘getDefaultKindForHost’ is not a member of ‘llvm::object::Archive’
 1391 |       if (object::Archive::getDefaultKindForHost() ==
      |                            ^~~~~~~~~~~~~~~~~~~~~
gmake[2]: *** [CMakeFiles/zigcpp.dir/build.make:93: CMakeFiles/zigcpp.dir/src/zig_llvm-ar.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:137: CMakeFiles/zigcpp.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2
[schwarzgerat](2) $ 

[dankamongmen]

17+ lack llvm-c/Initialization.h, leading to the first breakage. let's take a look at the commit which checked in the Findllvm.cmake test.

[dankamongmen]

the commit is ziglang/zig-bootstrap@d3e821f but it looks like this has always been "specify the major version".

[dankamongmen]

alright, i got zig-10-bootstrap compiling using LLVM17 by copying back zig-13-bootstrap's zig/src/zig_llvm.* and adding set(CMAKE_CXX_STANDARD 17) to zig/CMakeLists.txt and hacking all the cmake/Find* to look for 17. let's see how far up we can take it! and whether the binary produced actually works to build zig1.wasm!

[dankamongmen]

alright, i got zig-10-bootstrap compiling using LLVM17 by copying back zig-13-bootstrap's zig/src/zig_llvm.* and adding set(CMAKE_CXX_STANDARD 17) to zig/CMakeLists.txt and hacking all the cmake/Find* to look for 17. let's see how far up we can take it! and whether the binary produced actually works to build zig1.wasm!

this does not appear to work with llvm19, at least not easily (all the code in stage1 goes to hell). let me move back to llvm17, ensure that's working, and move forward.

[dankamongmen]

hrmmm, stage1 refers to all kinds of crap from zig_llvm.* (first off, ZigLLVMDIFile) so this doesn't look like it's going to fly. we might have to eat shit and actually migrate the zig_llvm.* and zig_clang.* code forward to a newer version of LLVM, which is a somewhat forbidding prospect--certainly more work than i'd hoped to do this weekend.

[dankamongmen]

so it's looking like we'll need:

• resurrect llvm-15 (https://packages.debian.org/trixie/llvm-15) for sid
• build-dep zig-10-bootstrap on llvm-15 and friends
• zig-10-bootstrap builds a basic zig (i believe this to be zigcpp?)
• zig-bootstrap build-deps on zig-10-bootstrap
• zig-bootstrap uses zig to build zig1.wasm
• zig build-deps on zig-bootstrap
• zig builds modern zig

this will require getting ftpmasters (and probably llvm team) to buy in to introducing llvm-15 (but as noted, we already have llvm-14 in sid, but this still isn't a good thing), in addition to introducing zig-10-bootstrap and zig-bootstrap.

and we'd need to carry all this forward through eternity, it seems. if any of this toolchain gets knocked out, we lose everything up through zig.

let me take another look at guix, knowing what i know now, and make sure they're not doing something i'm missing.

alternatively, we could dump this into contrib =].

[dankamongmen]

there's a new bug in DBTS for this: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1092895

we're not going to have to use contrib, nor mess with zig-bootstrap or zig 10 at all. zig1.wasm is fine for main since the tools and source necessary to recreate it are available in the package.

i expect this to be in the NEW queue by the end of this week.

[dankamongmen]

honestly this repo ought probably be archived up. it talks about the 0.6.0 release on the front page, and this ought be in debian+ubuntu very soon. but do as you will!

[McSinyx]

zig1.wasm is fine for main since the tools and source necessary to recreate it are available in the package.

Reflections on trusting trust aside, you can't recreate zig1.wasm from Zig 0.13's source since the one included with that version was created from the source code a few weeks before.

[dankamongmen]

zig1.wasm is fine for main since the tools and source necessary to recreate it are available in the package.

Reflections on trusting trust aside, you can't recreate zig1.wasm from Zig 0.13's source since the one included with that version was created from the source code a few weeks before.

right, but we could use the resulting zig to build a fresh zig1.wasm with current source and use that to build a successive zig (as you say, Thompson attacks aside).