Github action for building solc 0.3.6 - 0.6.8 on MacOS

[cameel]

Part of ethereum/solidity#9258.

This PR adds a github action that randomly selects one of the missing MacOS binaries, builds it and pushes it to a branch. When merged it will start building on schedule (every 15 minutes) until they're all available. The branch is meant to be later submitted in a separate PR (manually, after updating file lists). Then the action can be disabled or removed.

I also have the binaries already built in my own fork (#54) so maybe we want to take them instead.
I'm submitting it anyway so that I can get it reviewed and to preserve the script in case we ever need to rebuild again.

Points of note:

• I was able to make it work for versions down to 0.3.6. Below that it should still be doable but it's much more complex. The build configuration is shared with old versions of cpp-ethereum and there are many dependencies that are hard to track down in binary form and would have to be built from source.
• Below 0.5.13 there's no script for easily installing dependencies and I'm always linking with Z3 4.8.5.
• I managed to statically link even versions below 0.4.6 by backporting a change that added dynamic linking for jsoncpp.
  • Note that MacOS builds are not completely static. They all still dynamically link to libSystem and libc++ but that's the case even with 0.7.x builds.
• The script runs tests for each version but in the older ones the tests required a running cpp-ethereum/aleth node. I was not able to run these. Versions that are even older only have command-line tests or no tests at all - in that case I'm just doing a simple smoke test of running the compiler on a minimal contract. SMT tests are also disabled in old versions (not sure about newer ones).
• I did not do bytecode comparisons.

[cameel]

Here's a dump of my notes from working on these builds. Might help someone trying to build an older version on his Mac.

Dependencies

Static linking

Builds on OSX are not completely static. These two dynamic libraries are always present:

/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)

Z3

Our CMake config can link Z3 both dynamically and statically. If both are available, it depends on which is returned first by CMake's find_library(). In my builds it seems that the dynamic one gets chosen by default.

• Z3 from homebew provides both the dynamic and the static library.
• osx_install_dependencies.sh forces static linking by downloading a binary release of Z3 from github and only installing the static library.
• Similarly for Ubuntu we have a PPA that has a package containing only static Z3.

solc links against Z3 since vesion 0.4.16 where "experimental partial support for z3 smt checker" was introduced.

Boost

• Homebrew currently provides only the latest version of boost (1.73.0), which won't work with older solc (see Check why 0.4.x requires an older boost on Mac solidity#6349 (comment)).
• There are formulae for boost 1.60.0 and older but they're disabled (Error: boost@1.60 has been disabled because it is a versioned formula!).
• It's easy to build earlier versions by downloading earlier versions of the formula and running them locally
  • Boost 1.70.0 formula is still too new (see Check why 0.4.x requires an older boost on Mac solidity#6349 (comment))
  • Boost 1.60.0 formula fails because it depends on something else in homebrew that has been moved/removed.
  • Boost 1.63.0 formula worked for most of my builds but I eventually switched to 1.65.0.
  • Boost 1.65.0 formula works with all solc versions between 0.3.6 and 0.6.8. Likely with newer ones too.

jsoncpp

CMake config will download and build jsoncpp on its own. This works at least since 0.3.6, probably earlier. If you install the latest one with brew in addition to that (currently 1.9.3), your version will be used instead and, below 0.4.22, the build will fail with Undefined symbols for architecture x86_64 linker error. 0.4.23 will just refuse to use anything other than jsoncpp 1.7.7.

Version 0.5.0 adds scripts/install_obsolete_jsoncpp_1_7_4.sh, which automates the installation of older jsoncpp (ethereum/solidity@54f83ac). This version of jsoncpp works fine too (except for solc 0.4.23) but installing it doesn't seem really necessary. BTW, ./circleci/osx_install_dependencies.sh is used on CircleCI and runs this script it since 0.5.13.

Version-specific

solc 0.6.1

solc 0.6.1 and later compile without any workarounds.

solc 0.6.0

• This version stopped relying on buggy behavior of boost < 1.71.0 (See String output does not respect uppercase flag for hexadecimal base prefix boostorg/multiprecision#129) regarding serialization of multiprecision numbers as hexadecimal numbers. On solc versions earlier than this tests won't pass when compiled with boost >= 1.71.0. See Check why 0.4.x requires an older boost on Mac solidity#6349 (comment) for details.

solc 0.5.13

• .circleci/osx_install_dependencies.sh introduced (ethereum/solidity@1609831).
  Before that .circleci/config.yml was installing latest z3 from homebrew and evmone was not being installed at all.
• Before this version, scripts/tests.sh was hard-coded to run with the --no-smt flag on MacOS (ethereum/solidity@c9a56d5).

solc 0.5.12

• There are tons of linker warnings like this in the log, but nothing appears to be actually broken:

  ld: warning: direct access in function (..) means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.
  

solc 0.5.11

• The first version to use evmone for testing.
• --no-ipc flag no longer available and not needed in soltest.
• direct access linker warnings like in 0.5.12.

solc 0.5.10

• scripts/tests.sh no longer runs soltest with --no-ipc flag on MacOS.
  • Due to this it tries to install and run a Linux version of aleth, then hangs. A workaround is to change hard-coded IPC_ENABLED value to false and IPC_FLAGS to --no-ipc.
  • If realpath is not installed (included in the coreutils homebrew package), it also prints an error about that (but still continues).
• direct access linker warnings like in 0.5.12.

solc 0.5.8 - 0.5.9

• direct access linker warnings like in 0.5.12.

solc 0.5.0

• This version switched from cpp-ethereum to aleth for IPC tests in scripts/tests.sh (ethereum/solidity@273be20).
• This version fixes a bug in the way boost unit tests headers are included. On 0.4.x this bug causes linker errors (duplicate symbol). The workaround is to replace #include <boost/test/included/unit_test.hpp> with #include <boost/test/unit_test.hpp> in test/boostTest.cpp.
  • Reported in Compile failed, got "ld: 31 duplicate symbols for architecture x86_64" solidity#3845. Fixed in Do not include single-header boost_unit_test solidity#4572.
  • Docs on the usage of these headers: https://www.boost.org/doc/libs/1_74_0/libs/test/doc/html/boost_test/usage_variants.html
  • In Compile failed, got "ld: 31 duplicate symbols for architecture x86_64" solidity#3845 someone claims that just switching from boost 1.66.0 to boost 1.65.1 helped. For me the errors appeared on both boost 1.63.0 and boost 1.65.0.
• Commandline tests are broken on MacOS due to buggy sed call in test/cmdlineTests.sh.

  too_long_line.sol:2:164: Error: Identifier not found or not unique.
  Commandline tests FAILED
  

  This was fixed in 0.5.1 (Fix commandline tests. solidity#5419) and the fix can be cleanly backported.

solc 0.4.24

• Starting with this version scripts/tests.sh runs soltest with --no-ipc flag on MacOS (ethereum/solidity@743a714)
• This version and later ones build without errors with jsoncpp 1.9.3.

solc 0.4.23

• This particular version will build only with jsoncpp 1.7.7. It has a debug_assert() that fails otherwise.

  /Users/runner/work/solc-bin/solc-bin/solidity/libdevcore/JSON.cpp:30:1: error: static_assert failed due to requirement '(1 == 1) && (9 == 7) && (3 == 7)' "Unexpected jsoncpp version: 1.9.3. Expecting 1.7.7."
  

solc 0.4.22

• Tests in CI now run also against constantinople EVM (ethereum/solidity@4450034).

solc 0.4.21

• Ability to select between byzantium and homestead EVM introduced (ethereum/solidity@83515ea).

solc 0.4.17

• deps/ git submodule removed.

solc 0.4.16

• STATIC_LINKING option in CMake replaced with SOLC_LINK_STATIC which has the same function (ethereum/solidity@4974ff3).

solc 0.4.12

• scripts/get_version.sh added (ethereum/solidity@f13feed).
  • It's not usable on MacOS anyway because it uses -P/--perl-regexp option with grep, which is not available on that system.

solc 0.4.11

• Introduces --no-ipc option in soltest (ethereum/solidity@ea6d925).

solc 0.4.10

• scripts/tests.sh runs solfuzzer (ethereum/solidity@8be318e).

solc 0.4.9

• Introduces cmdlineTests.sh (ethereum/solidity@605455f). scripts/tests.sh runs it.

solc 0.4.5

• deps/ submodule updated to a revision that contains jsoncpp.cmake module which downloads and builds jsoncpp from source at cmake configuration stage (Build jsoncpp from source using jsoncpp.cmake script solidity#1252).

solc 0.4.2 - 0.4.4

• These versions download and install prebuilt jsoncpp from https://github.com/ethereum/cpp-dependencies/releases/download/vc140/jsoncpp-1.6.2.tar.gz.
• jsoncpp is linked dynamically. Build jsoncpp from source using jsoncpp.cmake script solidity#1252 can be cleanly backported to get static linking with 1.7.7 instead.

solc 0.4.0 - 0.4.1

• Same situation with jsoncpp as in 0.4.2 but Add Centos to install_deps.sh solidity#1085 needs to be backported too due to conflicts.

solc 0.3.6

• deps/ git submodule introduced.
• tests/boostTest.cpp has a main() function. Due to this linking fails with duplicate symbol '_main', even though it defines BOOST_TEST_NO_MAIN.
  • This affects only 0.3.6. Neither 0.3.5 nor 0.4.0 has its own main().
  • Cleanup of test runner. solidity#837 from 0.4.0 removes main() and can be cleanly backported to 0.3.6.
• Same situation with jsoncpp as in 0.4.0 but there are a few more PRs that needs to be backported due to conflicts: Added missing "stop on error" to scripts/install.sh solidity#1035, solc crashes without 'export LC_ALL=C' solidity#975, Stronger Linux distro detection solidity#914.

solc 0.3.0

• CMake configuration depends on https://github.com/ethereum/webthree-helpers. The necessary files have been removed from the latest revision, though they're still available in history.
• At this point cmake configuration was shared with cpp-ethereum. There are lots of extra dependencies and most of them probably not really needed for Solidity.

solc 0.1.2

• CMake configuration depends on https://github.com/ethereum/cpp-ethereum-cmake

Attempts to run IPC tests with cpp-ethereum/aleth

• Tests in 0.4.0 fail with Aleth 1.5.0 binary release for MacOS

  /Users/runner/work/solc-bin/solc-bin/solidity/test/RPCSession.cpp:312: fatal error: in "SolidityAuctionRegistrar/creation": Error on JSON-RPC call: INTERNAL_ERROR: : /Users/distiller/project/libethereum/ClientTest.cpp(72): Throw in function void dev::eth::ClientTest::setChainParams(const string &)
  Dynamic exception type: boost::exception_detail::clone_impl<dev::eth::ChainParamsInvalid>
  [dev::tag_comment*] = /Users/distiller/project/libdevcore/JsonUtils.cpp(89): Throw in function void dev::requireJsonFields(const json_spirit::mObject &, const std::string &, const std::map<std::string, JsonFieldOptions> &)
  Dynamic exception type: boost::exception_detail::clone_impl<dev::MissingField>
  [dev::tag_comment*] = Expected field 'difficulty' not found in config: ChainParams::loadConfig::genesis
  

  • That's because in that version of aleth the difficulty field in the RPC message that creates the genesis block became mandatory. Solidity started adding this field in 0.5.0 (ethereum/solidity@0319b18). IPC tests in earlier solc versions will only run with earlier versions of aleth.
• Aleth 1.2.9 binary release for MacOS won't run because it's linked against system-wide libsnappy.1.dylib rather than the one included in the package.

  dyld: Library not loaded: /usr/local/lib/libsnappy.1.dylib
  Referenced from: /Users/runner/work/solc-bin/solc-bin/cpp-ethereum/ethbin/libleveldb.1.dylib
  Reason: image not found
  

• Aleth 1.4.0 is not statically linked and the release package does not include the dependencies.
• Aleth 1.3.0 is referenced in various places, including Reddit (RELEASED - cpp-ethereum-v1.3.0 ("Homecoming")) but the binary release is gone. It's not in the aleth repo and the cpp-ethereum repo is now just a redirect to it.

[chriseth]

Looks good! I'll merge the PR containing the binaries. Please deactivate this job, then it can be merged.

[cameel]

@chriseth Sure. I'll do that tomorrow. Do we want to keep it after that or just merge to have it in history and then remove?

[chriseth]

Let's keep it - if it's just in the history, nobody will remember it is there.

[cameel]

Done and rebased. Now the action won't ever run unless someone modifies it.

[cameel]

Action updated not to use set-env (see #60).