Compile failure when LD=lld or LD=gold, but not LD=mold or LD (unset, using bfd)

[skull-squadron]

Environment

• CentOS 9 Stream x86_64
• Clang/LLVM 14.0.6 via standard packages
• gold, mold, lld, and ld (bfd) via standard packages

Configuration

• Occurs in both in-tree and out-of-tree builds
• Note: Use case requires multiple options and santizers specific to lld

MCE error repro (either lld or gold)

export LD=lld CC=clang CXX=clang++ # or LD=gold
./autogen.sh && \
./configure --prefix=/tmp/blah \
  --disable-dependency-tracking \
  --enable-jit \
  --enable-jit-sealloc \
  --enable-pcre2-16 \
  --enable-pcre2-32 \
  --enable-shared \
  --enable-static && \
  make -j check

No error (either mold or ld (bfd))

export LD=mold CC=clang CXX=clang++ # or LD=ld (bfd)
./autogen.sh && \
./configure --prefix=/tmp/blah \
  --disable-dependency-tracking \
  --enable-jit \
  --enable-jit-sealloc \
  --enable-pcre2-16 \
  --enable-pcre2-32 \
  --enable-shared \
  --enable-static && \
  make -j check

Error output

...
  CCLD     libpcre2-32.la
  CCLD     libpcre2-16.la
  CCLD     libpcre2-8.la
  CCLD     libpcre2-posix.la
  CCLD     pcre2grep
  CCLD     pcre2_jit_test
clang-14: error: no such file or directory: './.libs/libpcre2-8.so'
make[1]: *** [Makefile:1786: pcre2grep] Error 1
make[1]: *** Waiting for unfinished jobs....
clang-14: error: no such file or directory: './.libs/libpcre2-8.so'
clang-14: error: no such file or directory: './.libs/libpcre2-16.so'
clang-14: error: no such file or directory: './.libs/libpcre2-32.so'
make[1]: *** [Makefile:1774: pcre2_jit_test] Error 1
make[1]: Leaving directory '/tmp/pcre2'
make: *** [Makefile:3211: check] Error 2

Success output

...
  CC       src/libpcre2_32_la-pcre2_study.lo
  CC       src/libpcre2_32_la-pcre2_substitute.lo
  CC       src/libpcre2_32_la-pcre2_serialize.lo
  CC       src/libpcre2_32_la-pcre2_string_utils.lo
  CC       src/libpcre2_32_la-pcre2_substring.lo
  CC       src/libpcre2_32_la-pcre2_tables.lo
  CC       src/libpcre2_32_la-pcre2_ucd.lo
  CC       src/libpcre2_32_la-pcre2_xclass.lo
  CC       src/libpcre2_32_la-pcre2_chartables.lo
  CC       src/libpcre2_32_la-pcre2_valid_utf.lo
  CC       src/pcre2_jit_test-pcre2_jit_test.o
  CCLD     libpcre2-32.la
  CCLD     libpcre2-16.la
  CCLD     libpcre2-8.la
  CCLD     libpcre2-posix.la
  CCLD     pcre2grep
  CCLD     pcre2_jit_test
  CCLD     pcre2test
make
make[2]: Entering directory '/tmp/pcre2'
make  all-am
make[3]: Entering directory '/tmp/pcre2'
make[3]: Nothing to be done for 'all-am'.
make[3]: Leaving directory '/tmp/pcre2'
make[2]: Leaving directory '/tmp/pcre2'
make  check-TESTS
make[2]: Entering directory '/tmp/pcre2'
make[3]: Entering directory '/tmp/pcre2'
PASS: RunGrepTest
PASS: RunTest
PASS: pcre2_jit_test
============================================================================
Testsuite summary for PCRE2 10.40
============================================================================
# TOTAL: 3
# PASS:  3
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[3]: Leaving directory '/tmp/pcre2'
make[2]: Leaving directory '/tmp/pcre2'
make[1]: Leaving directory '/tmp/pcre2'
[root@41861e6f5aec pcre2]#

Error `{{build}}/.libs/` contents

[root@41861e6f5aec pcre2]# ls -la .libs
total 2864
drwxr-xr-x  2 root root    4096 Nov 12 09:45 .
drwxr-xr-x 14 root root    4096 Nov 12 09:45 ..
-rw-r--r--  1 root root  935202 Nov 12 09:45 libpcre2-16.a
lrwxrwxrwx  1 root root      17 Nov 12 09:45 libpcre2-16.la -> ../libpcre2-16.la
-rw-r--r--  1 root root     946 Nov 12 09:45 libpcre2-16.lai
lrwxrwxrwx  1 root root      21 Nov 12 09:45 libpcre2-16.so -> libpcre2-16.so.0.11.0
lrwxrwxrwx  1 root root      21 Nov 12 09:45 libpcre2-16.so.0 -> libpcre2-16.so.0.11.0
-rw-r--r--  1 root root  900696 Nov 12 09:45 libpcre2-32.a
lrwxrwxrwx  1 root root      17 Nov 12 09:45 libpcre2-32.la -> ../libpcre2-32.la
-rw-r--r--  1 root root     946 Nov 12 09:45 libpcre2-32.lai
lrwxrwxrwx  1 root root      21 Nov 12 09:45 libpcre2-32.so -> libpcre2-32.so.0.11.0
lrwxrwxrwx  1 root root      21 Nov 12 09:45 libpcre2-32.so.0 -> libpcre2-32.so.0.11.0
-rw-r--r--  1 root root 1057664 Nov 12 09:45 libpcre2-8.a
lrwxrwxrwx  1 root root      16 Nov 12 09:45 libpcre2-8.la -> ../libpcre2-8.la
-rw-r--r--  1 root root     939 Nov 12 09:45 libpcre2-8.lai
lrwxrwxrwx  1 root root      20 Nov 12 09:45 libpcre2-8.so -> libpcre2-8.so.0.11.0
lrwxrwxrwx  1 root root      20 Nov 12 09:45 libpcre2-8.so.0 -> libpcre2-8.so.0.11.0
-rw-r--r--  1 root root    7346 Nov 12 09:45 libpcre2-posix.a
lrwxrwxrwx  1 root root      20 Nov 12 09:45 libpcre2-posix.la -> ../libpcre2-posix.la
-rw-r--r--  1 root root     992 Nov 12 09:45 libpcre2-posix.lai
lrwxrwxrwx  1 root root      23 Nov 12 09:45 libpcre2-posix.so -> libpcre2-posix.so.3.0.2
lrwxrwxrwx  1 root root      23 Nov 12 09:45 libpcre2-posix.so.3 -> libpcre2-posix.so.3.0.2
[root@41861e6f5aec pcre2]#

Success `{{build}}/.libs/` contents

[root@41861e6f5aec pcre2]# ls -la .libs
total 5320
drwxr-xr-x  2 root root    4096 Nov 12 09:49 .
drwxr-xr-x 14 root root    4096 Nov 12 09:49 ..
-rw-r--r--  1 root root  935202 Nov 12 09:49 libpcre2-16.a
lrwxrwxrwx  1 root root      17 Nov 12 09:49 libpcre2-16.la -> ../libpcre2-16.la
-rw-r--r--  1 root root     946 Nov 12 09:49 libpcre2-16.lai
lrwxrwxrwx  1 root root      21 Nov 12 09:49 libpcre2-16.so -> libpcre2-16.so.0.11.0
lrwxrwxrwx  1 root root      21 Nov 12 09:49 libpcre2-16.so.0 -> libpcre2-16.so.0.11.0
-rwxr-xr-x  1 root root  687288 Nov 12 09:49 libpcre2-16.so.0.11.0
-rw-r--r--  1 root root  900696 Nov 12 09:49 libpcre2-32.a
lrwxrwxrwx  1 root root      17 Nov 12 09:49 libpcre2-32.la -> ../libpcre2-32.la
-rw-r--r--  1 root root     946 Nov 12 09:49 libpcre2-32.lai
lrwxrwxrwx  1 root root      21 Nov 12 09:49 libpcre2-32.so -> libpcre2-32.so.0.11.0
lrwxrwxrwx  1 root root      21 Nov 12 09:49 libpcre2-32.so.0 -> libpcre2-32.so.0.11.0
-rwxr-xr-x  1 root root  654448 Nov 12 09:49 libpcre2-32.so.0.11.0
-rw-r--r--  1 root root 1057664 Nov 12 09:49 libpcre2-8.a
lrwxrwxrwx  1 root root      16 Nov 12 09:49 libpcre2-8.la -> ../libpcre2-8.la
-rw-r--r--  1 root root     939 Nov 12 09:49 libpcre2-8.lai
lrwxrwxrwx  1 root root      20 Nov 12 09:49 libpcre2-8.so -> libpcre2-8.so.0.11.0
lrwxrwxrwx  1 root root      20 Nov 12 09:49 libpcre2-8.so.0 -> libpcre2-8.so.0.11.0
-rwxr-xr-x  1 root root  765256 Nov 12 09:49 libpcre2-8.so.0.11.0
-rw-r--r--  1 root root    7346 Nov 12 09:49 libpcre2-posix.a
lrwxrwxrwx  1 root root      20 Nov 12 09:49 libpcre2-posix.la -> ../libpcre2-posix.la
-rw-r--r--  1 root root     992 Nov 12 09:49 libpcre2-posix.lai
lrwxrwxrwx  1 root root      23 Nov 12 09:49 libpcre2-posix.so -> libpcre2-posix.so.3.0.2
lrwxrwxrwx  1 root root      23 Nov 12 09:49 libpcre2-posix.so.3 -> libpcre2-posix.so.3.0.2
-rwxr-xr-x  1 root root   17136 Nov 12 09:49 libpcre2-posix.so.3.0.2
-rwxr-xr-x  1 root root   95944 Nov 12 09:49 pcre2_jit_test
-rwxr-xr-x  1 root root   71712 Nov 12 09:49 pcre2grep
-rwxr-xr-x  1 root root  210056 Nov 12 09:49 pcre2test
[root@41861e6f5aec pcre2]#

[skull-squadron]

Similar results occur with CC=gcc instead of CC=clang: mold and ld work, while lld and gold fail.

[PhilipHazel]

I hope somebody else who follows this project may be able to help you, because I can't. I presume lld, mold and gold are alternative linkers, but I know nothing about the nitty-gritty of linking. Nor am I at all knowledgeable in the autotools stuff - it was all contributed by someone who was; I've just about been able to tweak it from time to time.

[skull-squadron]

ld (bfd) is the standard binutils' linker.

gold is the binutils' newer linker.

lld is clang/LLVM's linker.

mold is the one diverging away from FOSS going commercial with a radioactive license despite crowdfunding, so we might stop using it.

My guess is both failing linkers try to create shared objects with the wrong names. strace and verbosity would be probably be able to resolve it. (Technical people doing "business" footguns.)

It's a learning opportunity then. ;) Right as you rip-replace it with bazel, conan, or cmake to save your sanity.

[PhilipHazel]

Learning opportunities are hard work at my age (and remembering what's been learned is another issue :-) but: I have installed lld on my Arch Linux box, and I can reproduce the failure with just "LD=lld ./configure". It isn't generating the basic shared library file. Not quite sure how to proceed, but I've no more time left today anyway.

[PhilipHazel]

It's a libtool issue. When you run ./configure, it creates a script called "libtool" which it subsequently runs to do linking and library building. I ran a default ./configure (which uses ld of course) and saved the libtool file it created. Then I ran with LD=lld and compared the two files. In addition to the different linker name, there were other differences. I edited the "good" version of the script, changing /usr/bin/ld to /usr/bin/lld and ran it. It worked! I will try to find out which setting in the libtool script causes the problem and then the task will be to find out why setting LD=/usr/bin/lld has this effect.

[PhilipHazel]

I have narrowed this down to one line in the libtool script. The "good" version sets a value for archive_cmds, whereas the "bad" version does not. The setting is:

archive_cmds="$CC -shared $pic_flag $libobjs $deplibs $compiler_flags $wl-soname $wl$soname -o $lib"

and the comment above it is "Commands used to build a shared archive", which figures. It would appear that the code that builds the "libtool" script behaves differently when LD is not GNU ld (there's a setting for that too, but it doesn't affect the outcome). I will try to research if there is any way to alter this behaviour; if not, I suppose a bug needs reporting somewhere.

[carenas]

Does using instead LD=ld.lld or LD=ld.gold work as a workaround?, at least in FreeBSD and OpenBSD that use lld as a system linker by default, it works and the following is printed in a Linux Debian system that had lld added :

$ lld --version
lld is a generic driver.
Invoke ld.lld (Unix), ld64.lld (macOS), lld-link (Windows), wasm-ld (WebAssembly) instead

[PhilipHazel]

LD=ld.lld works for me. If this is the correct way to invoke lld, it isn't a workaround!

[PhilipHazel]

/usr/bin/ld.lld on my system is just a symbolic link to /usr/bin/lld and yet setting LD to the first one works, but the second does not. There must be something weird going on inside "configure". However, it appears that this issue is not PCRE2's problem.

[thesamesam]

You generally should not be setting LD at all and should be setting -fuse-ld=... in CFLAGS and LDFLAGS. Invoking ld directly is deprecated (and doesn't handle compiler arguments properly).

[PhilipHazel]

I am closing this as dealt with.