s390 all{mod,yes}config __ashlti3 and __lshrti3 after LLVM commit 1886aad9d03b

[nathanchance]

$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- allyesconfig all
...
s390x-linux-gnu-ld: drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.o: in function `otx2_sq_append_skb':
(.text+0x235e): undefined reference to `__ashlti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v0.o: in function `dr_ste_v0_build_eth_l2_src_or_dst_bit_mask':
dr_ste_v0.c:(.text+0x56f0): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: dr_ste_v0.c:(.text+0x5714): undefined reference to `__ashlti3'
s390x-linux-gnu-ld: dr_ste_v0.c:(.text+0x5738): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: dr_ste_v0.c:(.text+0x575e): undefined reference to `__ashlti3'
s390x-linux-gnu-ld: dr_ste_v0.c:(.text+0x5782): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v0.o: in function `dr_ste_v0_build_eth_l2_src_or_dst_tag':
dr_ste_v0.c:(.text+0x661e): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: dr_ste_v0.c:(.text+0x6642): undefined reference to `__ashlti3'
s390x-linux-gnu-ld: dr_ste_v0.c:(.text+0x6666): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: dr_ste_v0.c:(.text+0x668c): undefined reference to `__ashlti3'
s390x-linux-gnu-ld: dr_ste_v0.c:(.text+0x66b0): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v1.o: in function `dr_ste_v1_build_eth_l2_src_or_dst_bit_mask':
dr_ste_v1.c:(.text+0x540e): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: dr_ste_v1.c:(.text+0x5432): undefined reference to `__ashlti3'
s390x-linux-gnu-ld: dr_ste_v1.c:(.text+0x5456): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: dr_ste_v1.c:(.text+0x547c): undefined reference to `__ashlti3'
s390x-linux-gnu-ld: dr_ste_v1.c:(.text+0x54a0): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v1.o: in function `dr_ste_v1_build_eth_l2_src_or_dst_tag':
dr_ste_v1.c:(.text+0x6338): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: dr_ste_v1.c:(.text+0x635c): undefined reference to `__ashlti3'
s390x-linux-gnu-ld: dr_ste_v1.c:(.text+0x6380): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: dr_ste_v1.c:(.text+0x63a6): undefined reference to `__ashlti3'
s390x-linux-gnu-ld: dr_ste_v1.c:(.text+0x63ca): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v1.o: in function `dr_ste_v1_build_eth_ipv6_l3_l4_tag':
dr_ste_v1.c:(.text+0x74b8): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: dr_ste_v1.c:(.text+0x74dc): undefined reference to `__ashlti3'
s390x-linux-gnu-ld: dr_ste_v1.c:(.text+0x7500): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: dr_ste_v1.c:(.text+0x752c): undefined reference to `__ashlti3'
s390x-linux-gnu-ld: dr_ste_v1.c:(.text+0x7550): undefined reference to `__lshrti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v1.o: in function `dr_ste_v1_build_tnl_mpls_tag':
dr_ste_v1.c:(.text+0x83f4): undefined reference to `__lshrti3'
make[1]: *** [Makefile:1196: vmlinux] Error 1
make[1]: Target 'all' not remade because of errors.
make: *** [Makefile:346: __build_one_by_one] Error 2
make: Target 'allyesconfig' not remade because of errors.
make: Target 'all' not remade because of errors.

Bisect landed on llvm/llvm-project@1886aad

$ git bisect log
# bad: [c1c1df6347bff3167e9aa795b508f56b8fe5fbc1] [mlir] Fix region successor bug in forward dataflow analysis
# good: [1c10201d9660c1d6f43a7226ca7381bfa255105d] Update InstCombine to use undef matcher instead
git bisect start 'c1c1df6347bff3167e9aa795b508f56b8fe5fbc1' '1c10201d9660c1d6f43a7226ca7381bfa255105d'
# good: [f9c0859e9687294a3b6b2e797faf4b7ee62c60ce] [flang] Check for attributes specific to dummy arguments
git bisect good f9c0859e9687294a3b6b2e797faf4b7ee62c60ce
# bad: [6b938d2ead2cb0465436496c0171c7d750e11773] Recommit "[clang][driver] Use the provided arch name for a Darwin target triple
git bisect bad 6b938d2ead2cb0465436496c0171c7d750e11773
# good: [36430d44edba9063a08493c89864edf5f071d08c] [Driver] Use normalized triples for per-target runtimes
git bisect good 36430d44edba9063a08493c89864edf5f071d08c
# bad: [3d3782397a3f13970ead4b56cd404d0b20600497] [TableGen] Remove predicate filtering from GenerateVariants.
git bisect bad 3d3782397a3f13970ead4b56cd404d0b20600497
# good: [e7db8408d05778ff2cb20735f3bdab948d2e3edc] [mlir][python] Add python support for async dialect and passes.
git bisect good e7db8408d05778ff2cb20735f3bdab948d2e3edc
# good: [3b1325cbd39dabc294df409e244ad0b787602ff9] [libc++][NFC] Remove stray whitespace
git bisect good 3b1325cbd39dabc294df409e244ad0b787602ff9
# good: [29cb9dc4be257fd3afd8245fa421f0716feda5bd] NFC: Run clang-format over llvm-link.
git bisect good 29cb9dc4be257fd3afd8245fa421f0716feda5bd
# good: [a8e273f2ed769a3bf1207a02bf609518bb3ae5be] [NFC][SimplifyCFG] Add test showing that profitability check for sinking is broken
git bisect good a8e273f2ed769a3bf1207a02bf609518bb3ae5be
# bad: [b30521c28a4dc1b94d793385e4144ede5822b2c1] [analyzer] Wrong type cast occurs during pointer dereferencing after type punning
git bisect bad b30521c28a4dc1b94d793385e4144ede5822b2c1
# bad: [1886aad9d03b95c35260d6d8013d746bd39dc94a] [SimplifyCFG] Common code sinking: relax restriction on non-uncond predecessors
git bisect bad 1886aad9d03b95c35260d6d8013d746bd39dc94a
# good: [410d03aabf725abcd6e3c5d11f4c2f4c9604627b] [NFC][SimplifyCFG] Add test for sinking common code with multuple cond predecessors
git bisect good 410d03aabf725abcd6e3c5d11f4c2f4c9604627b
# first bad commit: [1886aad9d03b95c35260d6d8013d746bd39dc94a] [SimplifyCFG] Common code sinking: relax restriction on non-uncond predecessors

The functions in question:

• dr_ste_v0_build_eth_l2_src_or_dst_bit_mask
• dr_ste_v0_build_eth_l2_src_or_dst_tag
• dr_ste_v1_build_eth_l2_src_or_dst_bit_mask
• dr_ste_v1_build_eth_l2_src_or_dst_tag
• dr_ste_v1_build_eth_ipv6_l3_l4_tag
• dr_ste_v1_build_tnl_mpls_tag

I am guessing that this is somehow related to the DR_STE_SET_TAG macro, more specifically the MLX5_SET macro, since that seems to be common to all of these functions. The kernel does not link against libgcc or compiler-rt. If this is a valid transformation, then we would need to import those routines like commits fb8722735f50 ("arm64: support __int128 on gcc 5+") and 9bfe7553fadb ("arm64: Implement __lshrti3 library function").

cc @uweigand

[nickdesaulniers]

https://gcc.gnu.org/onlinedocs/gccint/Integer-library-routines.html

Runtime Function: long long __ashlti3 (long long a, int b)
These functions return the result of shifting a left by b bits.

Runtime Function: long long __lshrti3 (long long a, int b)
These functions return the result of logically shifting a right by b bits.

[nathanchance]

Enabling CONFIG_NET_SWITCHDEV in defconfigreproduces only one error around __lshrti3.

$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- defconfig

$ scripts/config -e MLX5_CORE -e NET_SWITCHDEV -e VXLAN

$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- olddefconfig all
...
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v0.o: in function `dr_ste_v0_build_tnl_mpls_tag':
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v0.c:1265: undefined reference to `__lshrti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v0.c:1265: undefined reference to `__lshrti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v0.c:1265: undefined reference to `__lshrti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v0.c:1265: undefined reference to `__lshrti3'
make[1]: *** [Makefile:1196: vmlinux] Error 1
make[1]: Target 'all' not remade because of errors.
make: *** [Makefile:346: __build_one_by_one] Error 2
make: Target 'olddefconfig' not remade because of errors.
make: Target 'all' not remade because of errors.

After enabling UBSAN, errors around __ashlti3 can be seen:

$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- defconfig

$ scripts/config -e MLX5_CORE -e NET_SWITCHDEV -e UBSAN -e VXLAN

$ make -skj"$(nproc)" ARCH=s390 CC=clang CROSS_COMPILE=s390x-linux-gnu- olddefconfig all
...
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v0.o: in function `dr_ste_v0_build_tnl_mpls_tag':
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v0.c:1265: undefined reference to `__lshrti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v0.c:1265: undefined reference to `__lshrti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v1.o: in function `dr_ste_v1_build_eth_ipv6_l3_l4_tag':
drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v1.c:1191: undefined reference to `__lshrti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v1.c:1191: undefined reference to `__ashlti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v1.c:1191: undefined reference to `__lshrti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v1.c:1191: undefined reference to `__ashlti3'
s390x-linux-gnu-ld: drivers/net/ethernet/mellanox/mlx5/core/steering/dr_ste_v1.c:1191: undefined reference to `__lshrti3'
make[1]: *** [Makefile:1196: vmlinux] Error 1
make[1]: Target 'all' not remade because of errors.
make: *** [Makefile:346: __build_one_by_one] Error 2
make: Target 'olddefconfig' not remade because of errors.
make: Target 'all' not remade because of errors.

[uweigand]

GCC implements 128-bit shifts via inline code using SLDL and related instructions, which LLVM currently does not. I guess we can add support for that to LLVM as well to eliminate this difference. I'll have a look.

[uweigand]

Should now be fixed by https://reviews.llvm.org/D101993

[nickdesaulniers]

Thanks @uweigand @nathanchance and Jonas! confirmed that I can build allyesconfig for s390 :) w/ llvm @ e2d774a3dbbbbff21531289889f6906b22f04cfe

[nathanchance]

Thanks for the quick fix @uweigand! Would you be opposed to allowing llvm/llvm-project@1c4cb51 to be added to LLVM 12.0.1? CKI will run into this once this issue is resolved because I see a build failure of this nature with Fedora's config + LLVM 12.0.0 unfortunately, not sure how I missed it (guess that LLVM was not solely to blame). I have verified that it cherry picks clean, passes the tests, and builds the kernel fine on top of release/12.x @ 4a12f51ad0090c3bcfea29c8dd021486ac3aa329 (which is the last commit where the tests pass successfully for me). I can open an LLVM bug for it if you agree. -rc1 is scheduled for May 11th it looks like so I would like to get a bug filed by tomorrow to give @tstellar enough time to look it over.

[uweigand]

Hi @nathanchance , I think this would be OK to backport. Please go ahead and open a bug.

[nathanchance]

Thanks! Done: https://llvm.org/pr50252