LLJIT running error "Segmentation fault: 11", while linking c++ standard libraries.

[lanistor]

I was using custom c++ functions, and it includes c++ standard libraries.
While i using c++ standard libraries, error occured:

./build/main;
Stack dump:
0.      Program arguments: ./build/main 
0  main                     0x000000010c4f7d8c llvm::sys::PrintStackTrace(llvm::raw_ostream&) + 60
1  main                     0x000000010c4f8349 PrintStackTraceSignalHandler(void*) + 25
2  main                     0x000000010c4f5dc6 llvm::sys::RunSignalHandlers() + 118
3  main                     0x000000010c4fbf0c SignalHandler(int) + 252
4  libsystem_platform.dylib 0x00007fff6af3642d _sigtramp + 29
5  libsystem_platform.dylib 000000000000000000 _sigtramp + 18446603338721827824
6  libsystem_platform.dylib 0x00000001162a8213 _sigtramp + 18446603343388679683
7  libsystem_platform.dylib 0x00000001162a80d1 _sigtramp + 18446603343388679361
8  libsystem_platform.dylib 0x00000001162a801a _sigtramp + 18446603343388679178
9  libsystem_platform.dylib 0x00000001162a7c2d _sigtramp + 18446603343388678173
10 libsystem_platform.dylib 0x00000001162a79ed _sigtramp + 18446603343388677597
11 libsystem_platform.dylib 0x00000001162a7147 _sigtramp + 18446603343388675383
12 libsystem_platform.dylib 0x00000001162a7032 _sigtramp + 18446603343388675106
13 main                     0x000000010c2f63d6 main + 1830
14 libdyld.dylib            0x00007fff6ad3d7fd start + 1
make: *** [start] Segmentation fault: 11

Codes:

main.cpp

#include "llvm/ExecutionEngine/JITLink/JITLinkMemoryManager.h"
#include "llvm/ExecutionEngine/JITSymbol.h"
#include "llvm/ExecutionEngine/Orc/LLJIT.h"
#include "llvm/ExecutionEngine/Orc/ObjectLinkingLayer.h"
#include "llvm/ExecutionEngine/Orc/ThreadSafeModule.h"
#include "llvm/IR/Verifier.h"
#include "llvm/Support/InitLLVM.h"
#include "llvm/Support/MemoryBuffer.h"
#include "llvm/Support/TargetSelect.h"
#include <fstream>
#include <iostream>

using namespace std;
using namespace llvm;
using namespace llvm::orc;

int main(int argc, char *argv[]) {

  InitLLVM X(argc, argv);
  InitializeNativeTarget();
  InitializeNativeTargetAsmPrinter();
  ThreadSafeContext context(std::make_unique<LLVMContext>());

  ExitOnError ExitOnErr;

  auto JTMB = ExitOnErr(JITTargetMachineBuilder::detectHost());
  JTMB.setCodeModel(CodeModel::Small);

  auto jit =
      ExitOnErr(LLJITBuilder()
                    .setJITTargetMachineBuilder(std::move(JTMB))
                    .create());

  jit->getMainJITDylib().addGenerator(
      ExitOnErr(orc::DynamicLibrarySearchGenerator::GetForCurrentProcess(
          jit->getDataLayout().getGlobalPrefix())));

  char        ffi_file[] = "build/ffi";
  llvm::Error err =
      jit->addObjectFile(ExitOnErr(errorOrToExpected(MemoryBuffer::getFileAsStream(ffi_file))));

  if (err) {
    cout << "error addObjectFile" << endl;
    return 1;
  };

  char func_name[]      = "add";
  auto func_add         = ExitOnErr(jit->lookup(func_name));
  int (*func)(int, int) = (int (*)(int, int))func_add.getAddress();
  int result            = func(111, 222);

  cout << "result: " << result << endl;

  return 0;
}

ffi.cpp, which will build and output build/ffifile.

#include <vector>

extern "C" int add(int a, int b) {
  std::vector<int> vc;
  vc.push_back(1);
  return a + b;
}

When vc.push_back(1); was included in ffi.cpp, error occoured, when delete this line , it runs ok.

Here is my environment

OS: MacOSX 10.15.2
LLVM: 10
Clang: 11

Here is my cmake file

cmake_minimum_required(VERSION 3.15.0)

project(jitdemo)

SET (CMAKE_C_COMPILER /usr/bin/clang)
SET (CMAKE_CXX_COMPILER /usr/bin/clang++)

SET ( CMAKE_BUILD_TYPE Debug )

find_package(LLVM REQUIRED CONFIG)

message(STATUS ">>Found LLVM ${LLVM_PACKAGE_VERSION}")
message(STATUS ">>Using LLVMConfig.cmake in: ${LLVM_DIR}")

add_compile_options(-std=c++17)
add_compile_options(-stdlib=libc++)
add_definitions(${LLVM_DEFINITIONS})

include_directories(${LLVM_INCLUDE_DIRS})
include_directories(${PROJECT_SOURCE_DIR})

add_subdirectory(link)

llvm_map_components_to_libnames(llvm_libs support core orcjit irreader nativecodegen)

add_executable(main main.cpp)
target_link_libraries(main link ${llvm_libs})

[androm3da]

Why does it fail? When you step through your program, do you get to func_add.getAddress()? What does it return?

[dwblaikie]

Probably best to build with debug info (-g) and with assertions enabled (including the LLVM libraries you're using - they should be built with both as well) - that's likely to produce a more informative error message.

[lanistor]

@dwblaikie LLVM was built by cmake -G "Ninja" -DLLVM_ENABLE_PROJECTS="clang;" -DCMAKE_BUILD_TYPE=Debug -DLLVM_ENABLE_ASSERTIONS=Yes ../llvm, this will include assertions.
And how to use -g?
This is all the error info now, there is no more.
Will it be the compatibility problem between clang-11 and llvm-10? Or the bug of MacOSX 10.15

[donhinton]

CMAKE will add -g when CMAKE_BUILD_TYPE=Debug. Run ninja -v to see what is actually passed to each target. hth... don

…

On Thu, Jan 2, 2020 at 3:29 AM vifird ***@***.***> wrote: @dwblaikie <https://github.com/dwblaikie> LLVM was built by cmake -G "Ninja" -DLLVM_ENABLE_PROJECTS="clang;" -DCMAKE_BUILD_TYPE=Debug -DLLVM_ENABLE_ASSERTIONS=Yes ../llvm, this will include assertions. And how to use -g? This is all the error info now, there is no more. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#84?email_source=notifications&email_token=AAQVGNXOK7BZ5XJBMUARCIDQ3XF3HA5CNFSM4KBKJVLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH6EWFA#issuecomment-570182420>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQVGNS3ZKV4IAN62I7MTCDQ3XF3HANCNFSM4KBKJVLA> .

[lhames]

Hi Vifird. I am not able to reproduce this locally. How are you building build/ffi?

[lanistor]

@lhames Thanks for concerning. It's built by clang++ -std=c++17 -stdlib=libc++ ffi.cpp -c -o build/ffi.

[lhames]

Are you building build/ffi with the same libc++ that you’re using to build LLVM? Or are you using the built clang++ / libc++ to build build/ffi?

It’s possible that this error is due an ABI incompatibility between libc++ versions.

Side note:

-c -o build/ffi

By convention relocatable objects should have a “.o” extension, so “-c -o build/ffi.o”. You won’t fundamentally break anything by not using the suffix (to my knowledge), but it might be confusing for some tools, or when others look at your project. :)

[lanistor]

@lhames Thanks a lot. I didn't install llvm's libc++, and i think they ( build/main & build/ffi.o ) are using the same libc++, see below output:
build/ffi was built by clang++ ffi.cpp -o build/ffi, without -c, intent to check it's dynamic linking libraries. (The normal build/ffi.o used in LLJIT is not built by this, it's built by clang++ ffi.cpp -c -o build/ffi.o)

➜  jit git:(master) ✗ otool -L build/main            
build/main:
        /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
        /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 800.7.0)
➜  jit git:(master) ✗ otool -L build/ffi
build/ffi:
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 800.7.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)
➜

I had changed my clang to llvm's clang(version 10), and here are my environment variables:
~/.bash_profile

# LLVM
export PATH="/Users/vifird/C/compile/llvm-project/build/bin:$PATH"
export CC="/Users/vifird/C/compile/llvm-project/build/bin/clang"
export CXX="/Users/vifird/C/compile/llvm-project/build/bin/clang++"
export LDFLAGS="-L/Users/vifird/C/compile/llvm-project/build/lib"
export CPPFLAGS="-I/Users/vifird/C/compile/llvm-project/build/include:/Users/vifird/C/compile/llvm-project/llvm/include"
export CPLUS_INCLUDE_PATH="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks"

ffi.cpp

#include <iostream>
#include <vector>

extern "C" int add(int a, int b) {
  std::vector<int> vc;
  vc.push_back(1);
  return a + b;
}

int main() {
  printf(">>main run\n");
  return add(1, 2);
}

[lanistor]

@lhames I had create a demo project to show this error in minimum code: https://github.com/vifird/jit. And i had tested it in two new MacBooks(MacOSX 10.15), those both displayed this error, so i think it may not be a problem of libc++.

[lhames]

@vifird: Is build/ffi an executable? You shouldn’t be able to load that into the JIT at all. But if it’s a relocatable object it shouldn’t show anything listed for otool -L.

I would be inclined to get hold of the preprocessed source for both main and ffi and compare the definition of their vector classes: The most likely explanation here is a difference in the vector definition being used by main and ffi.

I’m out on vacation this week so I may not get a chance to reply further. Please ping me next week though and I can take another look.

[lanistor]

@lhames build/ffi is an executable, but isn't used in LLJIT. What used in LLJIT is build/ffi.o, which is built by clang++ ffi.cpp -c -o build/ffi.o, and it displays this error.
The project vifird/jit can display this error stably.

[lanistor]

@lhames I had resolved this problem, by added setObjectLinkingLayerCreator:

auto jit =
    ExitOnErr(LLJITBuilder()
                  .setJITTargetMachineBuilder(std::move(JTMB))
                  .setObjectLinkingLayerCreator([&](ExecutionSession &ES, const Triple &TT) {
                    return make_unique<ObjectLinkingLayer>(
                        ES, make_unique<jitlink::InProcessMemoryManager>());
                  })
                  .create());

Thanks a lot. This problem blocked me for a month.

[lhames]

Hi @vifird. Sorry I was not able to get back to you sooner: I have been busy with other bugs. I am glad you were able to resolve this!

Now that I have this extra information (that JITLink worked, but RuntimeDyld did not) I can see the bug by inspection. Your original code contained this line, which I did not notice the first time I read it:

JTMB.setCodeModel(CodeModel::Small);

Using small code model is only valid if you are using JITLink. If you are using RuntimeDyld (the default linking layer) then you must use the large code model, which is the default for the JIT.

Sorry to only be giving you an explanation after you solve it, but hopefully it helps explain why your change worked.

[lanistor]

@lhames It's really helpful, i didn't know why my change worked before. Thanks a lot, i need to learn more about JITLink and RuntimeDyld.

[zhfeng20108]

@vifird 把配置写进PROJECT就没事了，写进TARGE我也报你说的错

[lanistor]

@zhfeng20108 感谢。不过问题点好像不太一样，你这个应该是Xcode应用开发，我遇到的问题是用LLVM的JIT引擎时遇到的问题。