ld Error during link faiss_test because of python functions

[Coien-rr]

Summary

when i using

make -C build install -j

cmake report ld error during Linking CXX executable faiss_test

Here is the complete cmake output：

$ make -C build -j install
make: Entering directory '/home/byte/Code/ANNS/faiss/build'
[ 0%] Built target swigfaiss_avx2_swig_compilation
[ 0%] Built target faiss_example_external_module_swig_compilation
[ 1%] Built target swigfaiss_swig_compilation
[ 1%] Built target faiss_python_callbacks
[ 3%] Built target gtest
[ 10%] Built target benchmark
[ 10%] Built target gtest_main
[ 12%] Built target benchmark_main
[ 43%] Built target faiss
[ 74%] Built target faiss_avx2
[ 75%] Built target swigfaiss
[ 77%] Built target faiss_perf_tests_utils
[ 86%] Built target faiss_c
[ 86%] Built target swigfaiss_avx2
[ 86%] Built target faiss_example_external_module
[ 89%] Built target bench_scalar_quantizer_decode
[ 89%] Built target bench_no_multithreading_rcq_search
[ 89%] Built target bench_scalar_quantizer_accuracy
[ 89%] Built target bench_scalar_quantizer_encode
[ 89%] Built target bench_scalar_quantizer_distance
[ 89%] Linking CXX executable faiss_test
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyUnicode_InternFromString' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to _Py_Dealloc'
/usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyByteArray_AsString' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyLong_FromLong'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyModule_Create2' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyByteArray_Type'
/usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyType_IsSubtype' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyErr_Fetch'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyExc_RuntimeError' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyErr_GivenExceptionMatches'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyEval_RestoreThread' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyExc_MemoryError'
/usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyUnicode_DecodeUTF8' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyErr_Print'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyUnicode_FromString' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyErr_WriteUnraisable'
/usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyBytes_AsString' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyLong_AsDouble'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyErr_Clear' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyCapsule_New'
/usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyErr_ExceptionMatches' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/libfaiss_python_callbacks.so: undefined reference to PyBytes_Size'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyExc_TypeError' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyErr_SetObject'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyLong_AsUnsignedLong' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyType_Ready'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyFloat_FromDouble' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyObject_IsTrue'
/usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyBool_Type' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyErr_Restore'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyObject_Call' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyObject_GetAttr'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyLong_FromSize_t' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyExc_SyntaxError'
/usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyDict_GetItem' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyTuple_New'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyImport_AddModule' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyObject_CallFunctionObjArgs'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyObject_GenericGetAttr' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyExc_ImportError'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyExc_AttributeError' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyErr_Occurred'
/usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/libfaiss_python_callbacks.so: undefined reference to PyBytes_FromStringAndSize' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyDict_SetItem'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyExc_ValueError' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyUnicode_FromFormat'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyDict_SetItemString' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyImport_ImportModule'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyExc_OverflowError' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyModule_GetDict'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyType_Modified' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyTuple_GetSlice'
/usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/libfaiss_python_callbacks.so: undefined reference to PyObject_CallFunction' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyGILState_Ensure'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyErr_SetString' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyObject_IsInstance'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyExc_IOError' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyType_Type'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to _PyObject_New' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyCapsule_GetPointer'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to _Py_NoneStruct' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyDict_New'
/usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyErr_CheckSignals' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to Py_DecRef'
/usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyGILState_Release' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyModule_AddObject'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to _Py_NotImplementedStruct' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyExc_SystemError'
/usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyCapsule_Type' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyObject_Free'
/usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyExc_ModuleNotFoundError' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyBool_FromLong'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyLong_AsLong' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyFloat_Type'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyExc_IndexError' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyFloat_AsDouble'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyEval_SaveThread' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyExc_StopIteration'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyArg_UnpackTuple' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyLong_FromVoidPtr'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyCapsule_Import' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyObject_GetAttrString'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyErr_Format' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyObject_SetAttr'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyExc_ZeroDivisionError' /usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyUnicode_Concat'
/usr/bin/ld: ../faiss/python/_faiss_example_external_module.so: undefined reference to PyLong_FromUnsignedLong' /usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to PyUnicode_AsUTF8AndSize'
/usr/bin/ld: /home/byte/Code/ANNS/faiss/build/faiss/python/_swigfaiss.so: undefined reference to `PyTuple_Size'
collect2: error: ld returned 1 exit status
make[2]: *** [tests/CMakeFiles/faiss_test.dir/build.make:583: tests/faiss_test] Error 1
make[1]: *** [CMakeFiles/Makefile2:1988: tests/CMakeFiles/faiss_test.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
make: Leaving directory '/home/byte/Code/ANNS/faiss/build'

Platform

OS: Linux

Faiss version: git commit 3daf48c

Installed from: compiled by myself

Faiss compilation options:

cmake -B build . -DFAISS_ENABLE_GPU=OFF -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBUILD_TESTING=ON -DFAISS_ENABLE_PYTHON=ON -DFAISS_OPT_LEVEL=avx2 -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON

Running on:

• CPU
• GPU

Interface:

• C++
• Python

Reproduction instructions

1. clone the latest commit of faiss (3daf48c for now)

git clone https://github.com/facebookresearch/faiss.git && cd faiss

2. config for build

cmake -B build . -DFAISS_ENABLE_GPU=OFF -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBUILD_TESTING=ON -DFAISS_ENABLE_PYTHON=ON -DFAISS_OPT_LEVEL=avx2 -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON

3. make for install

make -C build install -j

[Coien-rr]

but i tried add

find_package(PythonLibs REQUIRED)
target_link_libraries(faiss_test PUBLIC ${PYTHON_LIBRARIES})

into tests/CMakeLists.txt for faiss_test, It fixed

I'm not sure if this is a problem with my system configuration or if faiss is missing the relevant cmake configuration for Pythonlibs.

[Coien-rr]

Oddly enough, my build of the release version of faiss.1.9.0 went fine without any linking errors, which I'm guessing has something to do with the recent push of 847cde8 #3922

[Coien-rr]

Sad!!! After #3939, this issue still exists. :(

[mnorris11]

Hi, thanks for the clear repro steps, do you still see this after this change? #3946

[Coien-rr]

Hi, thanks for the clear repro steps, do you still see this after this change? #3946

thanks for your attention, still got error, and I currently have to fix it by manually introducing libpython3.12.so in tests/CMakeLists.txt

[mnorris11]

Thanks @Coien-rr , I can reproduce, I will raise a PR to fix it.

Edit: If you need to be unblocked, these steps were what I was following earlier, and I don't see the error from those:

$ cmake -B build \
      -DBUILD_TESTING=ON \
      -DBUILD_SHARED_LIBS=ON \
      -DFAISS_ENABLE_GPU=OFF \
      -DFAISS_ENABLE_RAFT=OFF \
      -DFAISS_OPT_LEVEL=generic \
      -DFAISS_ENABLE_C_API=ON \
      -DPYTHON_EXECUTABLE=$(which python) \
      -DCMAKE_BUILD_TYPE=Release \
      -DBLA_VENDOR=Intel10_64_dyn \
      -DCMAKE_CUDA_FLAGS="-gencode arch=compute_75,code=sm_75" \
      .
$ make -C build -j faiss
$ make -C build -j swigfaiss
$ make -k -C build -j$(nproc)
$ (cd build/faiss/python && python setup.py install)

I will see how this differs from make -C build install -j

[Coien-rr]

i rebuild faiss by following your step, but i got same error above during make -k -C build -j$(nproc).
But after synchronized with #3954, everything goes well 🤣
I think the building blocks may not have been prepared properly when faiss introduced faiss_example_external_module

[mnorris11]

We have reverted it on main branch in #3954 as you saw, so I will go ahead and close this for now. If you see the issue again after we re-add it, please let us know.

[mnorris11]

Hi @Coien-rr , we plan to re-add it in #3981.

I tried your cmake -> make steps, and could not reproduce.

My conda env:

conda remove --name ext_mod --all -y  # delete all existing conda envs, they caused linking errors for me

conda create -n ext_mod

conda activate ext_mod

conda install -y -q python=3.11 cmake make swig mkl=2023 mkl-devel=2023 numpy scipy pytest gxx_linux-64 sysroot_linux-64 gflags

which python3  # should print ~/miniconda3/envs/ext_mod/bin/python3

Then I ran your cmake -> make steps.