OpenMP problem with static library - Windows

[vincferr]

I am trying to move a project, compiled with VS 16.8.4 and MSVC, from Intel MKL to OpenBLAS.
I have been following the instructions on

https://github.com/xianyi/OpenBLAS/wiki/How-to-use-OpenBLAS-in-Microsoft-Visual-Studio

and they worked quite ok, with the sole addition of -DMSVC_STATIC_CRT=ON as option, to avoid linking problem when /MT is employed.

The annoying fact is that, when I use the libs in my project, compiled via MSVC, at linking time it requires to link against (beside the Fortran stuff)

ompstub.lib

and at run time it requires

ompstub.dll

This seems to modify the behaviour of my code, as now omp_get_max_threads() returns 1.
Any clue on why that happens?
This seems to happen with develop and v.0.3.13 alike.
Moreover I get this warning while linking:

3>LINK : warning LNK4098: defaultlib 'msvcrt' conflicts with use of other libs; use /NODEFAULTLIB:library
but it is mostly due to the fact that OpenBLAS is compiled with clang-cl, I guess,

The option I pass to cmake are:

cmake .. -G "Ninja" -DCMAKE_CXX_COMPILER=clang-cl -DCMAKE_C_COMPILER=clang-cl -DCMAKE_Fortran_COMPILER=flang -DBUILD_WITHOUT_LAPACK=no -DNOFORTRAN=0 -DDYNAMIC_ARCH=ON -DCMAKE_BUILD_TYPE=Release -DMSVC_STATIC_CRT=ON -DCPP_THREAD_SAFETY_GEMV=ON

[martin-frbg]

Please try adding -DUSE_OPENMP=1 to your cmake options - this should replace the dependency on the dummy ompstub library (which has a hardcoded "return 1" for omp_get_max_threads) with the proper OpenMP implementation

[vincferr]

Unfortunately it does not work, as it seems not able to find Fortran OPENMP:

-- Compiling a 64-bit binary.
-- Found OpenMP_C: -Xclang -fopenmp (found version "3.1")
CMake Error at C:/Users/Enzo/miniconda3/Library/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:218 (message):
  Could NOT find OpenMP_Fortran (missing: OpenMP_Fortran_FLAGS
  OpenMP_Fortran_LIB_NAMES)
Call Stack (most recent call first):
  C:/Users/Enzo/miniconda3/Library/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:582 (_FPHSA_FAILURE_MESSAGE)
  C:/Users/Enzo/miniconda3/Library/share/cmake-3.19/Modules/FindOpenMP.cmake:529 (find_package_handle_standard_args)
  cmake/arch.cmake:37 (find_package)
  cmake/system.cmake:217 (include)
  CMakeLists.txt:70 (include)

I will try to make it work with ICC and IFortran, as they are now freely available.
Beside that, is that intended that a non-parallel build has a dependency on (stub) OpenMP?

[martin-frbg]

Not intended by us here, but apparently a feature of flang (or perhaps even llvm and llvm-derived compilers in general)

[martin-frbg]

From TriBITSPub/TriBITS#244 (comment) the Fortran-OpenMP misdetection appears to be a problem in certain versions of cmake

[brada4]

Does it work with CMake from visual studio installer?

[vincferr]

Does it work with CMake from visual studio installer?

It requires an update of the ninja builder (does not support fortran), but after that yields exactly the same results.

The fix suggested :

  -DOpenMP_Fortran_FLAGS=-fopenmp \
  -DOpenMP_Fortran_LIB_NAMES=gomp \
  -DOpenMP_gomp_LIBRARY=-lgomp \

fixes the configuration step for the USE_OPENMP=1, but the build fails. For instance I get:

FAILED: CMakeFiles/openblas.dir/lapack-netlib/SRC/slagv2.f.obj
C:\Users\Enzo\miniconda3\Library\bin\flang.exe  -I..\lapack-netlib\SRC -I..\lapack-netlib\LAPACKE\include -fopenmp -fopenmp -Mrecursive -Kieee      -Mrecursive -Kieee -o CMakeFiles\openblas.dir\lapack-netlib\SRC\slagv2.f.obj -c CMakeFiles\openblas.dir\lapack-netlib\SRC\slagv2.f-pp.f
flang.exe: error: Fortran frontend to LLVM command failed due to signal (use -v to see invocation)
clang version 5.0.0
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Users\Enzo\miniconda3\Library\bin
flang.exe: note: diagnostic msg: PLEASE submit a bug report to  and include the crash backtrace, preprocessed source, and associated run script.
flang.exe: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
ninja: build stopped: subcommand failed.

[vincferr]

Just realized that the version of flang and clang-cl installed by miniconda are very old (LLVM 5). Probably that might part of the issue. Compilation with the new oneAPI intel compilers (icl and ifx) are not supported for now by cmake but it seems they are in the works:
https://gitlab.kitware.com/cmake/cmake/-/issues/21560
https://gitlab.kitware.com/cmake/cmake/-/issues/21561

[brada4]

OpenBLAS does not replace cmake detection of C and fortran compilers. If miniconda ships broken combination of those, it cannot be solved here. More straightforward approach would be to get clang-cl and flang via official LLVM download (Probably waiting a day or two for v11.0.1 to get pushed to download page), and trying to tame cmake from visual studio, failing that installing official cmake binary.
Probably title should not mention visual studio since visual studio compiler is not involved.

[vincferr]

@brada4 Yes probably it would be correct to do so,
Unfortunately it seems that flang 11.0.1 does not support windows (https://releases.llvm.org/11.0.0/tools/flang/docs/ReleaseNotes.html), so no chance there. Better to wait to icl/ifx to be supported by cmake.
as a side note, I compiled the library with -DBUILD_SHARED_LIBS=ON, and the dependency on ompstub is gone. Then it works as expected.

[brada4]

Early versions when flang was outside llvm main tree did work, crashing often though, probably anaconda includes something atound that.
There is another, at least still says supports windows, in anaconda - https://docs.lfortran.org/installation/

[h-vetinari]

There's a difference between the old flang compiler, which was repackaged by conda-forge, and the f18-renamed-to-flang compiler as it exists in llvm, see: https://github.com/llvm/llvm-project/tree/master/flang#flang

This newer compiler does not yet support compiling code:

Flang is able to parse a comprehensive subset of the Fortran language and check it for correctness. Flang is not yet able to generate LLVM IR for the source code and thus is unable to compile a running binary.

But then, flang was also only added for LLVM 11. Both the code generation, as well as windows support, are actively being pursued.

[brada4]

Gfortran would mix in gomp redist. Maybe it is possible to get it to link instead to ms-omp or iomp,

lfortran is different independent project, with primary distribution channel via anaconda..

[xoviat]

As one of the creators of that windows compiler, I decided that it would be a better use of my time to wait until the llvm developers added windows support officially rather than spent time maintaining a parallel compiler. It is possible to use with openmp though.

[xoviat]

I actually have an updated version. Let me see how much time it would take to get it working...

[brada4]

Drop a line here, there are FAQ sections to get users past disappointment with tricky build environments.

[xoviat]

The flang compiler itself needs the -mp argument to use openmp. If you change nothing from the wiki, but add the -mp compiler argument, then it should work.

[xoviat]

Well, I did manage to update flang to llvm 11, but ran into #3084.

[xoviat]

@vincferr I built OpenBLAS with LLVM 11. Are you interested?

[xoviat]

update: when attempting to compile with OpenMP, I received this error:

F90-W-0547-OpenMP feature, DEPEND, not yet implemented in this version of the compiler. (..\lapack-netlib\SRC\ssytrd_sb2st.F: 486)
F90-W-0547-OpenMP feature, DEPEND, not yet implemented in this version of the compiler. (..\lapack-netlib\SRC\ssytrd_sb2st.F: 486)
F90-W-0547-OpenMP feature, DEPEND, not yet implemented in this version of the compiler. (..\lapack-netlib\SRC\ssytrd_sb2st.F: 486)
F90-W-0547-OpenMP feature, DEPEND, not yet implemented in this version of the compiler. (..\lapack-netlib\SRC\ssytrd_sb2st.F: 497)
F90-W-0547-OpenMP feature, DEPEND, not yet implemented in this version of the compiler. (..\lapack-netlib\SRC\ssytrd_sb2st.F: 497)
  0 inform,   5 warnings,   0 severes, 0 fatal for ssytrd_sb2st

Checking the upstream version, flang-compiler/flang#904, it appears this isn't my fault. I'm also not capable currently of implementing this feature in the flang compiler. Therefore, it appears that it's not possible to use OpenMP with OpenBLAS on windows.

[martin-frbg]

Hm. That is from Reference-LAPACK and the OpenBLAS version has it ifdef'd requiring the compiler to announce _OPENMP >= 201307 compatibility. If flang makes a false claim here I could add another conditional to keep it from parallelizing this particular loop.

[brada4]

It is a warning. I'd trust them that it means no parallelism and correct result as a consequence
https://github.com/flang-compiler/flang/blob/5af987c0be6175db2c2f91c1410e096be4a1416f/tools/flang1/flang1exe/semsmp.c#L2406

[xoviat]

The error is my fault. It's an ICE, I have to track it down.

[xoviat]

note: I published it to https://github.com/xoviat/flang/tree/windows

[xoviat]

note: this issue has been resolved in flang upstream. there should be a new version out today or tomorrow that resolves the issue. no changes to OpenBLAS are required.

[xoviat]

ref conda-forge/flang-feedstock#19