-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/2127 add flatcc #2129
Feature/2127 add flatcc #2129
Conversation
Failure in build 1 (
|
recipes/flatcc/all/conanfile.py
Outdated
if tools.os_info.is_linux: | ||
self.env_info.LD_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib")) | ||
if tools.os_info.is_macos: | ||
self.env_info.DYLD_FALLBACK_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not needed. Conan takes care of it when run_environment=True
if tools.os_info.is_linux: | |
self.env_info.LD_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib")) | |
if tools.os_info.is_macos: | |
self.env_info.DYLD_FALLBACK_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib")) |
cmake = CMake(self) | ||
cmake.configure() | ||
cmake.build() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the reason you're adding the environment variables LD_LIBRARY_PATH
/DYLD_LIBRARY_PATH
is to let cmake.configure
work, then you can add these by using a RunEnvironment.
cmake = CMake(self) | |
cmake.configure() | |
cmake.build() | |
with tools.environment_append(RunEnvironment(self).vars): | |
cmake = CMake(self) | |
cmake.configure() | |
cmake.build() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that is indeed the reason.
For the test package it is no problem to use a RunEnvironment. But now all the packages that use flatcc also have to know that they should run the cmake commands using a RunEnvironment. That makes things more complicated than necessary IMHO.
Is it really a problem to add the LD_LIBRARY_PATH/DYLD_LIBRARY_PATH
variables in package_info() ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jgsogo it would be useful, I think, to add a run_environment
option to all build helpers.
It would be useful for at least autotools, cmake and Medion.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@wdobbe I'm not sure this would work on macos, due to System Integration Protection.
See #1179 (comment) (and the comments around) for my personal fight against SIP
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi! The problem with Macos and SIP is not solvable right now. Even if those variables are in the environment while running CMake, they won't be accessible inside or to those custom commands (check it with a shared build if flatcc
has dependencies, or check the issue linked by @madebr ).
An option would be for the build helpers to populate the environment with their build-requires, I agree, but it won't solve the issue with Mac and SIP 😞
I have a pending POC for this: the problem is not only Mac and SIP, but Conan populating the environment with all the variables from all the build-requirements, this is a problem because some variables may override values from other build-requires,... but also, if build-requires are independent graphs, some libraries could appear with different versions in different build-requires, and the system will find the first one according to the paths in DY/LD_LIBRARY_PATH
😱
Alternatives? Executables are listed in the cpp_info
attribute (conan-io/conan#7240), then Conan creates a wrapper for every executable, something like (for protobuf
build-requires):
protoc.bat
activate environment
path/to/build-requires/executable/protoc -- forward all arguments
deactivate environment
And these wrappers are added to the PATH
, Macos and SIP doesn't clean the PATH
variable, so a custom command protoc ...
will find and execute that wrapper, environment from different build-requires is not mixed, everything clean, everything works out-of-the-box.
wdyt? Do you think it would be a nice approach? Will it work?
Now, answering the question here, flatcc
will be used mainly as a build-requires, Conan already populates the environment with the values of DY/LD....
taken from the cpp_info.bin
attribute when using two profiles (new approach with the build and host contexts). So there is no need to add these variables to cpp_info.env_info
, only add the bin
directory to the PATH
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jgsogo
Are you sure that such a protoc.bat
/protoc.sh
will work on Macos? The activate environment
will also use DYLD_LIBRARY_PATH
.
How about fixing this problem on cmake as follows?
Let the cmake helpers create a cmake variable named e.g. CONAN_LIBRARY_PATHS
that appends all *_LIBRARY_PATHS
.
On Macos
it would contain: DYLD_LIBRARY_PATH=/path/to/lib1:/path/to/lib2/:...
Then a user can do:
add_custom_command(... COMMAND ${CONAN_LIBRARY_PATHS} program)
Now, answering the question here, flatcc will be used mainly as a build-requires, Conan already populates the environment with the values of DY/LD.... taken from the cpp_info.bin attribute when using two profiles (new approach with the build and host contexts). So there is no need to add these variables to cpp_info.env_info, only add the bin directory to the PATH
This would be really great! Imho, conan should have knowledge of a package's capability to be used as a requirement/build requirement.
Don't forget that some recipes also export environment variables to make it it work/override settings...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Being a build-requirement and a requirement is not a problem. It will work the same way as protobuf
does (it provides the library and the protoc
executable). Populating (DY)LD_LIBRARY_PATH
won't help if you call flatcc
from inside CMake, those values will be cleaned before running the executable flatcc
🤷 And, IMHO, I think Macos is doing it right.
The approach with the wrappers work (at least in Macos, I assume Linux will work too). You can have a look at some brute-force implementation here (conan-io/conan@develop...jgsogo:poc/xbuild-exec-wrapper) that works together with this example here (https://github.com/jgsogo/conan-xbuild-scenarios). I need to share these ideas with the team and evaluate if they make sense or not, but it works 😆 ...testing in Windows still pending.
I've also considered the CMake approach, but my main concern is that it would be exclusive to CMake and every other generator will need its own implementation. Not only the build systems ones, but also the virtualenvironments we can generate from Conan. With this wrapper approach I only need to create these wrappers, prepend them to the PATH
and it works out-of-the-box.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi Javier,
Thanks for your work on the wrappers. I will leave Macos shared as an invalid configuration until the wrappers (or other solution) is available in Conan.
Populating
(DY)LD_LIBRARY_PATH
won't help if you callflatcc
from inside CMake, those values will be cleaned before running the executableflatcc
In the test_package flatcc is called from cmake and this works fine when LD_LIBRARY_PATH is set by the flatcc recipe, also in our production code that uses flatcc with this package.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it is because flatcc
doesn't have dependencies (or they are linked statically). The problem is when you need DYLD_LIBRARY_PATH
to locate other dylib
libraries.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was refering to Linux. flatcc called from cmake works fine even when flatcc is build shared.
My Mac is a hackintosh, looks like SIP is always off there, as I have no DYLD_LIBRARY_PATH problems. But unfortunately that also means I cannot test with SIP on.
@@ -0,0 +1,18 @@ | |||
import os.path | |||
|
|||
from conans import ConanFile, CMake, tools |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
from conans import ConanFile, CMake, tools | |
from conans import ConanFile, CMake, tools, RunEnvironment |
Some configurations of 'flatcc/0.6.0' failed in build 2 (
|
Some configurations of 'flatcc/0.6.0' failed in build 3 (
|
COMMAND flatcc -a -o "${GEN_DIR}" "${FBS_DIR}/monster.fbs" | ||
DEPENDS flatcc "${FBS_DIR}/monster.fbs" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like (Macos) debug creates flatcc_d executable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I tested only with Release settings. Will fix that tomorrow.
recipes/flatcc/all/conanfile.py
Outdated
self.output.info('Appending PATH environment variable: %s' % bin_path) | ||
self.env_info.PATH.append(bin_path) | ||
if not self.options.runtime_lib_only: | ||
self.cpp_info.libs.append("flatcc") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some debug configurations create flatcc_d
.
Some configurations of 'flatcc/0.6.0' failed in build 4 (
|
… but fails on CI MacOS, try with DYLD_LIBRARY_PATH
Some configurations of 'flatcc/0.6.0' failed in build 5 (
|
… by importing shared libraries
Some configurations of 'flatcc/0.6.0' failed in build 6 (
|
…S due to problems with SIP
Recipe syntax error in build 7:
|
… previous commit
Some configurations of 'flatcc/0.6.0' failed in build 8 (
|
recipes/flatcc/all/conanfile.py
Outdated
self._cmake.definitions["FLATCC_FAST_DOUBLE"] = self.options.fast_double | ||
self._cmake.definitions["FLATCC_IGNORE_CONST_COND"] = self.options.ignore_const_condition | ||
self._cmake.definitions["FLATCC_TEST"] = False | ||
self._cmake.configure(source_folder=os.path.join(self.source_folder, self._source_subfolder)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By running the cmake script from the source_subfolder directly,
you're not executing
include(conanbuildinfo.cmake)
conan_basic_setup()
Which causes the compiler to use the wrong runtime. This is a problem with MSVC.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for (again) saving me a lot of analysis time. Strangely the current recipe builds fine on my Windows 10 VM with Visual Studio 2019.
I will try to fix the Windows build later this week, some other priorities came up.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The problem is always with build_Type=Debug
and runtime=MTd
or MDd
.
Real troublemakers, those runtimes are... 😄
… script from the source_subfolder directly conan_basic_setup() is not executed and MSVC chooses wrong runtime.
Some configurations of 'flatcc/0.6.0' failed in build 9 (
|
… script from the source_subfolder directly conan_basic_setup() is not executed and MSVC chooses wrong runtime.
Some configurations of 'flatcc/0.6.0' failed in build 11 (
|
…previous commit
All green in build 12 (
|
@@ -0,0 +1,23 @@ | |||
cmake_minimum_required(VERSION 3.12) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cmake_minimum_required(VERSION 3.12) | |
cmake_minimum_required(VERSION 2.8) |
too simple for cmake 3+
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, but the problem is that I never test with cmake 2.8. So yes, that cmake file will probably work with 2.8, but I have no evidence...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
keep cmake 2.8, if you don't use an explicit cmake 3 feature, keep 2.8. It is still used.
Co-authored-by: Uilian Ries <uilianries@gmail.com>
Co-authored-by: Uilian Ries <uilianries@gmail.com>
Co-authored-by: Uilian Ries <uilianries@gmail.com>
All green in build 14 (
|
All green in build 15 (
|
Co-authored-by: Uilian Ries <uilianries@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
almost good to go, thanks for your effort
CI stuck? |
Yes, even license CLA is frozen. |
Co-authored-by: Uilian Ries <uilianries@gmail.com>
Thanks also for your feedback, it was quite an educational exercise. |
All green in build 16 (
|
recipes/flatcc/all/conanfile.py
Outdated
if tools.os_info.is_linux: | ||
self.env_info.LD_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib")) | ||
if tools.os_info.is_macos: | ||
self.env_info.DYLD_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really think this is not needed. Can you provide a scenario where you need the recipe to populate these variables?
if tools.os_info.is_linux: | |
self.env_info.LD_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib")) | |
if tools.os_info.is_macos: | |
self.env_info.DYLD_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These lines can be removed. But my concern is that someone creates a Conan recipe for this project that uses flatcc (via cmake custom command) following the documentation to create the build() function. Then when he tries to create the package he will get the dreaded flatcc: error while loading shared libraries
might now know how to fix it.
I know the RunEnvironment is documented in the Conan reference pages but it is somewhat hidden. I actually didn't know about it until submitting this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Then we need to improve or highlight those lines in the docs (btw, using the two profiles approach, which will be the default in the future, there is no need to add the "RunEnvironment").
If we keep these lines here, then we need to add them to every recipe in Conan Center for the very same reason... and in that scenario, it would be something to manage inside Conan itself... and that's exactly what we have done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I removed the (DY)LD_LIBRARY_PATH lines. But is it still necessary to append the bin path to the PATH variable in package_info() ? Since RunEnvironment does that as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Edit: disregard this comment, it was fixed by @madebr.
@jgsogo I created a workaround in the test_package to run flatcc in the build function as you suggested, but this also fails because flatcc can't find its shared libraries. On my Hackintosh (which has SIP disabled) the test_package builds and runs fine for all build_type and 'shared' combinations.
Because dlopen searches for shared libraries in the current working directory another workaround would be to add an imports() function in the test_package recipe that copies the flatcc shared libs. But as soon as I add an imports() function the build is rejected by CI because the import() function is not allowed.
I could copy the flatcc dylib in the build function before running the flatcc executable. If you think that is acceptable than I can test that at work where we have a real Mac.
Co-authored-by: Javier G. Sogo <jgsogo@gmail.com>
Some configurations of 'flatcc/0.6.0' failed in build 17 (
|
…RY_PATH, remove redundant check for Windows in package() function
Some configurations of 'flatcc/0.6.0' failed in build 18 (
|
…ge on MacOS when flatcc is linked shared
Some configurations of 'flatcc/0.6.0' failed in build 19 (
|
Co-authored-by: Anonymous Maarten <madebr@users.noreply.github.com>
All green in build 20 (
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for the nice workaround and all the explanations around the MacOS System Integrity Protection. Thanks!
Specify library name and version: flatcc/0.6.0
conan-center hook activated.
resolves #2127
Tested on OpenSuse Linux, Windows 10/Visual Studo 2019 and MacOS 10.15.5.
Building on Windows with MinGW doesn't work -> disabled in recipe.
Building on Windows with Visual Studio with option 'shared' set to True doesn't work -> disabled in recipe.