Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/2127 add flatcc #2129

Merged
merged 22 commits into from
Aug 24, 2020
Merged

Conversation

wdobbe
Copy link
Contributor

@wdobbe wdobbe commented Jul 4, 2020

Specify library name and version: flatcc/0.6.0

  • I've read the guidelines for contributing.
  • I've followed the PEP8 style guides for Python code in the recipes.
  • I've used the latest Conan client version.
  • I've tried at least one configuration locally with the
    conan-center hook activated.

resolves #2127

Tested on OpenSuse Linux, Windows 10/Visual Studo 2019 and MacOS 10.15.5.

Building on Windows with MinGW doesn't work -> disabled in recipe.
Building on Windows with Visual Studio with option 'shared' set to True doesn't work -> disabled in recipe.

@conan-center-bot
Copy link
Collaborator

Failure in build 1 (4b648d5d242cad9863a906d5e5e3ec7f36605f60):
flatcc/0.6.0:

  • Error processing recipe: Windows x86_64, Debug, Visual Studio 14, MTd. Options: flatcc:shared-False
    Unexpected error happened:
ERROR: flatcc/0.6.0: 'options.shared' doesn't exist
Possible options are ['portable', 'gnu_posix_memalign', 'runtime_lib_only', 'verify_assert', 'verify_trace', 'reflection', 'native_optim', 'fast_double', 'ignore_const_condition']

Comment on lines 92 to 95
if tools.os_info.is_linux:
self.env_info.LD_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib"))
if tools.os_info.is_macos:
self.env_info.DYLD_FALLBACK_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not needed. Conan takes care of it when run_environment=True

Suggested change
if tools.os_info.is_linux:
self.env_info.LD_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib"))
if tools.os_info.is_macos:
self.env_info.DYLD_FALLBACK_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib"))

Comment on lines 11 to 13
cmake = CMake(self)
cmake.configure()
cmake.build()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the reason you're adding the environment variables LD_LIBRARY_PATH/DYLD_LIBRARY_PATH is to let cmake.configure work, then you can add these by using a RunEnvironment.

Suggested change
cmake = CMake(self)
cmake.configure()
cmake.build()
with tools.environment_append(RunEnvironment(self).vars):
cmake = CMake(self)
cmake.configure()
cmake.build()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is indeed the reason.

For the test package it is no problem to use a RunEnvironment. But now all the packages that use flatcc also have to know that they should run the cmake commands using a RunEnvironment. That makes things more complicated than necessary IMHO.
Is it really a problem to add the LD_LIBRARY_PATH/DYLD_LIBRARY_PATH variables in package_info() ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jgsogo it would be useful, I think, to add a run_environment option to all build helpers.
It would be useful for at least autotools, cmake and Medion.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wdobbe I'm not sure this would work on macos, due to System Integration Protection.

See #1179 (comment) (and the comments around) for my personal fight against SIP

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi! The problem with Macos and SIP is not solvable right now. Even if those variables are in the environment while running CMake, they won't be accessible inside or to those custom commands (check it with a shared build if flatcc has dependencies, or check the issue linked by @madebr ).

An option would be for the build helpers to populate the environment with their build-requires, I agree, but it won't solve the issue with Mac and SIP 😞


I have a pending POC for this: the problem is not only Mac and SIP, but Conan populating the environment with all the variables from all the build-requirements, this is a problem because some variables may override values from other build-requires,... but also, if build-requires are independent graphs, some libraries could appear with different versions in different build-requires, and the system will find the first one according to the paths in DY/LD_LIBRARY_PATH 😱

Alternatives? Executables are listed in the cpp_info attribute (conan-io/conan#7240), then Conan creates a wrapper for every executable, something like (for protobuf build-requires):

protoc.bat

activate environment
path/to/build-requires/executable/protoc -- forward all arguments
deactivate environment

And these wrappers are added to the PATH, Macos and SIP doesn't clean the PATH variable, so a custom command protoc ... will find and execute that wrapper, environment from different build-requires is not mixed, everything clean, everything works out-of-the-box.

wdyt? Do you think it would be a nice approach? Will it work?


Now, answering the question here, flatcc will be used mainly as a build-requires, Conan already populates the environment with the values of DY/LD.... taken from the cpp_info.bin attribute when using two profiles (new approach with the build and host contexts). So there is no need to add these variables to cpp_info.env_info, only add the bin directory to the PATH

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jgsogo
Are you sure that such a protoc.bat/protoc.sh will work on Macos? The activate environment will also use DYLD_LIBRARY_PATH.


How about fixing this problem on cmake as follows?

Let the cmake helpers create a cmake variable named e.g. CONAN_LIBRARY_PATHS that appends all *_LIBRARY_PATHS.
On Macos it would contain: DYLD_LIBRARY_PATH=/path/to/lib1:/path/to/lib2/:...

Then a user can do:

add_custom_command(... COMMAND ${CONAN_LIBRARY_PATHS} program)

Now, answering the question here, flatcc will be used mainly as a build-requires, Conan already populates the environment with the values of DY/LD.... taken from the cpp_info.bin attribute when using two profiles (new approach with the build and host contexts). So there is no need to add these variables to cpp_info.env_info, only add the bin directory to the PATH

This would be really great! Imho, conan should have knowledge of a package's capability to be used as a requirement/build requirement.

Don't forget that some recipes also export environment variables to make it it work/override settings...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Being a build-requirement and a requirement is not a problem. It will work the same way as protobuf does (it provides the library and the protoc executable). Populating (DY)LD_LIBRARY_PATH won't help if you call flatcc from inside CMake, those values will be cleaned before running the executable flatcc 🤷 And, IMHO, I think Macos is doing it right.

The approach with the wrappers work (at least in Macos, I assume Linux will work too). You can have a look at some brute-force implementation here (conan-io/conan@develop...jgsogo:poc/xbuild-exec-wrapper) that works together with this example here (https://github.com/jgsogo/conan-xbuild-scenarios). I need to share these ideas with the team and evaluate if they make sense or not, but it works 😆 ...testing in Windows still pending.

I've also considered the CMake approach, but my main concern is that it would be exclusive to CMake and every other generator will need its own implementation. Not only the build systems ones, but also the virtualenvironments we can generate from Conan. With this wrapper approach I only need to create these wrappers, prepend them to the PATH and it works out-of-the-box.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Javier,
Thanks for your work on the wrappers. I will leave Macos shared as an invalid configuration until the wrappers (or other solution) is available in Conan.

Populating (DY)LD_LIBRARY_PATH won't help if you call flatcc from inside CMake, those values will be cleaned before running the executable flatcc

In the test_package flatcc is called from cmake and this works fine when LD_LIBRARY_PATH is set by the flatcc recipe, also in our production code that uses flatcc with this package.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is because flatcc doesn't have dependencies (or they are linked statically). The problem is when you need DYLD_LIBRARY_PATH to locate other dylib libraries.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was refering to Linux. flatcc called from cmake works fine even when flatcc is build shared.
My Mac is a hackintosh, looks like SIP is always off there, as I have no DYLD_LIBRARY_PATH problems. But unfortunately that also means I cannot test with SIP on.

@@ -0,0 +1,18 @@
import os.path

from conans import ConanFile, CMake, tools
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from conans import ConanFile, CMake, tools
from conans import ConanFile, CMake, tools, RunEnvironment

@conan-center-bot
Copy link
Collaborator

Some configurations of 'flatcc/0.6.0' failed in build 2 (a4a1d326fa4ff8d8563c3e92998025dec5e6665f):

@conan-center-bot
Copy link
Collaborator

Some configurations of 'flatcc/0.6.0' failed in build 3 (1dabdd15d5601fbdf3a43543c45f17ad779ba848):

Comment on lines 17 to 18
COMMAND flatcc -a -o "${GEN_DIR}" "${FBS_DIR}/monster.fbs"
DEPENDS flatcc "${FBS_DIR}/monster.fbs"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like (Macos) debug creates flatcc_d executable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I tested only with Release settings. Will fix that tomorrow.

self.output.info('Appending PATH environment variable: %s' % bin_path)
self.env_info.PATH.append(bin_path)
if not self.options.runtime_lib_only:
self.cpp_info.libs.append("flatcc")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some debug configurations create flatcc_d.

@conan-center-bot
Copy link
Collaborator

Some configurations of 'flatcc/0.6.0' failed in build 4 (a5ecc2ee2f1cd87360ca88d6e33f82409892d7f5):

… but fails on CI MacOS, try with DYLD_LIBRARY_PATH
@conan-center-bot
Copy link
Collaborator

Some configurations of 'flatcc/0.6.0' failed in build 5 (ac88c46cc38c159d15ee65a1e75eea9cc9e04d79):

@conan-center-bot
Copy link
Collaborator

Some configurations of 'flatcc/0.6.0' failed in build 6 (95b7598d290a5df0b7c1334525239e3784fc4cb0):

  • Linux x86_64, Release, gcc 5, libstdc++ . Options: flatcc:shared-True
    • Hooks errors detected:
      • [HOOK - conan-center.py] pre_export(): ERROR: [TEST PACKAGE - NO IMPORTS() (KB-H034)] The method importsis not allowed in test_package/conanfile.py (https://github.com/conan-io/conan-center-index/wiki/Error-Knowledge-Base#KB-H034)
    • Log download
    • Log error Download
    • Profile
  • Access to all the logs

@conan-center-bot
Copy link
Collaborator

Recipe syntax error in build 7:

WARN: Remotes registry file missing, creating default one in /home/conan/workspace/onan-center-pull-request_PR-2129/7/cda4c277-4420-4931-96d6-6e023ef3fcdd/.conan/remotes.json
ERROR: Error loading conanfile at '/home/conan/workspace/onan-center-pull-request_PR-2129/recipes/flatcc/all/conanfile.py': Unable to load conanfile in /home/conan/workspace/onan-center-pull-request_PR-2129/recipes/flatcc/all/conanfile.py
  File "/opt/pyenv/versions/3.7.5/lib/python3.7/site-packages/conans/model/ref.py", line 197, in loads
    name, version, user, channel, revision = get_reference_fields(text)
  File "/opt/pyenv/versions/3.7.5/lib/python3.7/site-packages/conans/model/ref.py", line 66, in get_reference_fields
    el1, el2 = _split_pair(arg_reference, "/") or (arg_reference, None)
  File "/opt/pyenv/versions/3.7.5/lib/python3.7/site-packages/conans/model/ref.py", line 18, in _split_pair
    raise ConanException("The reference has too many '{}'".format(split_char))
conans.errors.ConanException: The reference has too many '/'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/pyenv/versions/3.7.5/lib/python3.7/site-packages/conans/client/loader.py", line 352, in _parse_conanfile
    loaded = imp.load_source(module_id, conan_file_path)
  File "/opt/pyenv/versions/3.7.5/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 860, in get_code
  File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/conan/workspace/onan-center-pull-request_PR-2129/recipes/flatcc/all/conanfile.py", line 57
    def source(self):
      ^
SyntaxError: invalid syntax


@conan-center-bot
Copy link
Collaborator

Some configurations of 'flatcc/0.6.0' failed in build 8 (26a54dc1d7fb681b84ca863e851fc4e81fb5d492):

self._cmake.definitions["FLATCC_FAST_DOUBLE"] = self.options.fast_double
self._cmake.definitions["FLATCC_IGNORE_CONST_COND"] = self.options.ignore_const_condition
self._cmake.definitions["FLATCC_TEST"] = False
self._cmake.configure(source_folder=os.path.join(self.source_folder, self._source_subfolder))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By running the cmake script from the source_subfolder directly,
you're not executing

include(conanbuildinfo.cmake)
conan_basic_setup()

Which causes the compiler to use the wrong runtime. This is a problem with MSVC.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for (again) saving me a lot of analysis time. Strangely the current recipe builds fine on my Windows 10 VM with Visual Studio 2019.
I will try to fix the Windows build later this week, some other priorities came up.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is always with build_Type=Debug and runtime=MTd or MDd.
Real troublemakers, those runtimes are... 😄

… script from the source_subfolder directly conan_basic_setup() is not executed and MSVC chooses wrong runtime.
@conan-center-bot
Copy link
Collaborator

Some configurations of 'flatcc/0.6.0' failed in build 9 (28fcfa0b3797ee221af28bca9639f7ab3cc44fe6):

Winfried Dobbe added 2 commits July 7, 2020 14:02
… script from the source_subfolder directly conan_basic_setup() is not executed and MSVC chooses wrong runtime.
@conan-center-bot
Copy link
Collaborator

Some configurations of 'flatcc/0.6.0' failed in build 11 (3dd80ae9bdb225da5e43466f2fbae49ee0070462):

@conan-center-bot
Copy link
Collaborator

All green in build 12 (1d6bc780ae8a20d87f896aa4b22e1a00d8292ff1)! 😊

@@ -0,0 +1,23 @@
cmake_minimum_required(VERSION 3.12)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
cmake_minimum_required(VERSION 3.12)
cmake_minimum_required(VERSION 2.8)

too simple for cmake 3+

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but the problem is that I never test with cmake 2.8. So yes, that cmake file will probably work with 2.8, but I have no evidence...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keep cmake 2.8, if you don't use an explicit cmake 3 feature, keep 2.8. It is still used.

wdobbe and others added 3 commits July 7, 2020 17:10
Co-authored-by: Uilian Ries <uilianries@gmail.com>
Co-authored-by: Uilian Ries <uilianries@gmail.com>
Co-authored-by: Uilian Ries <uilianries@gmail.com>
@conan-center-bot
Copy link
Collaborator

All green in build 14 (c8d46f1c1c1cdc1a53e48f46fe34dfa48da15097)! 😊

@conan-center-bot
Copy link
Collaborator

All green in build 15 (951bc0173a4b16577839cc51971f0bca9aa2d688)! 😊

Co-authored-by: Uilian Ries <uilianries@gmail.com>
Copy link
Member

@uilianries uilianries left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

almost good to go, thanks for your effort

@SSE4
Copy link
Contributor

SSE4 commented Jul 7, 2020

CI stuck?

@uilianries
Copy link
Member

Yes, even license CLA is frozen.

Co-authored-by: Uilian Ries <uilianries@gmail.com>
@wdobbe
Copy link
Contributor Author

wdobbe commented Jul 7, 2020

almost good to go, thanks for your effort

Thanks also for your feedback, it was quite an educational exercise.

@conan-center-bot
Copy link
Collaborator

All green in build 16 (ad8fcfeb3370c5811b64fbeeba5a994cad0ecd7d)! 😊

Comment on lines 114 to 117
if tools.os_info.is_linux:
self.env_info.LD_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib"))
if tools.os_info.is_macos:
self.env_info.DYLD_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really think this is not needed. Can you provide a scenario where you need the recipe to populate these variables?

Suggested change
if tools.os_info.is_linux:
self.env_info.LD_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib"))
if tools.os_info.is_macos:
self.env_info.DYLD_LIBRARY_PATH.append(os.path.join(self.package_folder, "lib"))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These lines can be removed. But my concern is that someone creates a Conan recipe for this project that uses flatcc (via cmake custom command) following the documentation to create the build() function. Then when he tries to create the package he will get the dreaded flatcc: error while loading shared libraries might now know how to fix it.
I know the RunEnvironment is documented in the Conan reference pages but it is somewhat hidden. I actually didn't know about it until submitting this PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then we need to improve or highlight those lines in the docs (btw, using the two profiles approach, which will be the default in the future, there is no need to add the "RunEnvironment").

If we keep these lines here, then we need to add them to every recipe in Conan Center for the very same reason... and in that scenario, it would be something to manage inside Conan itself... and that's exactly what we have done.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the (DY)LD_LIBRARY_PATH lines. But is it still necessary to append the bin path to the PATH variable in package_info() ? Since RunEnvironment does that as well.

Copy link
Contributor Author

@wdobbe wdobbe Jul 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edit: disregard this comment, it was fixed by @madebr.

@jgsogo I created a workaround in the test_package to run flatcc in the build function as you suggested, but this also fails because flatcc can't find its shared libraries. On my Hackintosh (which has SIP disabled) the test_package builds and runs fine for all build_type and 'shared' combinations.

Because dlopen searches for shared libraries in the current working directory another workaround would be to add an imports() function in the test_package recipe that copies the flatcc shared libs. But as soon as I add an imports() function the build is rejected by CI because the import() function is not allowed.
I could copy the flatcc dylib in the build function before running the flatcc executable. If you think that is acceptable than I can test that at work where we have a real Mac.

Co-authored-by: Javier G. Sogo <jgsogo@gmail.com>
@conan-center-bot
Copy link
Collaborator

Some configurations of 'flatcc/0.6.0' failed in build 17 (97d0a81fe499ddabc315efaab0b25dbc15e7e1c8):

…RY_PATH, remove redundant check for Windows in package() function
@conan-center-bot
Copy link
Collaborator

Some configurations of 'flatcc/0.6.0' failed in build 18 (4cce1c2491ca325f058353d70b7d45ff8c7090d8):

@conan-center-bot
Copy link
Collaborator

Some configurations of 'flatcc/0.6.0' failed in build 19 (1e408693a8c74e5c7a495f8e480afbe7de57a462):

Co-authored-by: Anonymous Maarten <madebr@users.noreply.github.com>
@conan-center-bot
Copy link
Collaborator

All green in build 20 (8dfbe08dc5947b23ff08236234f027b97150267c)! 😊

Copy link
Member

@uilianries uilianries left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@jgsogo jgsogo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for the nice workaround and all the explanations around the MacOS System Integrity Protection. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[request] <flatcc>/<0.6.0>
6 participants