tools.collect_libs should track library dependency order

[kmaragon]

Many packages are apt to create multiple libraries. Boost, for example, produces many. Often, within the set of libraries that get exported, there are inter-dependencies.

For example, in boost, asio dpends on coroutine which depends on context.
In OpenSSL, ssl depends on crypto.
conan.collect_libs should use ldd and other tools to determine a dependency graph of libraries and sort accordingly (in reverse). When compiling any package statically, the library link order is very important. if you try to link against a static OpenSSL and collect_libs were used, if the OS returned crypto first and then ssl, the link order would be libcrypto.a libssl.a and that will break because libssl.a depends on libcrypto.a and the gcc linker needs libraries in reverse dependency order.

Combined with this, if a package is using tools.collect_libs it's usually because the package is going to be producing several libraries based on the options provided, rather than a small known set. But since tools.collect_libs basically makes no guarantees about ordering, it's dangerous to use for static libraries because it will produce link orders that fail to compile. So either tools.collect_libs ought to be removed and maintainers be left with the headache of managing sets of libraries that need to be linked or it should be redone to actually track dependency ordering.

[memsharded]

conan.collect_libs should use ldd and other tools to determine a dependency graph of libraries and sort accordingly (in reverse).

This is a very complicated feature, to do it portable and robustly. Also it would make package_info() slower, which can seriously affect performance, even when the requested package binaries are already cached in the local cache.

So either tools.collect_libs ought to be removed and maintainers be left with the headache of managing sets of libraries that need to be linked or it should be redone to actually track dependency ordering.

Not really. There are other use cases: for example, when a package is creating more than one library that they do not depend on each other. Or to collect libraries names, because the build system is generating complex names like libmylib-x86-debug-en.lib, etc, which will be changing from configuration to configuration. collects_libs might be still useful, even if you have to define the order yourself:

def package_info(self):
      """ sure it can be implemented better, and this is not tested at all, just an idea
      """
      existing = tools.collect_libs(self)
      libs = []
      for lib in ["mylib", "myotherlib", "thirdlib"]:  # in correct linking order
           for ex in existing:
               if existing.startswith("lib%s" % lib):
                     libs.append(ex)
                     break
      self.cpp_info.libs = libs

So it is not going to be removed, it can still be useful as-is. In any case, I suggest adding a clear warning to the docs.

The requested link-order feature is not going to be implement by us, we have tried similar things in the past and it is almost impossible to do it right, and half-baked approachs lead to nightmares of debugging, user issues, and pain. Better keep it simple. But, if you want to try to contribute this link-order feature, of course it is very welcome! Otherwise, I think that with adding that warning to the docs we should be good.

Thanks very much for your feedback!

[forlayo]

Something that can help so much ( almost on Android/Linux platforms ) is to allow adding the flags

-Wl, --start-group
...
-Wl, --end-group

You know? to do something like:

-Wl, --start-group
/path/to/file.a
/path/to/file2.a
-lfile3
...
-Wl, --end-group

And then resolve the inter dependency automatically. I've been using it on my projects with success.

Links:

• https://linux.die.net/man/1/ld
• https://stackoverflow.com/questions/45135/why-does-the-order-in-which-libraries-are-linked-sometimes-cause-errors-in-gcc/409470#409470
• https://stackoverflow.com/questions/5651869/gcc-what-are-the-start-group-and-end-group-command-line-options

[lasote]

@forlayo interesting, thanks!

[memsharded]

Thanks for the feedback!

The implementation of relative order between libs belonging to the same package has now a dedicated issue: #2387

That issue will cover defining ordering for both "collect_libs", but also to manually defined libraries. Please follow up in that other issue. Thanks!