python.extension_module() linking with libpython when it should not

[dnicolodi]

Since version 0.63.0 Meson automatically adds a dependency on libpython when needed compiling extension modules. To determine whether to link to libpython Meson consults distutils. On Python 3.7 on Linux, distutils says that libpython should not be linked:

>>> import distutils.core
>>> cmd = distutils.core.Distribution().get_command_obj('build_ext')
>>> cmd.ensure_finalized()
>>> cmd.get_libraries(distutils.core.Extension('dummy', []))
[]

However, the pkg-config file for python contains these linker arguments

$ pkg-config python3 --libs
-L/opt/_internal/cpython-3.7.15/lib -lpython3.7m

This code

meson/mesonbuild/modules/python.py

Lines 524 to 543 in fd43842

	new_deps = []
	has_pydep = False
	for dep in mesonlib.extract_as_list(kwargs, 'dependencies'):
	if isinstance(dep, _PythonDependencyBase):
	has_pydep = True
	# On macOS and some Linux distros (Debian) distutils doesn't link
	# extensions against libpython. We call into distutils and mirror its
	# behavior. See https://github.com/mesonbuild/meson/issues/4117
	if not self.link_libpython:
	dep = dep.get_partial_dependency(compile_args=True)
	new_deps.append(dep)
	if not has_pydep:
	pydep = self._dependency_method_impl({})
	if not pydep.found():
	raise mesonlib.MesonException('Python dependency not found')
	new_deps.append(pydep)
	FeatureNew.single_use('python_installation.extension_module with implicit dependency on python',
	'0.63.0', self.subproject, 'use python_installation.dependency()',
	self.current_node)
	kwargs['dependencies'] = new_deps

deals with adding the libpython dependency as need. In the case the libpython dependency is explicitly requested, only the compile arguments are used, however, in the case when it is not explicitly requested, all the arguments are added, including the link arguments found via pkg-config. I think that only the compile arguments should be included in this case too when the extension module should not be linked with libpython.

I'll prepare a PR, but I would like to check with @eli-schwartz if he agrees on the diagnosis.

[eli-schwartz]

This diagnosis confuses me since we specifically check for link_libpython and remove the link arguments (keeping only the compile ones).

However, on 3.7 you are supposed to link to libpython, except on Debian...

[eli-schwartz]

Oh, I think I see it now.

Edit: or not? This should already work?

[dnicolodi]

I think the patch makes it clearer. I actually encountered the issue on the manylinux docker images, which are based on AlmaLinux. I suspect Debian may be patching the pkg-condig too.

[eli-schwartz]

The thing is that the patch is exactly what I thought you meant, and I don't see how that's possible to be a problem.

The internal dependency is embed: false. If you provide one we don't know what it is so we partial it.

[dnicolodi]

I stepped through the thing with pdb and the linker flags are added. I'm afraid I don't know enough of the surrounding code to understand exactly why. However, I just verified that this solves the issue on Linux and a segfault on macOS.

The pkg-config definition oh the system looks like this:

# cat /opt/_internal/cpython-3.7.15/lib/pkgconfig/python3.pc
# See: man pkg-config
prefix=/opt/_internal/cpython-3.7.15
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: Python
Description: Python library
Requires:
Version: 3.7
Libs.private: -lcrypt -lpthread -ldl  -lutil
Libs: -L${libdir} -lpython3.7m
Cflags: -I${includedir}/python3.7m

Let me know if there is anything else that may be useful to better understand the root cause.

Edit: yes, I verified that this is the pkg-config picked up by Meson.

[dnicolodi]

I think the bug is most likely in the discrepancy between distutils and pkg-config for Python 3.7. But fixing it in the Python seems unlikely to happen at this point. I think that dealing with it in Meson is the only option.

[dnicolodi]

Also note that the distinction from python3.pc and python3-embed.c comes only starting with Python 3.8, so as long as Meson supports Python 3.7, we need to filter out the link flags.

[dnicolodi]

The internal dependency is embed: false.

I had a closer look at the Meson python module code and I understood better what you mean with the above sentence. When the dependency is looked up with embed=False the python3.pc is looked up and when embed=True the python3-embed.pc is used instead. As I was writing above, up to Python 3.7 there is no python3-embed.pc thus this does not have the effect of removing the linking to libpython as per the python3.pc reported above.

[eli-schwartz]

The internal dependency is embed: false. If you provide one we don't know what it is so we partial it.

So for clarity, I looked at this last night, and I apologize, because what I said was incredibly bogus and clueless. I double and triple checked, and actually py.dependency() itself is broken with pkg-config, then crudely hacked around inside extension_module using partial_dependency.

The correct fix is: eli-schwartz@889f913

...

When I added the implicit dependency I did probably assume that embed: false actually did what it's supposed to, and didn't bother to copy the hack over. And yesterday, I assumed that embed: false actually did what it's supposed to, and the hack "must have some reason to exist, not just that py.dependency() is broken".

[dnicolodi]

Thanks @eli-schwartz Your fix looks good to me, FWIW.

[eli-schwartz]

Alright, PR'ed.