Embed C API into Python extension

[mtreinish]

Summary

This commit adds the C extension crate into the Python extension build.
This is done by adding the qiskit-cext crate as a dependency for
qiskit-pyext. Then the contents of cext are re-exported from pyext. This
enables linking against the _accelerate shared library object using the
C API. This facilitates writing compiled Python extensions that
interface with Qiskit directly via C. For example, building a
SparseObservable object in a compiled language via the C API and then
returning that to Python for use with the rest of Qiskit.

Details and comments

This PR is based on top of #13694 and will need to be rebased once that merges. To view the contents of just this PR you can look at the HEAD commit on this PR branch: f2aa474

[qiskit-bot]

One or more of the following people are relevant to this code:

• @Qiskit/terra-core

[coveralls]

Pull Request Test Coverage Report for Build 13695147517

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 13 (0.0%) changed or added relevant lines in 1 file are covered.
• 1588 unchanged lines in 60 files lost coverage.
• Overall coverage increased (+1.0%) to 88.198%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
crates/cext/src/sparse_observable.rs	0	13	0.0%

Files with Coverage Reduction	New Missed Lines	%
qiskit/circuit/_classical_resource_map.py	1	76.62%
qiskit/circuit/commutation_checker.py	1	94.12%
qiskit/circuit/controlflow/_builder_utils.py	1	96.97%
qiskit/circuit/parameter.py	1	96.72%
qiskit/compiler/transpiler.py	1	98.78%
qiskit/primitives/utils.py	1	92.86%
qiskit/result/distributions/quasi.py	1	95.92%
qiskit/transpiler/passes/layout/apply_layout.py	1	96.67%
qiskit/transpiler/passes/optimization/contract_idle_wires_in_control_flow.py	1	98.04%
crates/accelerate/src/unitary_synthesis.rs	2	94.39%

Totals
Change from base Build 13673594587:	1.0%
Covered Lines:	71854
Relevant Lines:	81469

---

💛 - Coveralls

[jakelishman]

I think building this into the Python-space object is fine, but I don't think we should modify qiskit/__init__.py, given that I think this way of suggesting the linkage is not our intended long-term solution, and in the short term it's really fragile.

There's so much fiddling you need to do compared to a normal build process to build an executable against this current version, which is fine for our MVP, but I think we need a lot more guide on how to make it work, the caveats of it all, and I think it oughtn't to be visible to Python users.

[jakelishman]

For posterity: I was able to use this PR to build the test executable in crates/cext/README.md on my Raspberry Pi (which is Linux with a GNU build chain). The input file obs.c:

#include <Python.h>
#include <complex.h>
#include <qiskit.h>
#include <stdint.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    // build a 100-qubit empty observable
    uint32_t num_qubits = 100;
    QkSparseObservable *obs = qk_obs_zero(num_qubits);

    // add the term 2 * (X0 Y1 Z2) to the observable
    complex double coeff = 2;
    QkBitTerm bit_terms[3] = {QkBitTerm_X, QkBitTerm_Y, QkBitTerm_Z};
    uint32_t indices[3] = {0, 1, 2};
    QkSparseTerm term = {coeff, 3, bit_terms, indices, num_qubits};
    qk_obs_add_term(obs, &term);

    // print some properties
    printf("num_qubits: %u\n", qk_obs_num_qubits(obs));
    printf("num_terms: %lu\n", qk_obs_num_terms(obs));

    // free the memory allocated for the observable
    qk_obs_free(obs);

    return 0;
}

Note that I've inserted #include <Python.h> up top, even though I don't use it in the source file, just to account for the problem in qiskit.h.

I built Qiskit just with python setup.py build_rust --inplace.

Then the build flags:

# Where the root of my Qiskit checkout is.
QISKITROOT="$HOME/code/qiskit/terra"
# Where the root of my Python dev components are.
PYROOT="$HOME/code/pyenv/versions/3.13.2"

CFLAGS="-I$QISKITROOT/dist/c/include -I$PYROOT/include/python3.13"
LDFLAGS="-Wl,-rpath,$QISKITROOT/qiskit,-rpath,$PYROOT/lib -L$QISKITROOT/qiskit -l:_accelerate.cpython-313-aarch64-linux-gnu.so -L$PYROOT/lib -lpython3.13"

gcc $CFLAGS obs.c -o obs $LDFLAGS

I inserted the libraries in to the rpath during linking, but you could equally choose to insert libpython3.13.so at runtime with LD_LIBRARY_PATH.

I haven't managed to work out the incantations you need to get the Apple linker to use a library file with an "incorrect" name.

[jakelishman]

This doesn't seem to be working - the generated file still has the prefix.

I'm fine just using renaming_overrides_prefixing = true and manually adding Qk to the renames for now, while we figure out a nicer pattern.

Probably to do later (because of additional infrastructure), but I think we ought to consider adding a test of the Python-interfacing method.

[Cryoris]

To add to the above: the current version with the manual renaming works (on Mac) and with the up-to-date main we can compile a Qiskit C Python extension that creates a SparseObservable and is recognized as Python SparseObservable. The test looked something like

from qiskit import transpile
from qiskit.circuit import QuantumCircuit
from qiskit.circuit.library import PauliEvolutionGate
from qiskit.quantum_info import SparseObservable

import cextension  # C Python extension

if __name__ == "__main__":
    obs = cextension.build_observable()
    print("-- SparseObservable")
    print(obs)
    print(isinstance(obs, SparseObservable))  # True

    # build an evolution circuit
    evo = PauliEvolutionGate(obs, time=2)
    circuit = QuantumCircuit(obs.num_qubits)
    circuit.append(evo, circuit.qubits)

    tqc = transpile(circuit, basis_gates=["u", "cx"]) 
    print("-- Evolution circuit")
    print(tqc.draw(idle_wires=False))

The thing I had to change from Jake's workflow above it so symlink _accelerate.<....>.so -> libqiskit.so, since the Mac linker wasn't happy with the original library name.

[jakelishman]

If you're actually building a completely custom C extension for Python, you shouldn't need to do the -rpath stuff I did to insert libpython at load time (or alternatively the LD_LIBRARY_PATH [Linux] or DYLD_LIBRARY_PATH [Mac] bits), because Python should arrange for those to be preloaded before loading an extension module.

On Mac (or more strictly, when using the LLVM linker toolchain), I couldn't work out a set of commands to use a library file with a bad name (i.e. other than lib<library>.so [Linux] or lib<library>.dylib [Mac]), so symlink seems to be the short-term way to go.

For SparseObservable, strictly, I think we'd see the same behaviour if we copied the .so file (not symlink), but I think that will be very fragile, because we'll have two address spaces. That would break things like the uniqueness-tracking globals for QuantumCircuit (Bit, in particular) once they're referenced by C, and anyway, it'd be duplicating a load of stuff in the memory space.

[mtreinish]

This needs to move to the feature so we only directly depend on pyo3 in pyyhon_binding mode

[Cryoris]

But implicitly we'd still depend on pyo3 through qiskit-accelerate, until we move the Rust-only code into a separate crate, right?

[jakelishman]

cext itself only depends on pyo3 if the Python feature is active. It happens to have transitive dependencies that depend on pyo3 anyway, but that's an implementation detail as far as it's concerned.

[jakelishman]

Thanks everyone. We need to add more documentation on this, but we can follow up after 2.0rc1.