[windows] fastkde==2.1.3 nuifft fails when running with numpy < 2: ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'long'

[jvesely]

Probably related to changes to type width in numpy 2 [0,1]

[0] https://numpy.org/doc/2.1/release/2.0.0-notes.html#default-integer-dtype-is-now-64-bit-on-64-bit-windows
[1] https://numpy.org/doc/2.1/numpy_2_0_migration_guide.html#migration-windows-int64

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
psyneulink\core\globals\context.py:742: in wrapper
    return func(*args, context=context, **kwargs)
psyneulink\core\compositions\parameterestimationcomposition.py:965: in run
    results = super(ParameterEstimationComposition, self).run(*args, **kwargs)
psyneulink\core\globals\context.py:742: in wrapper
    return func(*args, context=context, **kwargs)
psyneulink\core\compositions\composition.py:11522: in run
    trial_output = self.execute(inputs=execution_stimuli,
psyneulink\core\globals\context.py:742: in wrapper
    return func(*args, context=context, **kwargs)
psyneulink\core\compositions\composition.py:12249: in execute
    self._execute_controller(
psyneulink\core\compositions\composition.py:11840: in _execute_controller
    self.controller.execute(context=context)
psyneulink\core\globals\context.py:742: in wrapper
    return func(*args, context=context, **kwargs)
psyneulink\core\components\mechanisms\mechanism.py:2557: in execute
    value = self._execute(variable=variable,
psyneulink\core\compositions\parameterestimationcomposition.py:1331: in _execute
    super(ControlMechanism,self)._execute(
psyneulink\core\components\component.py:3432: in _execute
    value = self.function(variable=function_variable, context=context, params=runtime_params, **kwargs)
psyneulink\core\components\functions\function.py:686: in __call__
    return self.function(*args, **kwargs)
psyneulink\core\globals\context.py:742: in wrapper
    return func(*args, context=context, **kwargs)
psyneulink\core\components\functions\function.py:733: in function
    value = self._function(
psyneulink\core\components\functions\nonstateful\fitfunctions.py:508: in _function
    results = self._fit(obj_func=f, context=context)
psyneulink\core\components\functions\nonstateful\fitfunctions.py:565: in _fit
    return self._fit_optuna(
psyneulink\core\components\functions\nonstateful\fitfunctions.py:832: in _fit_optuna
    study.optimize(
D:\a\_temp\_venv\Lib\site-packages\optuna\study\study.py:475: in optimize
    _optimize(
D:\a\_temp\_venv\Lib\site-packages\optuna\study\_optimize.py:63: in _optimize
    _optimize_sequential(
D:\a\_temp\_venv\Lib\site-packages\optuna\study\_optimize.py:160: in _optimize_sequential
    frozen_trial = _run_trial(study, func, catch)
D:\a\_temp\_venv\Lib\site-packages\optuna\study\_optimize.py:248: in _run_trial
    raise func_err
D:\a\_temp\_venv\Lib\site-packages\optuna\study\_optimize.py:197: in _run_trial
    value_or_values = func(trial)
psyneulink\core\components\functions\nonstateful\fitfunctions.py:795: in objfunc_wrapper_wrapper
    return objfunc_wrapper(list(trial.params.values()))
psyneulink\core\components\functions\nonstateful\fitfunctions.py:594: in objfunc_wrapper
    obj_val = obj_func(*x)
psyneulink\core\components\functions\nonstateful\fitfunctions.py:482: in objfunc
    return self._pec_objective_function(sim_data)
psyneulink\core\compositions\parameterestimationcomposition.py:827: in f
    like = simulation_likelihood(
psyneulink\core\components\functions\nonstateful\fitfunctions.py:192: in simulation_likelihood
    fKDE = fastKDE.fastKDE(dsub, do_save_marginals=False)
D:\a\_temp\_venv\Lib\site-packages\fastkde\fastKDE.py:392: in __init__
    ecfObj = ecf.ECF(
D:\a\_temp\_venv\Lib\site-packages\fastkde\empiricalCharacteristicFunction.py:120: in __init__
    myECF = nufft.nuifft(
src/fastkde/nufft.pyx:30: in fastkde.nufft.nuifft
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>   ???
E   ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'long'
src/fastkde/nufft.pyx:119: ValueError

[jvesely]

The failing line

fastkde/src/fastkde/nufft.pyx

Line 119 in fe626ab

cdef np.int64_t [:] frequencySizes = np.zeros([numDimensions],dtype=np.int_)

is assigning and array of dtype np.int_ (32bit on windows and numpy<2) to array reference of dtype np.int64, so I guess the failure is expected

[taobrienlbl]

thanks @jvesely for raising this issue and identifying the cause! I think the likely fix will be to set dtype=np.int64_.

A short-term fix might be to install a version of fastkde less than 2.1.1, which is when support for numpy>2 was introduced and the change to line 119 was made to support numpy 2 (previously it was cdef np.int_). Note that there is a bug in fastkde.conditional in versions less than 2.1.1, but based on your above stack trace, it looks like you're using fastKDE.fastKDE directly, so maybe that's not an issue.

[jvesely]

thanks for looking at this. there appear to be several instances of this problem in the nufft.pyx file:

~/py313/lib/python3.13/site-packages $ grep np.int_ -r fastkde
fastkde/nufft.pyx:    cdef np.int64_t [:] frequencySizes = np.zeros([numDimensions],dtype=np.int_)
fastkde/nufft.pyx:    #cdef np.ndarray[np.int64_t,ndim=1] hyperSlabShape = nspread*np.ones([numDimensions],dtype=np.int_)
fastkde/nufft.pyx:    cdef np.int64_t [:] hyperSlabShape = nspread*np.ones([numDimensions],dtype=np.int_)
fastkde/nufft.pyx:    #cdef np.ndarray[long,ndim=1] mvec = np.zeros([numDimensions],dtype=np.int_)
fastkde/nufft.pyx:    cdef np.int64_t [:] mvec = np.zeros([numDimensions],dtype=np.int_)
fastkde/nufft.pyx:    cdef np.int64_t [:] m0vec = np.zeros([numDimensions],dtype=np.int_)
fastkde/nufft.pyx:    cdef np.int64_t [:] dimInds = np.zeros([numDimensions],dtype=np.int_)
fastkde/nufft.pyx:    cdef np.int64_t [:] frequencySizes = np.zeros([numDimensions],dtype=np.int_)
fastkde/nufft.pyx:    cdef np.int64_t [:] dimInds = np.zeros([numDimensions],dtype=np.int_)
fastkde/nufft.pyx:    cdef np.int64_t [:] frequencySizes = np.zeros([numDimensions],dtype=np.int_)
fastkde/nufft.pyx:    cdef np.int64_t [:] dimInds = np.zeros([numDimensions],dtype=np.int_)

we're currently using fastkde==2.0.1 as that is the latest older version available on pypi which works great for us.

We ran into this issue when testing our code with numpy 2+, because fastkde==2.0.1 doesn't work with numpy 2+:

    from fastkde import fastKDE
../py312/lib/python3.12/site-packages/fastkde/__init__.py:10: in <module>
    import fastkde.fastKDE
../py312/lib/python3.12/site-packages/fastkde/fastKDE.py:5: in <module>
    import fastkde.empiricalCharacteristicFunction as ecf
../py312/lib/python3.12/site-packages/fastkde/empiricalCharacteristicFunction.py:3: in <module>
    import fastkde.nufft as nufft
nufft.pyx:1: in init fastkde.nufft
    ???
E   ImportError: numpy.core.multiarray failed to import (auto-generated because you didn't call 'numpy.import_array()' after cimporting numpy; use '<void>numpy._import_array' to disable if you are certain you don't need it).

This is expected since packages built against numpy 1.x are not expected to work with numpy 2.

However, it means that both faskde and numpy versions must be moved together, which makes fixing our code to support numpy 2 a bit cumbersome.