Prebuilt executables crash in release 0.9.3

[shclift]

The latest version (0.9.3) of pyfdax_win.exe fails on a couple of Win 11 systems I tried it on.

C:\Users\sclif\Downloads>pyfdax_win.exe
Traceback (most recent call last):
  File "jschon\catalog\__init__.py", line 46, in __call__
  File "jschon\utils.py", line 48, in json_loadf
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\sclif\\AppData\\Local\\Temp\\_MEI207842\\jschon\\catalog\\json-schema-2020-12\\schema.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pyfda\pyfdax.py", line 23, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 384, in exec_module
  File "pyfda\pyfda_rc.py", line 21, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 384, in exec_module
  File "pyfda\libs\pyfda_lib.py", line 114, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 384, in exec_module
  File "amaranth\__init__.py", line 13, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 384, in exec_module
  File "amaranth\hdl\__init__.py", line 13, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 384, in exec_module
  File "amaranth\hdl\_rec.py", line 10, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "PyInstaller\loader\pyimod02_importers.py", line 384, in exec_module
  File "amaranth\lib\wiring.py", line 1725, in <module>
  File "abc.py", line 106, in __new__
  File "amaranth\lib\meta.py", line 64, in __init_subclass__
  File "amaranth\lib\meta.py", line 37, in __jschon_schema
  File "jschon\__init__.py", line 49, in create_catalog
  File "jschon\catalog\_2020_12.py", line 103, in initialize
  File "jschon\catalog\__init__.py", line 215, in create_metaschema
  File "jschon\catalog\__init__.py", line 158, in load_json
  File "jschon\catalog\__init__.py", line 53, in __call__
jschon.exc.CatalogError: No such file or directory: 'C:\\Users\\sclif\\AppData\\Local\\Temp\\_MEI207842\\jschon\\catalog\\json-schema-2020-12\\schema.json'
[PYI-19192:ERROR] Failed to execute script 'pyfdax' due to unhandled exception!

[chipmuenk]

Confirmed, also on Win10 and Ubuntu :-(

The issue is related to amaranth and / or pyinstaller. I'll try to use newer releases, if this doesn't work I'll exclude amaranth from the requirements.

[chipmuenk]

Simply re-building (pre-release v0.9.4b1) did not improve things.

Root cause seems to be related to the jschon library that is used by amaranth. A pip install pyfda fetches all needed components but pyinstaller seems to miss something. I'm removing amaranth from the requirements for now, it can be easily installed afterwards with a pip install amaranth[builtin-yosys] if needed. The amaranth fixpoint filter is commented out in pyfda.conf anyway, otherwise it just gives a warning about "amaranth not found".

[shclift]

Slightly off topic, but I want to thank you for pyfda. It has been my go-to program for analyzing and/or designing digital filters since I stumbled upon it several years ago.

[chipmuenk]

Also off topic ;-) : You would help me a lot by giving me a few hints (per PM or here) about what you like / dislike / miss in pyfda. It's the usual open source problem of developing for unknown users ...

[chipmuenk]

The reason for the crash was indeed that pyinstaller does not include the '*.json' data files of the jschon package as pointed out by the amaranth community amaranth-lang/amaranth#1549 . They can be added manually by the following snippet in the spec file for pyinstaller:

import os
import importlib
# add all json files from jschon package (used by amaranth) to datas. The directory
# specified in the second part of the datas tuple is only the jschon/... dir part, e.g.
# ('python3.10/site-packages/jschon/catalog/json-schema-2020-12/schema.json',
#  'jschon/catalog/json-schema-2020-12')
jschon_root = os.path.dirname(importlib.import_module('jschon').__file__)  # root dir of jschon package
json_files = [(os.path.join(root, name), 
               os.path.join('jschon', root.partition('jschon/')[2]))
                for root, dirs, files in os.walk(jschon_root)
                    for name in files
                        if name.endswith(".json")]
datas += json_files

Unfortunately, amaranth still doesn't play nice with pyinstaller, I now get the warning "

Error during import of "pyfda.fixpoint_widgets.fir_df.fir_df_amaranth_ui":
invalid literal for int() with base 10: 'unknown'

[whitequark]

What's the backtrace for that? It might be an issue with this code in amaranth/__init__.py:

# Extract version for this package from the environment package metadata. This used to be a lot
# more difficult in earlier Python versions, and the `__version__` field is a legacy of that time.
import importlib.metadata
try:
    __version__ = importlib.metadata.version(__package__)
except importlib.metadata.PackageNotFoundError:
    # No importlib metadata for this package. This shouldn't normally happen, but some people
    # prefer not installing packages via pip at all. Although not recommended we still support it.
    __version__ = "unknown" # :nocov:
del importlib

[whitequark]

It might be a problem with your own code here:

pyfda/pyfda/libs/pyfda_fix_lib_amaranth.py

Line 17 in cb152c8

if cmp_version("amaranth", "0.3") >= 0:

[chipmuenk]

True, I was not prepared to process a __version__ string "unknown" - sorry & thanks for helping with my own code :-) , I fixed that in v0.9.4.

Now that I've got your attention ;-) - somewhat off-topic: I have been struggling to embed myHDL / migen / nmigen / amaranth into my code literally for years now. It's not so much the actual RTL description (I have some Verilog / VHDL background), I have difficulties wrapping my head around the embedding of amaranth code into my python project. In principle, simulation and Verilog export with amaranth work but my current problem is a max recursion error after some 140 steps.

I know I'm not interacting correctly with my amaranth module, but I'm a bit tired of running into walls. Do you know someone from the amaranth project who could discuss the basic concepts with me some other time? As an alternative, I've started looking into using a Jinja/Verilog based template approach for generating code and possibly icarus / cocotb for simulation / verification which might be a better suited for small dsp blocks with parametrizable word length.

[whitequark]

Do you know someone from the amaranth project who could discuss the basic concepts with me some other time?

You could file an issue or open a discussion on the https://github.com/amaranth-lang/amaranth/ repository, or you could reach out via IRC (#amaranth-lang at libera.chat) or Matrix (#amaranth-lang:matrix.org) to have someone assist you.

but my current problem is a max recursion error after some 140 steps.

This isn't strictly speaking enough information to figure out what the problem is, but you probably generate very large expressions that need to be broken up into chunks with m.d.comb += intermediate.eq(...). You will actually get the same problem in Python itself; try running a file that has, for example, ten thousand elif: branches in a single if: statement. However, it is more visible in Amaranth due to the ease with which you can generate code.

Do you have a reproducer?

As an alternative, I've started looking into using a Jinja/Verilog based template approach for generating code and possibly icarus / cocotb for simulation / verification which might be a better suited for small dsp blocks with parametrizable word length.

That would almost certainly be more painful to contributors, especially those on Windows.

[chipmuenk]

Fixed in 0.9.5b1