Fix binary extraction flow #7
Conversation
The code has been updated to include support for macOS in the build and extraction process of the project. A new section is added to perform build and compression for macOS, in addition to existing mechanisms for Windows and Linux. The extraction method in engines.py is also modified to handle macOS specific binaries.
rishabh-sagar-20
changed the title
|
Hey @rishabh-sagar-20, running into some issues getting this packaged on PyPi due to the large binary size of the build redlining engines. I have some thoughts on how to fix this, but still working on it. If you have thoughts, definitely welcome any suggestions. Max file size we can put on PyPi is 100 MB. I'm thinking we don't distribute the binaries and instead build them on the client OR host them on AWS or something and have an install step to pull the binaries after install - like how Spacy and other NLP libraries often download large binaries. |
|
It's feasible. Additionally, we can create distinct builds for each platform. By doing so, you can circumvent size constraints and upload it to PyPi. |
|
What's the overall size? I came across a page stating that we can discuss with PyPi to waive the size limit. |
|
Yes, I did see that, but it seems the limit is 60 MB (compressed) and the binaries (at least on my machine) are about 65 - 80 MB each. |
|
I think the method you mentioned earlier, which is similar to Spacy models, should be effective. For hosting, GitHub Releases should work fine. We can update it later if we encounter any issues. |
Refactor:
Update: