fix: remove Python 2 legacy packaging code

[henryiii]

A small update to packaging code; universal=True/1 is only used to replace the py3 tag with py2.py3, which is incorrect since 1.0. Also adding python_requires - this should always be set as soon as a python version is dropped, as pip will back-solve to find the final release that has a passing python_requires for the current Python version.

If you are interested, I'd recommend moving to static configuration (everything in setup.cfg or pyproject.toml). I'd recommend reading https://scikit-hep.org/developer/packaging or https://packaging.python.org/en/latest/tutorials/packaging-projects/ for the classic way to do this, or https://scikit-hep.org/developer/pep621 for the new way. (We are working on transitioning packaging.python.org to the PEP 621 method as well in the near future).

My initial goal was to see if mypyc would provide the same speedup for CPython that PyPy provided, but there are some missing static types¹. :'( I did compare a few versions of Python 3 to see if the faster CPython project really is working. I've got a fast computer and I wasn't able to generate a matching json file (2.9 MB for me), but here are the relative timings for the slowest parser if you were curious:

Version	Time
CPython 3.6	46.34 secs
CPython 3.9	42.05 secs
CPython 3.10	37.25 secs
CPython 3.11b1	29.31 secs
PyPy 3.7	8.46 secs

Footnotes

1. there might actually be enough to try it! ↩

[erezsh]

Thanks, I'll have a look.

I like using a pyproject.toml, except that I have to update the version twice, once there and once in the __init__.py file. I think there's supposed to be a way to read the package info from Python, but it didn't work when I tried.

but there are no static types

What do you mean?

[erezsh]

Also, very nice! I had no idea CPython has been getting so much faster with each minor version!

[henryiii]

I didn't see any of the trappings for mypy (no config, no .pre-commit-config.yaml, no entry in tox.ini or a noxfile), but there are (not complete) static types! I think I can try it. Edit: there are some typing errors; you can't compile with typing errors.

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      lark/utils.py:222: error: Incompatible types in assignment (expression has type "None", variable has type Module)
      lark/utils.py:222: note: Error code "assignment" not covered by "type: ignore" comment
      lark/tree.py:89: error: Name "rich" is not defined
      lark/lark.py:270: error: Item "Grammar" of "Union[Grammar, str, IO[str]]" has no attribute "name"
      lark/lark.py:270: error: Item "str" of "Union[Grammar, str, IO[str]]" has no attribute "name"
      lark/lark.py:278: error: Item "Grammar" of "Union[Grammar, str, IO[str]]" has no attribute "read"
      lark/lark.py:278: error: Item "str" of "Union[Grammar, str, IO[str]]" has no attribute "read"
      lark/tree_templates.py:150: error: Argument 1 to "apply_vars" of "Template" has incompatible type "Dict[str, Union[Tree[str], str]]"; expected "Mapping[str, Tree[str]]"
      [end of output]

Flit automatically reads __version__.py from __init__.py, I can help you try it if you want. Though not right away, as I want to see if mypyc helps; in that case you'd want Hatch or Setuptools (for now).

I had no idea CPython has been getting so much faster with each minor version!

See https://github.com/faster-cpython/ideas/blob/main/FasterCPythonDark.pdf. That project is fully in action these days! The initial goal was 50% every year, but it's more like 25% for the first (full) year in Python 3.11. But that's still very nice! PS: version numbers in that presentation are off by at least 1; the project started a bit later than proposed.

[henryiii]

(I have tooling that helps with setup.py -> setup.cfg, and I know of tooling that helps with setup.cfg -> pyproject.toml PEP 621)

[henryiii]

FYI, setup.cfg Setuptools is supposed to be able to pull a version attribute without running the file (and since you have no deps, even if it fell back to exec'ing the file, it should still work).

[erezsh]

there are some typing errors; you can't compile with typing errors

They all seem solvable. Let me know if you want me to have a look.

setup.cfg Setuptools is supposed to be able to pull a version attribute without running the file

How? And does that apply to pyproject too?

[henryiii]

I can try this after #1149 if you'd like to see it. Basically it looks something like this:

version = attr: lark.__version__

setuptools will try to read it from the AST before falling back on importing lark to get it (which would also work since there's no deps). I think the way to do this in pyproject.toml config (assuming you mean using setuptools pyproject.toml config) is to use dynamic = ["version"] and then specify the above using [tool.setuptools.<something>], but using tool.setuptools is still considered experimental. However, flit does this automatically without configuration.

[erezsh]

However, flit does this automatically without configuration

I'm not familiar with flit. It's supposed to be a poetry replacement, right?

So it would automatically know to read the version off of lark.__version__?

[henryiii]

Technically, I meant flit_core, which is a simple PEP 517 backend. PEP 517 described the system which builders (like build and pip) use to talk to backends (PyPA projects setuptools, flit_core, and hatchling, and non-PyPA projects pdm-pep517, poetry_core, whey, trampolim, and others). The backend's job is to create the SDist or the wheel. Of all those, only Poetry is missing support for standard configuration language introduced in PEP 621, so transitioning between backends is really trivial unless it's Poetry¹.

If you produce a pure Python wheel, that doesn't use the backend at all to install, so it's not something that affects users much, mostly a developer choice based on what you find nice and easy to configure.

Reminder: I'd like to keep the current backend for now, until I play with mypyc, which is tied to setuptools (and hatchling has an adaptor), but likely won't be easy to tie to other backend. I plan on working on a PEP to address this, probably in the fall, but we are stuck with the status quo at the moment, which is extensions are tied to a backend. So if you see a backend you really want, please wait a bit. :)

---

Flit has the built-in feature that it pulls <package>.__version__ if version is left dynamic. It also can pull description from the module docstring, too. Other backends tend to have ways to get the version from Git (which is what I usually do, avoids any source changes on releases, fewer ways to make mistakes).

---

Almost every existing backend comes with a command-line tool as well. Flit (and Poetry) existed before there was standard tooling for it; I think that's primarily why there is a flit command. Though you don't need to use it (I don't). Nowadays, with pipx and build and twine, there's really no need to use a custom command to build SDists, wheels, and upload them. I also really like using the right tool for the job, so I mostly ignore what all-in-one CLI tools are doing.

Most of the CLI tools are trying to be "all-in-one", the question is how much is all on one? Flit does the least and is really easy to just use standard tooling instead. Poetry and PDM do environment management (and PDM follows standards, so I like it better - and you can use any build backend with PDM, not just it's own, which is awesome and I'd like to see that happen with other systems). The far end of the spectrum would be Hatch, which tries to also do multi-environment management (ala Tox/Nox) too².

Footnotes

1. Poetry has promised to support this in Poetry 2, but also promised that 2 is along way off. ↩

2. Hatch is currently missing lockfiles, though - Ofek really has been trying to follow standards, and the standardization attempt for lockfiles fell through. ↩

[erezsh]

Thanks for the overview! So far, just using setup.py seems to work for Lark. But I'll be happy to switch to something nicer when it's ready for use. I'll keep flit in mind.