Add support for python_implementation

[gsnedders]

We currently support platform.python_implementation and platform_python_implementation. setuptools' previous fork of packaging did support python_implementation until it was replaced by platform_python_implementation in pypa/setuptools@3bd5118#diff-0c04a5cfcbd2a4bdd1121db108b79d3d.

python_implementation existed in setuptools until that commit and the death of MarkerEvaluation, see https://github.com/pypa/setuptools/blob/f8b1293c408bbb652bec3f2ae6e5b4f33f3ca55e/pkg_resources/__init__.py#L1389 for the full (?) list of what was supported there. It also briefly existed in PEP 508 until February (https://hg.python.org/peps/rev/655a101719a5 replaced it with platform_python_implementation).

The one thing we're missing from what setuptools historically supported is python_implementation; we have everything else. This adds more duplication, but its removal broke some packages (e.g., some versions of html5lib).

[gsnedders]

https://mail.python.org/pipermail/distutils-sig/2015-September/026873.html seems to be about this; platform_python_implementation didn't exist in setuptools until https://github.com/jaraco/setuptools/pull/28 landed in 18.5, whereas python_implementation has existed since 0.7 (pypa/setuptools@e7d341a).

[gsnedders]

So for anyone who stumbles across here, the summary as to what versions of setuptools support what markers:

• python_implementation from 0.7 (2 Jun 2013) till 20.1.1 (12 Feb 2016), and after Fix #72: Add support for python_implementation marker #73 was merged from 20.10.0 (25 Apr 2016).
• platform.python_implementation from 0.6.29 (21 Oct 2012) till 20.1.1 (12 Feb 2016), and after Re-add support for deprecated PEP-345 markers #71 was merged from 20.6.6 (30 Mar 2016)
• platform_python_implementation from 18.5 (2 Nov 2015)

[jayvdb]

@gsnedders, thanks for your breakdown of the version ranges supported by setuptools versions.

platform.python_implementation from 0.6.29 (21 Oct 2012) till 20.1.1 (12 Feb 2016), and after #71 was merged from 20.6.6 (30 Mar 2016)

when I try with platform.python_implementation for tox-dev/tox-travis#23, it fails on setuptools ~0.7-19.
https://travis-ci.org/jayvdb/tox-travis/builds/144867443 (v13+ , on py35)
https://travis-ci.org/jayvdb/tox-travis/builds/144870320 (v0.6+, on py35)
https://travis-ci.org/jayvdb/tox-travis/builds/144870715 (v0.6+, on py27)

Am I missing something?

[gsnedders]

@jayvdb Oh, I concluded the above was wrong, but obviously forgot to post about it here. https://gsnedders.github.io/python-marker-test/results.html is a summary of what supports what.

[jayvdb]

Interesting, so the rub is then how do we force setuptools into "without parser" mode... ;-)

[gsnedders]

See https://github.com/gsnedders/python-marker-test/blob/master/test_pkgresources.py which is how I did so. Note more recent releases don't have the variants, so you need to handle that too.

[jayvdb]

Indeed; that gets us into the right mode, some of the time during python setup.py install, but not all of the time.(see https://travis-ci.org/jayvdb/tox-travis/jobs/144879946)
I needed to add an extra bit of patching, and then it "works" with setuptools 2.2+ (except ~18) on Python 2.7
jayvdb/tox-travis@0496b5d...50cede9
https://travis-ci.org/jayvdb/tox-travis/builds/144880986
even with wheels
https://travis-ci.org/jayvdb/tox-travis/builds/144882204

but then we run into platform problems on at least 3.2 and the new PyPy3 (all quite solvable, by the look of things)
https://travis-ci.org/jayvdb/tox-travis/builds/144882999
Now running a bigger python implementation/version matrix...
https://travis-ci.org/jayvdb/tox-travis/builds/144883858

[jayvdb]

OK, a bit of mucking around, and I have two solutions for a setup.py that works with all setuptools versions 2.2+ except for 18 (and it should be trivial to support 18 also), using platform_python_implementation.

Option 1: Python 3 compatible _markerlib_evaluate

(wrap env.keys() in list(..))

import sys

try:
    import _markerlib
except ImportError:
    pass
else:
    def _markerlib_evaluate(cls, text):
        env = _markerlib.default_environment()
        for key in list(env.keys()):
            new_key = key.replace('.', '_')
            env[new_key] = env.pop(key)
        try:
            result = _markerlib.interpret(text, env)
        except NameError:
            e = sys.exc_info()[1]
            raise SyntaxError(e.args[0])
        return result

    try:
        import pkg_resources
        del pkg_resources.parser
        pkg_resources.MarkerEvaluation.evaluate_marker = classmethod(_markerlib_evaluate)
        pkg_resources.evaluate_marker = pkg_resources.MarkerEvaluation.evaluate_marker
    except (ImportError, AttributeError):
        pass

from setuptools import setup

...

Option 2: Create the markers dict in setup.py and prevent _markerlib from modifying it

(as it would try to modify in a non-py3 compatible way)

try:
    import _markerlib.markers
except ImportError:
    pass
else:
    class ReadOnlyDict(dict):

        def __setitem__(self, key, value):
            pass

        def pop(self, i=-1):
            return self[i]


    # _markerlib.default_environment() obtains its data from _VARS
    env = _markerlib.markers._VARS
    for key in list(env.keys()):
        new_key = key.replace('.', '_')
        if new_key != key:
            env[new_key] = env[key]

    _markerlib.markers._VARS = ReadOnlyDict(env)

    def default_environment():
        return _markerlib.markers._VARS


    _markerlib.default_environment = default_environment

try:
    import pkg_resources
    del pkg_resources.parser
    pkg_resources.evaluate_marker = pkg_resources.MarkerEvaluation._markerlib_evaluate
    pkg_resources.MarkerEvaluation.evaluate_marker = pkg_resources.MarkerEvaluation._markerlib_evaluate
except (ImportError, AttributeError):
    pass

from setuptools import setup

...

No doubt both of these can be improved upon. I've still yet to test other scenarios. e.g. working only with wheels, different dist commands, MS Windows, etc. I'd love suggestions on how either approach might be broken.
I'll polish it and put them into a test repo for people to play with, and hopefully try to break. (Or @gsnedders , can I dump it into your python-marker-test repo, and then maybe the repo is renamed -demo.)

[jayvdb]

https://travis-ci.org/jayvdb/tox-travis/builds/145049141 shows most setuptools versions working correctly, with .travis.yml matrix: include: and allow_failures used to show specific areas where it does or doesnt work.
(The pypy3-5.2-alpha errors should be fixed in https://travis-ci.org/jayvdb/tox-travis/builds/145144692 , but that will take a while to complete.)

[jayvdb]

To add some closure, I built a very hacky patch for html5lib that allows all environment markers on nearly every setuptools version. See html5lib/html5lib-python#285 and another version of it at tox-dev/tox-travis#24 . Those are both MIT, so use it accordingly. I am happy to send a similar PR for any project having similar requirements of working with older setuptools, if that helps your licensing situation.