remove alterations of sys.path

[ltalirz]

fix #3293

python allows to modify the search path for modules programmatically via the sys.path variable (sys.path is initialized from the PYTHONPATH environment variable).
Doing so, however, can easily have unintended consequences (one of them surfaced in #3293) - in particular, when new paths are inserted at the beginning of sys.path.

AiiDA was manipulating sys.path in a number of places:

1. aiida-core/aiida/cmdline/commands/cmd_run.py

   Line 113 in a500d2b

   sys.path.insert(0, os.path.abspath(os.curdir))

2. aiida-core/aiida/sphinxext/tests/workchain_source/conf.py

   Line 32 in d750d51

   sys.path.insert(0, os.path.abspath('.'))

3. aiida-core/utils/validate_consistency.py

   Line 190 in cc94a1e

   sys.path.insert(0, ROOT_DIR)

4. aiida-core/aiida/common/files.py

   Line 194 in cc94a1e

   path.insert(0, os.curdir)

5. aiida-core/aiida/backends/djsite/settings.py

   Line 31 in 91b205b

   sys.path = [BASE_DIR] + sys.path

This PR removes/remedies those manipulations (except for point 4, which is part of shutil.which and is relevant only for sys.platform == 'win32').

[ltalirz]

Not sure what exactly this comment means / why this should be needed.
Anyhow, it seems the code works also without this

[CasperWA]

Excellent work @ltalirz, it's always nice to remove more files from the linter's exclusion list :)

I have only a few questions and comments, the final review I will leave for @giovannipizzi and/or @sphuber, I think. Especially @giovannipizzi, since he may be the original author of many of the lines?

[CasperWA]

This seems weird, but I guess this is the main reason why this PR is relevant, I presume? To make sure sys.path is properly cleaned up from AiiDA side?

[ltalirz]

What is weird about this? It's just adding a path to import a module (which we need just for the aiida version) and then cleaning up the path again so that things aren't messed up further down the line.

[CasperWA]

It just seems intuitively inelegant that's all - this of course does not mean it isn't the best way to do it :)
I was wondering however, if one could not use importlib instead or similar, i.e., if it is absolutely necessary to add a path to sys.path? But I may be missing when this file is used/its purpose?

[ltalirz]

Feel free to suggest a better approach

[giovannipizzi]

You can do something like the following:

import pkgutil

loaders = [
    module_loader for (module_loader, name, ispkg) 
    in pkgutil.iter_modules(path=['ROOT_DIR'])
    if name == 'aiida' and ispkg]

if not loaders:
    raise ImportError("xxx") # or ignore?
loader = loaders[0]
version = loader.find_module('aiida').load_module('aiida').__version__
print(version)

[ltalirz]

@giovannipizzi Casper is done with his review - could you please give a quick look as well?

[giovannipizzi]

Just one change (on the lines of @CasperWA's suggestion).
If it keeps working, I think it's good to do.

Also, did you check if this fixed the issue with the tests not working for modules names sqlachemy, django, ...?

BTW, do you know why coverage dropped to ~30%?

[giovannipizzi]

You can do something like the following:

import pkgutil

loaders = [
    module_loader for (module_loader, name, ispkg) 
    in pkgutil.iter_modules(path=['ROOT_DIR'])
    if name == 'aiida' and ispkg]

if not loaders:
    raise ImportError("xxx") # or ignore?
loader = loaders[0]
version = loader.find_module('aiida').load_module('aiida').__version__
print(version)

[ltalirz]

@giovannipizzi Thanks - I've implemented your suggestion and checked that it works.
Not quite sure whether this is really more elegant, but it does have the advantage of leaving the sys.path untouched (I checked).

Also, did you check if this fixed the issue with the tests not working for modules names sqlachemy, django, ...?

I don't remember exactly what the problem was but issues related to name clashes with modules/folders inside aiida/backends might be resolved.

BTW, do you know why coverage dropped to ~30%?

It happened on the merge done via travis, so no idea...