From bf12e427aea2eb5d7f11f0480c57508ae4f7326b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?S=C3=A9bastien=20Eustace?= Date: Fri, 20 Mar 2020 14:06:30 +0100 Subject: [PATCH] Merge master into develop (#2206) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * export: fix exporting extras sub-dependencies (#1294) * Support POETRY_HOME for install (#794) Allow the `POETRY_HOME` environment variable to be passed during installation to change the default installation directory of `~/.poetry`: ``` POETRY_HOME=/etc/poetry python get-poetry.py ``` * * check if relative filename is in excluded file list (#1459) * * check if relative filename is in excluded file list * removed find_excluded_files() method from wheel.py * added test for excluding files in wheels * creating an own test data folder, for testing excluding files by pyproject.toml * use as_posix() to respect windows file path delimiters * Exclude nested items (#784) (#1464) * This PR impliments the feature request #784. When a folder is explicit defined in `pyproject.toml` as excluded, all nested data, including subfolder, are excluded. It is no longer neccessary to use the glob `folder/**/*` * use `Path` instead of `os.path.join` to create string for globbing * try to fix linting error * create glob pattern string by concatenating and not using Path * using `os.path.isdir()`` for checking of explicit excluded name is a folder, because pathlib's `is_dir()` raises in exception under windows of name contains globing characters * Remove nested data when wildcards where used. Steps to do this are: 1. expand any wildcard used 2. if expanded path is a folder append **/* and expand again * fix linting * only glob a second time if path is dir * implement @sdispater 's suggestion for better readability * fix glob for windows? * On Windows, testing if a path with a glob is a directory will raise an OSError * pathlibs glob function doesn't return the correct case (https://bugs.python.org/issue26655). So switching back to glob.glob() * removing obsolete imports * Update dependencies * Deprecate allows-prereleases in favor of allow-prereleases for consistency * Fix tests for Python 2.7 * Fix linting * Fix linting * Fix linting * Fix typing import * Correct a couple typos in get-poetry.py (#573) * Docs: `self:update` changed to `self update` (#1588) * Fix GitHub actions cache issues on develop (#1918) * Fix Github actions cache issues * Fix Github Actions cache issues (#1928) * Add --source option to "poetry add" (#1912) * Add --source option to 'poetry add' * Add tests for 'poetry add --source' * Merge master into develop (#2070) * Fix Github actions cache issues (#1908) * Fix case of `-f` flag * Make it clearer what options to pass to `--format` * fix (masonry.api): `get_requires_for_build_wheel` must return additional list of requirements for building a package, not listed in `pyproject.toml` and not dependencies for the package itself (#1875) fix (tests): adopted tests * Lazy Keyring intialization for PasswordManager (#1892) * Fix Github Actions cache issues (#1928) * Avoid nested quantifiers with overlapping character space on git url parsing (#1902 (#1913) * fix (git): match for `\w` instead of `.` for getting user * change (vcs.git): hold pattern of the regex parts in a dictionary to be consistent over all regexs * new (vcs.git): test for `parse_url` and some fixes for the regex pattern * new (vcs.git): test for `parse_url` with string that should fail * fix (test.vcs.git): make flake8 happy * fix: correct parsing of wheel version with regex. (#1932) The previous regexp was only taking the first integer of the version number, this presented problems when the major version number reached double digits. Poetry would determine that the version of the dependency is '1', rather than, ie: '14'. This caused failures to solve versions. * Fix errors when using the --help option (#1910) * Fix how repository credentials are retrieved from env vars (#1909) # Conflicts: # poetry/utils/password_manager.py * Fix downloading packages from Simplepypi (#1851) * fix downloading packages from simplepypi * unused code removed * remove unused imports * Upgrade dependencies for the 1.0.3 release (#1965) * Bump version to 1.0.3 (#1966) * Fix non-compliant Git URL matching RFC 3986 § 2.3 permits more characters in a URL than were matched. This corrects that, though there may be other deficiencies. This was a regression from v1.0.2, where at least “.” was matched without error. * Update README.md "Updating Poetry" Currently the note in "Updating Poetry" is different from the one below in "Enable tab completion for Bash, Fish, or Zsh". This MR is to make them more consistent. * init: change dev dependency prompt * Fix CI issues (#2069) Co-authored-by: brandonaut Co-authored-by: finswimmer Co-authored-by: Yannick PÉROUX Co-authored-by: Edward George Co-authored-by: Jan Škoda Co-authored-by: Andrew Marshall Co-authored-by: Andrew Selzer Co-authored-by: Andrii Maletskyi * pre-commit: replace isort mirror with isort upstream (#2118) The isort pre-commit mirror has been deprecated. This change updates configuration to use the upstream package repository instead of the mirror. * Add cache list command (#1187) * Add poetry.locations.REPOSITORY_CACHE_DIR The repository cache directory is used in multiple places in the codebase. This change ensures that the value is reused. * Add cache list command This introduces a new cache sub-command that lists all available caches. Relates-to: #1162 Co-authored-by: Tom Milligan Co-authored-by: David Cramer Co-authored-by: finswimmer Co-authored-by: Kyle Altendorf Co-authored-by: Justin Mayer Co-authored-by: Yannick PÉROUX Co-authored-by: brandonaut Co-authored-by: Edward George Co-authored-by: Jan Škoda Co-authored-by: Andrew Marshall Co-authored-by: Andrew Selzer Co-authored-by: Andrii Maletskyi Co-authored-by: Arun Babu Neelicattu --- .pre-commit-config.yaml | 4 +- CHANGELOG.md | 1 - docs/docs/cli.md | 12 +++++ poetry/console/commands/add.py | 13 ++++- poetry/console/commands/cache/cache.py | 4 +- poetry/console/commands/cache/clear.py | 8 ++- poetry/console/commands/cache/list.py | 21 ++++++++ poetry/console/commands/init.py | 11 ++-- poetry/locations.py | 3 ++ poetry/repositories/legacy_repository.py | 4 +- poetry/repositories/pypi_repository.py | 4 +- poetry/version/version_selector.py | 3 +- tests/console/commands/test_add.py | 68 ++++++++++++++++++++++++ tests/console/commands/test_cache.py | 60 +++++++++++++++++++++ 14 files changed, 197 insertions(+), 19 deletions(-) create mode 100644 poetry/console/commands/cache/list.py create mode 100644 tests/console/commands/test_cache.py diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index b296497d29f..426692139b4 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -9,8 +9,8 @@ repos: hooks: - id: flake8 - - repo: https://github.com/pre-commit/mirrors-isort - rev: v4.3.21 + - repo: https://github.com/timothycrosley/isort + rev: 4.3.21 hooks: - id: isort additional_dependencies: [toml] diff --git a/CHANGELOG.md b/CHANGELOG.md index d196bd38d52..2a49380e6be 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,7 +27,6 @@ - Fixed an error when parsing some git URLs ([#2018](https://github.com/python-poetry/poetry/pull/2018)). - ## [1.0.3] - 2020-01-31 ### Fixed diff --git a/docs/docs/cli.md b/docs/docs/cli.md index 89d19a8cc18..63d63fbc990 100644 --- a/docs/docs/cli.md +++ b/docs/docs/cli.md @@ -452,3 +452,15 @@ The `env` command regroups sub commands to interact with the virtualenvs associated with a specific project. See [Managing environments](./managing-environments.md) for more information about these commands. + +## cache + +The `cache` command regroups sub commands to interact with Poetry's cache. + +### cache list + +The `cache list` command lists Poetry's available caches. + +```bash +poetry cache list +``` diff --git a/poetry/console/commands/add.py b/poetry/console/commands/add.py index 275fc44f5f1..bb210265703 100644 --- a/poetry/console/commands/add.py +++ b/poetry/console/commands/add.py @@ -33,6 +33,12 @@ class AddCommand(EnvCommand, InitCommand): "Platforms for which the dependency must be installed.", flag=False, ), + option( + "source", + None, + "Name of the source to use to install the package.", + flag=False, + ), option("allow-prereleases", None, "Accept prereleases."), option( "dry-run", @@ -86,7 +92,9 @@ def handle(self): raise ValueError("Package {} is already present".format(name)) requirements = self._determine_requirements( - packages, allow_prereleases=self.option("allow-prereleases") + packages, + allow_prereleases=self.option("allow-prereleases"), + source=self.option("source"), ) for _constraint in requirements: @@ -123,6 +131,9 @@ def handle(self): if self.option("platform"): constraint["platform"] = self.option("platform") + if self.option("source"): + constraint["source"] = self.option("source") + if len(constraint) == 1 and "version" in constraint: constraint = constraint["version"] diff --git a/poetry/console/commands/cache/cache.py b/poetry/console/commands/cache/cache.py index 469eb56be05..695e27e0af7 100644 --- a/poetry/console/commands/cache/cache.py +++ b/poetry/console/commands/cache/cache.py @@ -1,3 +1,5 @@ +from poetry.console.commands.cache.list import CacheListCommand + from ..command import Command from .clear import CacheClearCommand @@ -7,7 +9,7 @@ class CacheCommand(Command): name = "cache" description = "Interact with Poetry's cache" - commands = [CacheClearCommand()] + commands = [CacheClearCommand(), CacheListCommand()] def handle(self): return self.call("help", self._config.name) diff --git a/poetry/console/commands/cache/clear.py b/poetry/console/commands/cache/clear.py index 347f534afc0..c46dab01281 100644 --- a/poetry/console/commands/cache/clear.py +++ b/poetry/console/commands/cache/clear.py @@ -16,19 +16,17 @@ class CacheClearCommand(Command): def handle(self): from cachy import CacheManager - from poetry.locations import CACHE_DIR - from poetry.utils._compat import Path + from poetry.locations import REPOSITORY_CACHE_DIR cache = self.argument("cache") parts = cache.split(":") root = parts[0] - base_cache = Path(CACHE_DIR) / "cache" / "repositories" - cache_dir = base_cache / root + cache_dir = REPOSITORY_CACHE_DIR / root try: - cache_dir.relative_to(base_cache) + cache_dir.relative_to(REPOSITORY_CACHE_DIR) except ValueError: raise ValueError("{} is not a valid repository cache".format(root)) diff --git a/poetry/console/commands/cache/list.py b/poetry/console/commands/cache/list.py new file mode 100644 index 00000000000..6a030fa2eba --- /dev/null +++ b/poetry/console/commands/cache/list.py @@ -0,0 +1,21 @@ +import os + +from ..command import Command + + +class CacheListCommand(Command): + + name = "list" + description = "List Poetry's caches." + + def handle(self): + from poetry.locations import REPOSITORY_CACHE_DIR + + if os.path.exists(str(REPOSITORY_CACHE_DIR)): + caches = list(sorted(REPOSITORY_CACHE_DIR.iterdir())) + if caches: + for cache in caches: + self.line("{}".format(cache.name)) + return 0 + + self.line("No caches found") diff --git a/poetry/console/commands/init.py b/poetry/console/commands/init.py index 825f8c68e2a..40ed33a17f1 100644 --- a/poetry/console/commands/init.py +++ b/poetry/console/commands/init.py @@ -203,7 +203,7 @@ def handle(self): f.write(content) def _determine_requirements( - self, requires, allow_prereleases=False + self, requires, allow_prereleases=False, source=None ): # type: (List[str], bool) -> List[Dict[str, str]] if not requires: requires = [] @@ -299,7 +299,9 @@ def _determine_requirements( elif "version" not in requirement: # determine the best version automatically name, version = self._find_best_version_for_package( - requirement["name"], allow_prereleases=allow_prereleases + requirement["name"], + allow_prereleases=allow_prereleases, + source=source, ) requirement["version"] = version requirement["name"] = name @@ -314,6 +316,7 @@ def _determine_requirements( requirement["name"], requirement["version"], allow_prereleases=allow_prereleases, + source=source, ) requirement["name"] = name @@ -323,13 +326,13 @@ def _determine_requirements( return result def _find_best_version_for_package( - self, name, required_version=None, allow_prereleases=False + self, name, required_version=None, allow_prereleases=False, source=None ): # type: (...) -> Tuple[str, str] from poetry.version.version_selector import VersionSelector selector = VersionSelector(self._get_pool()) package = selector.find_best_candidate( - name, required_version, allow_prereleases=allow_prereleases + name, required_version, allow_prereleases=allow_prereleases, source=source ) if not package: diff --git a/poetry/locations.py b/poetry/locations.py index 17fb4d42a99..003950d500d 100644 --- a/poetry/locations.py +++ b/poetry/locations.py @@ -1,6 +1,9 @@ +from .utils._compat import Path from .utils.appdirs import user_cache_dir from .utils.appdirs import user_config_dir CACHE_DIR = user_cache_dir("pypoetry") CONFIG_DIR = user_config_dir("pypoetry") + +REPOSITORY_CACHE_DIR = Path(CACHE_DIR) / "cache" / "repositories" diff --git a/poetry/repositories/legacy_repository.py b/poetry/repositories/legacy_repository.py index 4bb0496eb29..c2eac509ae8 100644 --- a/poetry/repositories/legacy_repository.py +++ b/poetry/repositories/legacy_repository.py @@ -15,7 +15,7 @@ import poetry.packages -from poetry.locations import CACHE_DIR +from poetry.locations import REPOSITORY_CACHE_DIR from poetry.packages import Package from poetry.packages import dependency_from_pep_508 from poetry.packages.utils.link import Link @@ -174,7 +174,7 @@ def __init__( self._client_cert = client_cert self._cert = cert self._inspector = Inspector() - self._cache_dir = Path(CACHE_DIR) / "cache" / "repositories" / name + self._cache_dir = REPOSITORY_CACHE_DIR / name self._cache = CacheManager( { "default": "releases", diff --git a/poetry/repositories/pypi_repository.py b/poetry/repositories/pypi_repository.py index 767a7353b3c..0f1ac5208ed 100644 --- a/poetry/repositories/pypi_repository.py +++ b/poetry/repositories/pypi_repository.py @@ -15,7 +15,7 @@ from requests import session from requests.exceptions import TooManyRedirects -from poetry.locations import CACHE_DIR +from poetry.locations import REPOSITORY_CACHE_DIR from poetry.packages import Package from poetry.packages import dependency_from_pep_508 from poetry.packages.utils.link import Link @@ -57,7 +57,7 @@ def __init__(self, url="https://pypi.org/", disable_cache=False, fallback=True): self._disable_cache = disable_cache self._fallback = fallback - release_cache_dir = Path(CACHE_DIR) / "cache" / "repositories" / "pypi" + release_cache_dir = REPOSITORY_CACHE_DIR / "pypi" self._cache = CacheManager( { "default": "releases", diff --git a/poetry/version/version_selector.py b/poetry/version/version_selector.py index 8c71daec87b..2077e322699 100644 --- a/poetry/version/version_selector.py +++ b/poetry/version/version_selector.py @@ -15,6 +15,7 @@ def find_best_candidate( package_name, # type: str target_package_version=None, # type: Union[str, None] allow_prereleases=False, # type: bool + source=None, # type: str ): # type: (...) -> Union[Package, bool] """ Given a package name and optional version, @@ -26,7 +27,7 @@ def find_best_candidate( constraint = parse_constraint("*") candidates = self._pool.find_packages( - package_name, constraint, allow_prereleases=True + package_name, constraint, allow_prereleases=True, repository=source ) only_prereleases = all([c.version.is_prerelease() for c in candidates]) diff --git a/tests/console/commands/test_add.py b/tests/console/commands/test_add.py index 6a64ba7e4eb..8e7f4a870c8 100644 --- a/tests/console/commands/test_add.py +++ b/tests/console/commands/test_add.py @@ -4,6 +4,8 @@ from cleo.testers import CommandTester +from poetry.repositories.legacy_repository import LegacyRepository +from poetry.semver import Version from poetry.utils._compat import Path from tests.helpers import get_dependency from tests.helpers import get_package @@ -634,6 +636,72 @@ def test_add_constraint_with_platform(app, repo, installer): } +def test_add_constraint_with_source(app, poetry, installer): + repo = LegacyRepository(name="my-index", url="https://my-index.fake") + repo.add_package(get_package("cachy", "0.2.0")) + repo._cache.store("matches").put("cachy:0.2.0", [Version.parse("0.2.0")], 5) + + poetry.pool.add_repository(repo) + + command = app.find("add") + tester = CommandTester(command) + + tester.execute("cachy=0.2.0 --source my-index") + + expected = """\ + +Updating dependencies +Resolving dependencies... + +Writing lock file + + +Package operations: 1 install, 0 updates, 0 removals + + - Installing cachy (0.2.0) +""" + + assert expected == tester.io.fetch_output() + + assert len(installer.installs) == 1 + + content = app.poetry.file.read()["tool"]["poetry"] + + assert "cachy" in content["dependencies"] + assert content["dependencies"]["cachy"] == { + "version": "0.2.0", + "source": "my-index", + } + + +def test_add_constraint_with_source_that_does_not_exist(app): + command = app.find("add") + tester = CommandTester(command) + + with pytest.raises(ValueError) as e: + tester.execute("foo --source i-dont-exist") + + assert 'Repository "i-dont-exist" does not exist.' == str(e.value) + + +def test_add_constraint_not_found_with_source(app, poetry, mocker): + repo = LegacyRepository(name="my-index", url="https://my-index.fake") + mocker.patch.object(repo, "find_packages", return_value=[]) + + poetry.pool.add_repository(repo) + + pypi = poetry.pool.repositories[0] + pypi.add_package(get_package("cachy", "0.2.0")) + + command = app.find("add") + tester = CommandTester(command) + + with pytest.raises(ValueError) as e: + tester.execute("cachy --source my-index") + + assert "Could not find a matching version of package cachy" == str(e.value) + + def test_add_to_section_that_does_no_exist_yet(app, repo, installer): command = app.find("add") tester = CommandTester(command) diff --git a/tests/console/commands/test_cache.py b/tests/console/commands/test_cache.py new file mode 100644 index 00000000000..8a22eee7f1b --- /dev/null +++ b/tests/console/commands/test_cache.py @@ -0,0 +1,60 @@ +import uuid + +import pytest + +from cleo.testers import CommandTester + + +@pytest.fixture +def repository_cache_dir(monkeypatch, tmpdir): + import poetry.locations + from poetry.utils._compat import Path + + path = Path(str(tmpdir)) + monkeypatch.setattr(poetry.locations, "REPOSITORY_CACHE_DIR", path) + return path + + +@pytest.fixture +def repository_one(): + return "01_{}".format(uuid.uuid4()) + + +@pytest.fixture +def repository_two(): + return "02_{}".format(uuid.uuid4()) + + +@pytest.fixture +def mock_caches(repository_cache_dir, repository_one, repository_two): + (repository_cache_dir / repository_one).mkdir() + (repository_cache_dir / repository_two).mkdir() + + +def test_cache_list(app, mock_caches, repository_one, repository_two): + command = app.find("cache list") + tester = CommandTester(command) + + tester.execute() + + expected = """\ +{} +{} +""".format( + repository_one, repository_two + ) + + assert expected == tester.io.fetch_output() + + +def test_cache_list_empty(app, repository_cache_dir): + command = app.find("cache list") + tester = CommandTester(command) + + tester.execute() + + expected = """\ +No caches found +""" + + assert expected == tester.io.fetch_output()