Use a fork of pip-compile to provide test-requirements functionality

[lfdebrux]

Ticket: https://trello.com/c/iLQca47V/251-replace-custom-script-to-freeze-requirements-with-pip-tools

Add a script test whether pip-compile needs to be run, take 2.

This PR is a follow on from #218, where I looked at a writing a script to test if pip-compile needed to be run. One comment that came up more than once is that would be nicer if the script were Python rather than shell; to me it seemed like the logical continuation of that was that it would be nicest if it were part of pip-tools already... So I forked it ;)

I would eventually like to see this feature merged into mainline, but first we should look at it as a team and talk about a) whether we like it b) whether it fits our needs.

The way that it works is by adding a checksum for the input requirement files to the output requirement files; this is simpler and more straightforward than the previous shell script (so if pip-tools reject this out of hand we could always go back to a shell script).

I've created a draft PR jazzband/pip-tools#1070 for people to review my changes to their code and comment on; also feel free to leave comments here.

[lfdebrux]

Example of Travis failure:

https://travis-ci.com/alphagov/digitalmarketplace-user-frontend/jobs/288542061

[katstevens]

I tried running the following after checking out this branch:

$ make requirements-dev    # This worked fine
$ make requirements
...
/path/to/my/code/digitalmarketplace-user-frontend/venv/bin/pip-check requirements.txt requirements-dev.txt
/bin/bash: /path/to/my/code/digitalmarketplace-user-frontend/venv/bin/pip-check: No such file or directory
make: *** [test-requirements] Error 127

Is there another step I'd need to do? If so can we document it?

[lfdebrux]

I tried running the following after checking out this branch:

$ make requirements-dev    # This worked fine
$ make requirements
...
/path/to/my/code/digitalmarketplace-user-frontend/venv/bin/pip-check requirements.txt requirements-dev.txt
/bin/bash: /path/to/my/code/digitalmarketplace-user-frontend/venv/bin/pip-check: No such file or directory
make: *** [test-requirements] Error 127

Is there another step I'd need to do? If so can we document it?

That's weird, it should work as you've written it...

I can't reproduce on my machine, do you get the same thing if you remove the venv and run the commands again?

[katstevens]

I can't reproduce on my machine, do you get the same thing if you remove the venv and run the commands again?

Same issue. However if I just run make requirements-dev then source the venv, it works if I run pip-check manually 🤔 :

(venv)$ pip-check requirements-dev.txt requirements.txt
requirements-dev.txt: lock(s) are out-of-date    # I manually changed something...
1 errors found. Run pip-compile to fix

Something in the Makefile, maybe?

[lfdebrux]

I can't reproduce on my machine, do you get the same thing if you remove the venv and run the commands again?

Same issue. However if I just run make requirements-dev then source the venv, it works if I run pip-check manually 🤔 :

(venv)$ pip-check requirements-dev.txt requirements.txt
requirements-dev.txt: lock(s) are out-of-date    # I manually changed something...
1 errors found. Run pip-compile to fix

Something in the Makefile, maybe?

That looks like it's working now :D (assuming that you did make a change to requirements-dev.in).

[katstevens]

Let's fix the pip-tools fork to a commit before we merge this 👍

[lfdebrux]

Let's fix the pip-tools fork to a commit before we merge this 👍

Done. I pinned it to a tag because pip-compile didn't like using a commit ref; I promise not to move the tag! 😆