Core Change: Multi-repository support

[Sharpie]

This is an evolution of pull request #6086 which presented an external command called brew-tap that makes it easier for Homebrew users to access formulae from alternate repositories. Refer to that pull request for the development history behind the patches presented here.

This pull request distils brew-tap into five patches that integrate into the Homebrew core without causing too much disruption or requiring massive re-writes. Multi-repository support is still under discussion and I expect these patches will evolve or be supplanted by an alternate implementation before we settle on a solution.

Summary of Core Chages

• A new file, Library/Homebrew/taproom.rb, has been added that contains the implementation for two new classes: Brewery and Taproom. Brewery is a class that represents a GitHub repository and can be used to retrieve information from the GitHub API. Taproom is a class that manages a folder where Git repositories may be cloned and aids in the discovery of brew files contained in those repositories. Together these classes form the core of multi-repository support.
• A new Homebrew command: Library/Homebrew/cmd/tap.rb. Compared to the brew-tap command presented in pull request New external command: brew-tap #6086, this version is focused on managing external repositories using three sub-commands: list, add and remove. The repositories available for cloning through brew-tap are defined to be Adam's Homebrew-alt repository and anything in its fork network. This constraint arises from the use of a global variable, HOMEBREW_TAPROOM, which is an object of class Taproom initiated from a Brewery object that represents Adam's repository. See comments in Library/Formula/Homebrew/taproom.rb for reasoning behind restricting te available repositories to the Homebrew-alt network---at least at first.
• The canonical_name method of the Formula class is extended such that it can discover brew files using the HOMEBREW_TAPROOM. This simple change makes formulae in tapped repositories available to Formula.factory and commands like install, uninstall, info and home.
• The update command is where the largest re-write occurred---but nothing much was changed. The RefreshBrew class gained some arguments to its initializer that allows it to update arbitrary repositories. The update code that is specific to the structure of the master repository was moved into a new subclass called RefreshCore.
• External dependency resolution was implemented through the addition of an :alt keyword to depends_on. Formula.expand_deps can now find dependencies flagged as :alt by querying HOMEBREW_TAPROOM.

Design Issues

• The Brewery class currently uses the httparty gem to communicate with the GitHub API and de-serialize JSON data to Ruby objects. The communication could be done through system calls to curl, but we still need to figure out how to deal with the JSON. Some possibilities:
  • Keep the gem dependency and add a gem install httparty to the Homebrew install script.
  • Bundle a copy of HTTParty or some other JSON parser into the main Homebrew repository.
  • Maintain our own JSON parsing code and use curl for communication.
• Easy acces to alternate repositories will allow users to get easily around Homebrew rules such as "no duplicates". I think this is a good thing as it allows consenting adults to build whatever they want the exact way they want it. However, we should build in a system that flags alternate installs in a way that brew doctor can easily detect so that bug reports stay manageable.

Remaining Work

• Tests related to multi-repository support need to be written. Currently there are four failing tests in the Homebrew testsuite due to changes in brew update and one failing test due to the use of a global variable, HOMEBREW_TAPROOM, in formula.rb.
• Some commands such as brew edit and things like bash completion need to be tweaked so that they pick up formulae in external repositories.
• Material from the old brew-tap manpage needs to be merged into the brew manpage.

Of course, design work is ongoing so comments and suggestions are greatly appreciated!

[Sharpie]

Updated. Added a patch that fixes broken tests. The tests made it clear that having formula.rb include global.rb in order to ensure HOMEBREW_TAPROOM is defined is a nightmare---so HOMEBREW_TAPROOM is now defined in taproom.rb to keep all kinds of constant craziness from creeping into anything that requires formula.

[samueljohn]

Sounds great! I'd really like to use this. Go for it !!
(why is nobody commenting? Are we two the only ones who think this would really be cool?)

[Sharpie]

@samueljohn

This one will take a little while. At the very least, I have to write tests and we have to figure out what to do with the gem dependency. What would help is if people would use the external command version of brew tap and leave feedback on things that they like/don't like about the approach I'm taking.

Installation instructions can be found in the External Commands wiki page:

https://github.com/mxcl/homebrew/wiki/External-Commands

[ashgti]

I noticed when I ran the command brew tap list my fork was not showing up in the list of repositories. I tracked down the problem and it appears to be due to pagination. The requests from github are only listing 30 forks per page and currently this only parses the first page of 30.

Adding in ?page=2 to the request gets the 2nd page. Also, there appears to be a per_page variable as well, so you can increase the number listed per page.

[Sharpie]

@ashgti

Thanks for letting me know about the pagination! @samueljohn had an issue with this and I assumed it was due to some synching error because he showed up on the list returned from the v2 API.

Both this pull request and #6086 have been updated with new code that deals with pagination.

[ashgti]

Using the command brew tap list results in:

∫ brew tap list
==> Repositories on tap:

Error: uninitialized constant Homebrew::HOMEBREW_TAPROOM
Please report this bug:
    https://github.com/mxcl/homebrew/wiki/checklist-before-filing-a-new-issue
/usr/local/Library/Homebrew/cmd/tap.rb:37:in `tap'
/usr/local/bin/brew:82:in `send'
/usr/local/bin/brew:82

I am not sure why I am getting this and no one else is, but adding

require 'taproom'

to the top of cmd/tap.rb fixed it.

Btw:

∫ brew --config
HOMEBREW_VERSION: 0.8
HEAD: d90b4ec81fcf44e28f761a0eb4837db74b51aaee
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CELLAR: /usr/local/Cellar
HOMEBREW_REPOSITORY: /usr/local
HOMEBREW_LIBRARY_PATH: /usr/local/Library/Homebrew
Hardware: dual-core 64-bit penryn
OS X: 10.7.1
Kernel Architecture: x86_64
Ruby: 1.8.7-249
/usr/bin/ruby => /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
Xcode: 4.1
GCC-4.0: N/A 
GCC-4.2: build 5666 
LLVM: build 2335 
MacPorts or Fink? false
X11 installed? true

In case that matters.

[Sharpie]

Thanks for the catch.

Note that the brew-tap-core branch will be more unstable than the brew-tap external command presented in pull request #6086. This is because the code in brew-tap-core is integrated with the Homebrew internals an is more sensitive to changes. I will also rebase this branch frequently in order to keep the number of commits small.

The external command is not as sensitive to core changes (outside of changes to brew update) and I will never rebase that branch so it should be easy to synch with.

[Sharpie]

Note concerning HTTParty:

Apparently Leopard ships with version 1.0.1 of RubyGems which is ancient. It also ships with Ruby 1.8.6 which means it cannot use a version of RubyGems that is newer than 1.3.x. This means it will be a PITA for Leopard users to install HTTParty for the system ruby in order to use multi-repo suport.

So, we either need to:

• Bundle HTTParty with homebrew.
• Come up with our own api-fetching, JSON-parsing code for the Brewery class (I hate this idea).

[adamv]

I vote for "vendorize library".

[Sharpie]

Allright, the code now uses a vendorized copy of HTTParty. If this works out we are pretty close to a candidate for merging.

[mxcl]

So I always thought we'd (or others) would setup dedicated repos for things, like homebrew-games, homebrew-dupes. Rather than have this single alt repository. Then users would add repos they wanted. Much like how Debian works.

[jacknagel]

If I'm not mistaken, this implementation supports any number of different repos as long as they are within the GitHub fork network of Homebrew-alt.

[mxcl]

Yes, sorry I do see that.

I envisaged that people would step forward to maintain formula repos of formula that they care about. It's messy that they would fork alt and then add their stuff. They can delete everything, then it is clearer what they maintain, but it's messy. A clean start with new formula gives a clear git history, and the issue tracker will be specific. And forks of those repos will be easily identified with that 'category' of formula.

I'm not really arguing any particular way here. Just bringing up points of discussion one by one. I have a list.

[jacknagel]

All good points.

[Sharpie]

@mxcl

They can delete everything, then it is clearer what they maintain, but it's messy.

git checkout --orphan is a pretty clean way to start a new line of history on a fork.

People can also git clone repositories into <brew_prefix>/Library/Taproom and everything will pretty much work. The things that are highly tied to the Homebrew-alt fork network are:

• Listing alternate repositories
• Cloning an alternate repository via brew tap

I want to relax these restrictions, but they are here for now to place boundaries around a problem that is a manageable size to solve.

[adamv]

@mxcl pushed his brew-tap, so closing this