New capability to use git submodule config to checkout externals

[gold2718]

New capability to use git submodule info to checkout externals

By default, an external with no specified sub-externals configuration file will
have its submodules checked out according to the information in that repository's
.gitmodules file. This process is recursive.

To prevent submodules from being checked out, the external description for that
repository should be listed as "externals = none".

To use submodule URL, local path, and reference hash information for a
sub-external, replace the 'local_path', 'repo_url', and 'branch' 'hash' or 'tag'
keywords with "from_submodule = True".

User interface changes?: Yes
externals keyword can be "none" to prevent loading submodules.
from_submodule keyword can be used to replace external configuration information
with information from a repository's submodule configuration.

Fixes: #96

Testing:
test removed: none
unit tests: all pass
system tests: all pass, added TestSubrepoCheckout to test new functionality
manual testing: Tests with ESCOMP/MOM_interface (see below)

1. Just run checkout_externals: no change
2. Remove "externals = ../Externals_MOM.cfg" from Externals.cfg: All four submodules
   of MOM6 checked out
3. Set "externals = none" in Externals.cfg: No submodules checked out (empty directories)
4. Modify da_hooks external in Externals_MOM.cfg (see below): da_hooks is checked out with
   MOM6 .gitmodules configuration.

[pkg/MOM6_DA_hooks]
protocol = git
from_submodule = True
required = True

[jedwards4b]

this documentation change reads worse than the note it is replacing.

[mnlevy1981]

I'm going to punt on reviewing this and ask @dabail10 to do it for me - he can make sure this is working the way he expects it to in CICE (I don't have a sandbox using submodules anywhere). I can't add Dave to the reviewer list, though, so I guess I'll give this a green check after getting a thumbs-up from him.

[billsacks]

Thank you for taking this on @gold2718 ! And thanks in particular for adding a number of tests of the new behavior.

I haven't reviewed any of the new submodule-specific code, because I haven't really gotten my head around what's wanted in that respect, so I'll trust you and others on that. So I focused my review on the relatively few pieces of this that potentially affect the general operation of the code, as well as documentation.

To that end, things look pretty good, though see a few inline comments.

I do see, however, that some of the travis-ci tests are failing: these should all be in a passing state before this is merged. At a glance, I see failures in pylint (can be run locally with 'make lint', though I find that different versions of pylint pick up different things; that may be what's going on here) and a few tests with python3.

[dabail10]

I need the dummies guide to this. Here is what I imagine:

1. I have added an Externals_CICE.cfg file to my CESM_CICE wrapper. See here:

https://github.com/ESCOMP/CESM_CICE/tree/test_submodule

2. My test sandbox is:

/glade/work/dbailey/cesm2.1.1_sub

3. icepack is a submodule of https://github.com/ESCOMP/CICE. CICE is an external of CESM_CICE and goes in src.

However, I am getting the error:
fatal: No url found for submodule path "src" in .gitmodules

Dave

[gold2718]

1. My test sandbox is:

/glade/work/dbailey/cesm2.1.1_sub

Well, that is a weird error message (I think it should say "src/icepack" instead of "src"). However, when I look at your sandbox, I do not see any .gitmodules file. Where is it?

[mnlevy1981]

@gold2718 (cc @dabail10): it looks like src/ is empty, despite the block in Externals.cfg that should fill it

[cice]
branch = master
protocol = git
repo_url = https://github.com/ESCOMP/CICE
local_path = src
required = True
externals = ../Externals_CICE.cfg

components/cice/src should contain ESCOMP/CICE, which has a .gitmodules file. And then Dave wants to get src/icepack from .gitmodules (but we thought it should be [icepack] since presumably the working directory is already src/ when it looks for ../Externals_CICE.cfg?)

[gold2718]

@dabail10, @mnlevy1981,

Which Externals.cfg did you run? At the top level (Externals.cfg in the CESM root directory), there is no externals=Externals_CICE.cfg line so that was not run. That config puts ESCOMP/CICE into <root>/components/cice.

I do not see how this is supposed to work unless the ESCOMP/CICE repo has a submodule.

[mnlevy1981]

So you're saying the fix is in this block of the CESM Externals.cfg?

[cice]
branch = test_submodule
protocol = git
repo_url = https://github.com/ESCOMP/CESM_CICE
local_path = components/cice
required = True
+externals=Externals.cfg

That makes a little sense, but then why would @dabail10 get an error instead of just not trying to populate src/ at all?

[gold2718]

@mnlevy1981, I think I fixed the source of the error but maybe it is best if I can sit with someone to get CICE configured correctly (and fix new bugs that crop up).

[coveralls]

Coverage decreased (-2.2%) to 88.883% when pulling f72ffe7 on gold2718:submodules into 1926530 on ESMCI:master.

[billsacks]

Thanks for the fixes!

[mnlevy1981]

@dabail10 was able to sit down with @gold2718 and get this to do what he wants with CICE