show_doc errors if a dependency in the nbdev group has a sub-dependency that isn't installed

[warner-benjamin]

In a working env with pillow-simd installed instead of pillow, the following minimal code (split into separate notebook cells) will error out with a DistributionNotFound error whether pillow-simd is imported, referenced by any code in the module, or neither imported nor referenced.

from nbdev.showdoc import *

class test():
    "this is a test"
    def __init__(self):
        pass

show_doc(test)

show_doc also errors out on pillow-simd in all library nbdev notebooks, including when running nbdev_test and nbdev_preview. However, importing and running code from the subset of modules which use pillow-simd doesn't trigger any errors, suggesting this is exclusively a nbdev2 issue.

Full stack trace of the error:

DistributionNotFound in untitled.ipynb:
===========================================================================

While Executing Cell #16:
---------------------------------------------------------------------------
DistributionNotFound                      Traceback (most recent call last)
~/miniconda3/lib/python3.9/site-packages/IPython/core/formatters.py in __call__(self, obj)
    343             method = get_real_method(obj, self.print_method)
    344             if method is not None:
--> 345                 return method()
    346             return None
    347         else:

~/miniconda3/lib/python3.9/site-packages/nbdev/showdoc.py in _repr_markdown_(self)
    166     def _repr_markdown_(self):
    167         doc = '---\n\n'
--> 168         src = NbdevLookup().code(self.fn)
    169         if src: doc += _ext_link(src, 'source', 'style="float:right; font-size:smaller"') + '\n\n'
    170         h = '#'*self.title_level

~/miniconda3/lib/python3.9/site-packages/nbdev/doclinks.py in __init__(self, strip_libs, incl_libs, skip_mods)
    193         if incl_libs is not None: incl_libs = (L(incl_libs)+strip_libs).unique()
    194         # Dict from lib name to _nbdev module for incl_libs (defaults to all)
--> 195         self.entries = {o.name: _qual_syms(o.load()) for o in list(pkg_resources.iter_entry_points(group='nbdev'))
    196                        if incl_libs is None or o.dist.key in incl_libs}
    197         py_syms = merge(*L(o['syms'].values() for o in self.entries.values()).concat())

~/miniconda3/lib/python3.9/site-packages/nbdev/doclinks.py in <dictcomp>(.0)
    193         if incl_libs is not None: incl_libs = (L(incl_libs)+strip_libs).unique()
    194         # Dict from lib name to _nbdev module for incl_libs (defaults to all)
--> 195         self.entries = {o.name: _qual_syms(o.load()) for o in list(pkg_resources.iter_entry_points(group='nbdev'))
    196                        if incl_libs is None or o.dist.key in incl_libs}
    197         py_syms = merge(*L(o['syms'].values() for o in self.entries.values()).concat())

~/miniconda3/lib/python3.9/site-packages/pkg_resources/__init__.py in load(self, require, *args, **kwargs)
   2447             )
   2448         if require:
-> 2449             self.require(*args, **kwargs)
   2450         return self.resolve()
   2451 

~/miniconda3/lib/python3.9/site-packages/pkg_resources/__init__.py in require(self, env, installer)
   2470         # requirements for that extra are purely optional and skip over them.
   2471         reqs = self.dist.requires(self.extras)
-> 2472         items = working_set.resolve(reqs, env, installer, extras=self.extras)
   2473         list(map(working_set.add, items))
   2474 

~/miniconda3/lib/python3.9/site-packages/pkg_resources/__init__.py in resolve(self, requirements, env, installer, replace_conflicting, extras)
    770                     if dist is None:
    771                         requirers = required_by.get(req, None)
--> 772                         raise DistributionNotFound(req, requirers)
    773                 to_activate.append(dist)
    774             if dist not in req:

DistributionNotFound: The 'pillow>6.0.0' distribution was not found and is required by the application

[seeM]

I'm unable to reproduce this issue. Could you please help me figure out what I'm missing?

Details:

• repo: test-pillow-simd
• tests succeed in GH action
• docs build successfully in GH action
• action installed pillow-simd since it's in our settings.ini file
• test notebook as described.

[warner-benjamin]

In a new environment I cloned test-pillow-simd and the notebook ran without issue.

I then installed PyTorch^ and the notebook ran without issue.

After installing fastai^ I was able to reproduce the original error in the new environment while running the test-pillow-simd notebook.

^ Both of these steps required uninstalling pillow and reinstalling pillow-simd, since these packages will install pillow as a dependency.

[seeM]

I was able to repro the issue as you said. Starting from the repo above:

pip install -e '.[dev]'
pip install fastai
pip uninstall Pillow
nbdev_test

IIUC the error is raised because doclinks uses EntryPoint.load (source) which runs both EntryPoint.require (checks that requirements are satisfied) and EntryPoint.resolve (imports and returns the module). I think require is too strict for our use-case -- I don't think doclinks should care about whether a user's dependencies are met. Will make a fix. cc @hamelsmu @jph00