Custom manager causes phantom errors and sometimes crash

[christianbundy]

Bug report

What's wrong

Still working on a minimal repro, but here's my stack trace:

Traceback (most recent call last):
  File "/.venv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/.venv/lib/python3.9/site-packages/mypy/__main__.py", line 11, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "mypy/main.py", line 87, in main
  File "mypy/main.py", line 165, in run_build
  File "mypy/build.py", line 179, in build
  File "mypy/build.py", line 254, in _build
  File "mypy/build.py", line 2697, in dispatch
  File "mypy/build.py", line 3021, in process_graph
  File "mypy/build.py", line 3137, in process_stale_scc
  File "mypy/errors.py", line 565, in file_messages
  File "mypy/errors.py", line 538, in format_messages
IndexError: list index out of range

I have a FooManager:

from django.db import models

class FooManager(models.Manager):

    def create_with_foo(self, *args, **kwargs) -> models.Model: # Line 7
        pass

    def save_foo(self, model: models.Model) -> None: # Line 25
        pass

It's often used on models like this:

objects = FooManager()

For some reason, three completely unrelated files are getting these errors:

some_file.py:7: error: Name "models" is not defined  [name-defined]
    )
    ^
some_file.py:25: error: Name "models" is not defined  [name-defined]
            does_not_matter_anything_could_be_here()

I tried adding whitespace to the top of some_file.py but the line numbers didn't change. Odd. I looked through my project for files where lines 7 and 25 referenced models and found my FooManager. I added 1 line to the top of FooManager and the mypy errors incremented by 1. I added 100 lines to the top of FooManager and got this sweet stack trace.

How is that should be

There should be no errors.

System information

• OS:
• python version: 3.9.6
• django version: 3.1
• mypy version: 0.910
• django-stubs version: 1.8.0
• django-stubs-ext version: 0.3.1

[antonagestam]

I'm running into this too. I realized probably all of our models that define custom managers using objects = BaseManager.from_queryset(...)() reveals Any for reveal_type(Models.objects), and therefor also for Model.objects.{get,all,...}().

At first sight it looked like moving the manager definition outside of the model class body fixed the issue, but after doing that a whole sea of odd errors appear in the manager module. Things like this:

src/catalog/models.py:399: error: Name 'Sequence' is not defined  [name-defined]
            state = getattr(self, annotation)

[flaeppe]

At first sight it looked like moving the manager definition outside of the model class body fixed the issue, but after doing that a whole sea of odd errors appear in the manager module. Things like this:

src/catalog/models.py:399: error: Name 'Sequence' is not defined  [name-defined]
            state = getattr(self, annotation)

@antonagestam I think I've found the cause of these kind of errors. A closer look in

django-stubs/mypy_django_plugin/transformers/managers.py

Line 9 in fafb200

def create_new_manager_class_from_from_queryset_method(ctx: DynamicClassDefContext) -> None:

Where it seems like a new "manager node" is created and from what I can tell, that new manager node ends up where the ... = BaseManager.from_queryset(CustomQuerySet) call lives

django-stubs/mypy_django_plugin/transformers/managers.py

Lines 52 to 59 in fafb200

	new_manager_info.line = ctx.call.line
	new_manager_info.type_vars = base_manager_info.type_vars
	new_manager_info.defn.type_vars = base_manager_info.defn.type_vars
	new_manager_info.defn.line = ctx.call.line
	new_manager_info.metaclass_type = new_manager_info.calculate_metaclass_type()
	current_module = semanal_api.cur_mod_node
	current_module.names[ctx.name] = SymbolTableNode(GDEF, new_manager_info, plugin_generated=True)

I'm quite convinced that the error you're encountering appears when:

1. A custom QuerySet is declared in a module different from where the using Model is declared
2. The custom QuerySet declares methods with types that are not present in the module where the using Model is declared (e.g. Sequence)

---

An additional note regarding

... models that define custom managers using objects = BaseManager.from_queryset(...)() reveals Any for reveal_type(Models.objects), and therefor also for Model.objects.{get,all,...}().

It seems that the reason for Any is that create_new_manager_class_from_from_queryset_method is never called when doing: objects = BaseManager.from_queryset(...)()

[antonagestam]

@flaeppe I think your findings could explain why I had a hard time creating a minimal reproduction, I think I missed trying with manager and model in different modules. While I was completely dumbfounded by these errors, it sounds like you've figured it out 👍

[aleksanb]

I just hit Name 'Sequence' is not defined [name-defined] when doing

MyManager = MyBaseManager.from_queryset(MyBaseQueryset)

class SomeClass(models.Model):
  objects = MyManager()

Even though all models / managers / querysets were defined in the same function.
I ended up working around this by removing the manager methods and doing objects = MyBaseQueryset.as_manager() instead.

Using newest django-stubs.

[flaeppe]

@aleksanb That's interesting, can you produce a repro? As far as I can tell the test that I found, below, should cover the case you describe:

django-stubs/tests/typecheck/managers/querysets/test_from_queryset.yml

Lines 284 to 311 in 8d8b8cd

	- case: from_queryset_can_resolve_explicit_any_methods
	main: |
	from myapp.models import MyModel
	reveal_type(MyModel.objects) # N: Revealed type is "myapp.models.NewManager[myapp.models.MyModel]"
	reveal_type(MyModel.objects.queryset_method(1)) # N: Revealed type is "Any"
	reveal_type(MyModel.objects.queryset_method) # N: Revealed type is "def (qarg: Any) -> Any"
	reveal_type(MyModel.objects.manager_method(2)) # N: Revealed type is "Any"
	reveal_type(MyModel.objects.manager_method) # N: Revealed type is "def (marg: Any) -> Any"
	installed_apps:
	- myapp
	files:
	- path: myapp/__init__.py
	- path: myapp/models.py
	content: |
	from django.db import models
	from typing import Any
	class ModelQuerySet(models.QuerySet):
	def queryset_method(self, qarg: Any) -> Any:
	return 'hello'
	class MyManager(models.Manager):
	def manager_method(self, marg: Any) -> Any:
	return 'hello'
	NewManager = MyManager.from_queryset(ModelQuerySet)
	class MyModel(models.Model):
	objects = NewManager()

i.e. Custom QuerySet, custom Manager, building a new manager via Manager.from_queryset(QuerySet) where both the custom queryset and manager has method signatures containing imported types.

Also, be aware of that if you use .as_manager() I think the type of the manager will be Any.

[aleksanb]

I'll try making a minimal reproduce, but won't have time till next week.
Perhaps it's some interaction with our model structure or something.

[flaeppe]

Cool, think you should open a new issue if you figure out a way to reproduce the error.

[christianbundy]

I think I might have a lead.

Pass:

class FooManager(models.Manager['Foo']):
    pass

FooModelManager = FooManager.from_queryset(FooQuerySet)

class Foo(models.Model):
    objects = FooModelManager()

Fail:

FooModelManager = models.Manager.from_queryset(FooQuerySet)

class Foo(models.Model):
    objects = FooModelManager()

I've opened #1017 to try to run this failing test in CI.

[jose-reveni]

@aleksanb I have been getting a similar error too!

[aleksanb]

I've opened #1017 to try to run this failing test in CI.

Seems like CI doesn't run automatically anymore?
Someone needs to approve the CI build to run.

[ljodal]

I'm seeing the same issue and have traced it back to #991, will try to see if I can reproduce it in a test

[ljodal]

This test reproduces the issue locally for me, but not on CI for some reason (see #1021):

-   case: from_queryset_custom_auth_user_model
    main: |
        from users import User
    custom_settings: |
        AUTH_USER_MODEL = "users.User"
        INSTALLED_APPS = ("django.contrib.auth", "django.contrib.contenttypes", "users")
    files:
        -   path: users/__init__.py
        -   path: users/models.py
            content: |
                from django.contrib.auth.models import AbstractBaseUser
                from django.db import models

                from .querysets import UserQuerySet

                UserManager = models.Manager.from_queryset(UserQuerySet)

                class User(AbstractBaseUser):
                    email = models.EmailField(unique=True)
                    objects = UserManager()
                    USERNAME_FIELD = "email"

        -   path: users/querysets.py
            content: |
                from django.db import models
                from typing import Optional, TYPE_CHECKING

                if TYPE_CHECKING:
                    from .models import User

                class UserQuerySet(models.QuerySet["User"]):
                    pass