No way to achieve typing of managers built from querysets

[antonagestam]

Currently there seems to be no way to get non-Any typing for Model.objects and by extension also Model.objects.get(), Model.objects.all() et al, for models that specify a custom manager built from a QuerySet. I'm submitting this as a tracking-issue, even though there are individual issues for some of the underlying problems, I thought it'd be nice to have a unified discussion about how to find a way forward here, whether it's a workaround or a need of a fix in the stubs or in the extension.

Option 1

Managers built dynamically within the model class body, just seem to not work and induces Any.

CustomQSSelf = TypeVar("CustomQSSelf", bound="CustomQS")

class CustomQS(QuerySet["A"]):
    def custom_filter(self: CustomQSSelf) -> CustomQSSelf:
        return self.filter(pk=1)

class A(Model):
    objects = BaseManager.from_queryset(CustomQS)()

reveal_type(A.objects)  # -> Any
reveal_type(A.objects.get())  # -> Any
reveal_type(A.objects.all())  # -> Any

Option 2

Moving the manager construction to outside of the model body. This induces very strange phantom errors, as well as errors complaining that the "self annotations" on the QuerySet are unbound.

CustomQSSelf = TypeVar("CustomQSSelf", bound="CustomQS")

class CustomQS(QuerySet["A"]):
    def custom_filter(self: CustomQSSelf) -> CustomQSSelf:
        return self.filter(pk=1)

AManager = BaseManager.from_queryset(CustomQS)

class A(Model):
    objects = AManager()

error: Type variable "...CustomQSSelf" is unbound  [valid-type]

Option 3

Using QuerySet.as_manager(). I tried this even though it seems generally recommended against in other issues, and to have less support than BaseManager.from_queryset(). As expected this also yields Any, whether or not it's defined inside the model body.

CustomQSSelf = TypeVar("CustomQSSelf", bound="CustomQS")

class CustomQS(QuerySet["A"]):
    def custom_filter(self: CustomQSSelf) -> CustomQSSelf:
        return self.filter(pk=1)

class A(Model):
    objects = CustomQS.as_manager()

reveal_type(Lease.objects.get())  # -> Any
reveal_type(Lease.objects.all())  # -> django.db.models.manager.Manager[Any]

I'd be really happy if someone has a workaround for this issue as it was a real cold shower realizing a large portion of our models are inducing Any.

[kevinmarsh]

Moving the manager construction to outside of the model body

I end up doing this a lot, I haven't run into the phantom errors but will keep an eye out for it.

This feels a bit hacky (so hopefully this spurs further discussion) but could you try option 2 and change the return annotation from

- def custom_filter(self: CustomQSSelf) -> CustomQSSelf:
+ def custom_filter(self: CustomQSSelf) -> "CustomQS":

That gets around the unbound issue, and avoids the Any return type issue you've described.

reveal_type(A.objects)  # -> AManager[A]
reveal_type(A.objects.get())  # -> A*
reveal_type(A.objects.all())  # -> CustomQS[A*]

Can't say it's a fail safe solution but what about a fuller example like

from typing import TypeVar
from django.db import models

CustomQuerySetSelf = TypeVar("CustomQuerySetSelf", bound="CustomQuerySet")

class CustomQuerySet(models.QuerySet["MyModel"]):
    def my_method(self: CustomQuerySetSelf) -> "CustomQuerySet":
        return self.filter(pk=1)

MyManager = models.Manager.from_queryset(CustomQuerySet)

class MyModel(models.Model):
    objects = MyManager()

# Manager
reveal_type(MyModel.objects)  # MyManager[MyModel]

# QuerySet
reveal_type(MyModel.objects.all())              # CustomQuerySet[MyModel*]
reveal_type(MyModel.objects.my_method())        # CustomQuerySet
reveal_type(MyModel.objects.all().my_method())  # CustomQuerySet

# Instance
reveal_type(MyModel.objects.get())                    # MyModel*
reveal_type(MyModel.objects.my_method().get())        # MyModel*
reveal_type(MyModel.objects.all().my_method().get())  # MyModel*

# Optional Instance
reveal_type(MyModel.objects.first())                    # Union[MyModel*, None]
reveal_type(MyModel.objects.my_method().first())        # Union[MyModel*, None]
reveal_type(MyModel.objects.all().my_method().first())  # Union[MyModel*, None]

The reveal_type(MyModel.objects.my_method()) doesn't look like it's bound to the model (compared to reveal_type(MyModel.objects.all())) but calling .get on it does seem to return the correct model type?

[antonagestam]

@kevinmarsh Oh, I'll try getting rid of the self types and see if that helps with the phantom errors! Thanks for the input 👍

[antonagestam]

@kevinmarsh So, I've inspected another model in our codebase now that doesn't use self types, and tried breaking out the manager creation from the model class body. It seems to be an improvement at least, because I don't get weird phantom errors, and it does achieve typing of .objects.get() et al (could tell because I found a few None-related bugs ...).

But now I instead bump into another issue:

error: Argument "queryset" to "_resolve_extras" has incompatible type "MyModel_MyModelManager1[MyModel]";
expected "MyModelQuerySet"

Now it instead seems like something is broken with the descriptor typing, and everything that comes out of MyModel.objects.filter() and similar are typed as a manager, instead of a queryset.

[antonagestam]

... and while the distinction between queryset and the generated manager class mostly isn't important, I have no idea how to write a union type that accepts the generated manager class.

[RJPercival]

I suspect the problem with unbound type vars is that django-stubs copies all of the methods from the QuerySet onto the Manager. However, I don't think it copies the type arguments along with them, so those methods end up with unbound type vars.

[antonagestam]

This has been fixed in recent versions and we were able to update by changing to Option 2 mentioned in the description.