[FIXED] Always Defer a Field in Django

Issue

How do I make a field on a Django model deferred for all queries of that model without needing to put a defer on every query?

Research

This was requested as a feature in 2014 and rejected in 2022.

Baring such a feature native to Django, the obvious idea is to make a custom manager like this:

class DeferedFieldManager(models.Manager):

    def __init__(self, defered_fields=[]):
        super().__init__()
        self.defered_fields = defered_fields

    def get_queryset(self, *args, **kwargs):
        return super().get_queryset(*args, **kwargs
            ).defer(*self.defered_fields)

class B(models.Model):
    pass

class A(models.Model):
    big_field = models.TextField(null=True)
    b = models.ForeignKey(B, related_name="a_s")

    objects = DeferedFieldManager(["big_field"])

class C(models.Model):
    a = models.ForeignKey(A)

class D(models.Model):
    a = models.OneToOneField(A)

class E(models.Model):
    a_s = models.ManyToManyField(A)

However, while this works for A.objects.first() (direct lookups), it doesn't work for B.objects.first().a_s.all() (one-to-manys), C.objects.first().a (many-to-ones), D.objects.first().a (one-to-ones), or E.objects.first().a_s.all() (many-to-manys).

The thing I find particularly confusing here is that this is the default manager for my object, which means it should also be the default for the reverse lookups (the one-to-manys and many-to-manys), yet this isn't working. Per the Django docs:

By default the RelatedManager used for reverse relations is a subclass of the default manager for that model.

An easy way to test this is to drop the field that should be deferred from the database, and the code will only error with an OperationalError: no such column if the field is not properly deferred. To test, do the following steps:

1. Data setup:

   b = B.objects.create()
   a = A.objects.create(b=b)
   c = C.objects.create(a=a)
   d = D.objects.create(a=a)
   e = E.objects.create()
   e.a_s.add(a)
   

2. Comment out big_field
3. manage.py makemigrations
4. manage.py migrate
5. Comment in big_field
6. Run tests:

   from django.db import OperationalError
   def test(test_name, f, attr=None):
       try:
           if attr:
               x = getattr(f(), attr)
           else:
               x = f()
           assert isinstance(x, A)
           print(f"{test_name}:\tpass")
       except OperationalError:
           print(f"{test_name}:\tFAIL!!!")
   
   test("Direct Lookup", A.objects.first)
   test("One-to-Many", B.objects.first().a_s.first)
   test("Many-to-One", C.objects.first, "a")
   test("One-to-One", D.objects.first, "a")
   test("Many-to-Many", E.objects.first().a_s.first)
   

If the tests above all pass, the field has been properly deferred.

I'm currently getting:

Direct Lookup:  pass
One-to-Many:    FAIL!!!
Many-to-One:    FAIL!!!
One-to-One:     FAIL!!!
Many-to-Many:   FAIL!!!

Partial Answer

@aaron's answer solves half of the failing cases.

If I change A to have:

class Meta:
    base_manager_name = 'objects'

I now get the following from tests:

Direct Lookup:  pass
One-to-Many:    FAIL!!!
Many-to-One:    pass
One-to-One:     pass
Many-to-Many:   FAIL!!!

This still does not work for the revere lookups.

Solution

Set Meta.base_manager_name to 'objects'.

class A(models.Model):
    big_field = models.TextField(null=True)
    b = models.ForeignKey(B, related_name="a_s")

    objects = DeferedFieldManager(["big_field"])

    class Meta:
        base_manager_name = 'objects'

From https://docs.djangoproject.com/en/4.1/topics/db/managers/#using-managers-for-related-object-access:

Using managers for related object access

By default, Django uses an instance of the Model._base_manager manager class when accessing related objects (i.e. choice.question), not the _default_manager on the related object. This is because Django needs to be able to retrieve the related object, even if it would otherwise be filtered out (and hence be inaccessible) by the default manager.

If the normal base manager class (django.db.models.Manager) isn’t appropriate for your circumstances, you can tell Django which class to use by setting Meta.base_manager_name.

Reverse Many-to-One and Many-to-Many managers

The "One-To-Many" case in the question is a Reverse Many-To-One.

Django subclasses the manager class to override the behaviour, and then instantiates it — without the defered_fields argument passed to __init__ since django.db.models.Manager and its subclasses are not expected to have parameters.

Thus, you need something like:

def make_defered_field_manager(defered_fields):
    class DeferedFieldManager(models.Manager):
        def get_queryset(self, *args, **kwargs):
            return super().get_queryset(*args, **kwargs).defer(*defered_fields)
    return DeferedFieldManager()

Usage:

# objects = DeferedFieldManager(["big_field"])
objects = make_defered_field_manager(["big_field"])

Answered By - aaron