[FIXED] Django - fresh database and no such table

Issue

I'm using my complex jobs app in production and it works fine, with sqlite database. I'm trying to create new database from scratch and I cannot do that nor using migrations nor trying to make them once again:

When I'm trying to reuse my migrations:

rm db.sqlite3
python manage.py migrate
Traceback (most recent call last):
  File "/backend/server/manage.py", line 22, in <module>
    main()
  File "/backend/server/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/backend/env/lib/python3.10/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/backend/env/lib/python3.10/site-packages/django/core/management/__init__.py", line 377, in execute
    django.setup()
  File "/backend/env/lib/python3.10/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/backend/env/lib/python3.10/site-packages/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/backend/env/lib/python3.10/site-packages/django/contrib/admin/apps.py", line 24, in ready
    self.module.autodiscover()
  File "/backend/env/lib/python3.10/site-packages/django/contrib/admin/__init__.py", line 24, in autodiscover
    autodiscover_modules('admin', register_to=site)
  File "/backend/env/lib/python3.10/site-packages/django/utils/module_loading.py", line 47, in autodiscover_modules
    import_module('%s.%s' % (app_config.name, module_to_search))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/backend/server/jobs/admin.py", line 57, in <module>
    class TagCategoryDefaultImportanceAdmin(admin.TabularInline):
  File "/backend/server/jobs/admin.py", line 59, in TagCategoryDefaultImportanceAdmin
    extra = len(Role.objects.all())
  File "/backend/env/lib/python3.10/site-packages/django/db/models/query.py", line 269, in __len__
    self._fetch_all()
  File "/backend/env/lib/python3.10/site-packages/django/db/models/query.py", line 1303, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/backend/env/lib/python3.10/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/backend/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/backend/env/lib/python3.10/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/backend/env/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/backend/env/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/backend/env/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
    with self.db.wrap_database_errors:
  File "/backend/env/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/backend/env/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/backend/env/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: jobs_role

When I've tried to remove all migration files:

  mv jobs/migrations jobs/old_migrations
  rm -Rf db.sqlite3
  python manage.py makemigrations
Traceback (most recent call last):
  File "/backend/server/manage.py", line 22, in <module>
    main()
  File "/backend/server/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/backend/env/lib/python3.10/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/backend/env/lib/python3.10/site-packages/django/core/management/__init__.py", line 377, in execute
    django.setup()
  File "/backend/env/lib/python3.10/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/backend/env/lib/python3.10/site-packages/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/backend/env/lib/python3.10/site-packages/django/contrib/admin/apps.py", line 24, in ready
    self.module.autodiscover()
  File "/backend/env/lib/python3.10/site-packages/django/contrib/admin/__init__.py", line 24, in autodiscover
    autodiscover_modules('admin', register_to=site)
  File "/backend/env/lib/python3.10/site-packages/django/utils/module_loading.py", line 47, in autodiscover_modules
    import_module('%s.%s' % (app_config.name, module_to_search))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/backend/server/jobs/admin.py", line 57, in <module>
    class TagCategoryDefaultImportanceAdmin(admin.TabularInline):
  File "/backend/server/jobs/admin.py", line 59, in TagCategoryDefaultImportanceAdmin
    extra = len(Role.objects.all())
  File "/backend/env/lib/python3.10/site-packages/django/db/models/query.py", line 269, in __len__
    self._fetch_all()
  File "/backend/env/lib/python3.10/site-packages/django/db/models/query.py", line 1303, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/backend/env/lib/python3.10/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/backend/env/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/backend/env/lib/python3.10/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/backend/env/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/backend/env/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/backend/env/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
    with self.db.wrap_database_errors:
  File "/backend/env/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/backend/env/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/backend/env/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: jobs_role

How can I investigate what is going on? Where is the problem with jobs_role table?

It figures out, that problem is related to my admin.py file

class TagCategoryDefaultImportanceAdmin(admin.TabularInline):
    model = TagCategoryDefaultImportance
    extra = len(Role.objects.all())
    max_num = len(Role.objects.all())
    formset = TagCategoryDefaultImportanceAdminFormse

The extra and max_num fields cause the issue. How can I handle that in a correct manner?

Solution

As the error indicates:

File "/backend/server/jobs/admin.py", line 59, in TagCategoryDefaultImportanceAdmin
    extra = len(Role.objects.all())

This meansthat when interpreting the file, it will already run this query. This is thus also the case if the database is not (yet) fully migrated, hence the error.

As a (strong) rule of thumb, you should try to avoid running queries when interpreting the file (reading the file into memory), since these will thus run before the server starts running, and before the database might be properly set up. Even if that would work, it would mean that extra takes as value the number if items when you start the server: if you would later add/remove a role, then extra would not change.

Usually you should try to wrap the logic into a function such that it only will evaluate Role.objects.all() when it runs the function, and such admin functions are then typically only triggered when the user visits the admin pages.

As for this specific case, you can use the .get_extra(…) ^[Django-doc] and .get_max_num(…) methods ^[Django-doc]:

class TagCategoryDefaultImportanceAdmin(admin.TabularInline):
    model = TagCategoryDefaultImportance
    formset = TagCategoryDefaultImportanceAdminFormset
    
    def get_max_num(request, obj=None, **kwargs):
        return Role.objects.count()
    
    def get_extra(self, request, obj=None, **kwargs):
        return Role.objects.count()

Answered By - Willem Van Onsem