[FIXED] Heroku + Celery: Critical Worker Timeout (503)

Issue

I have a Flask app deployed on Heroku, and I'm using Celery+Redis for async processing. The test page doesn't load when I send a request, and I receive this error:

2021-10-17T13:32:59.278920+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/test_route" host=www.mysite.dev request_id=f387a680-dca6-4e20-8f97-7f4d4a45d17b fwd="73.252.142.129,108.162.215.101" dyno=web.1 connect=0ms service=30000ms status=503 bytes=0 protocol=https
2021-10-17T13:32:59.776324+00:00 app[web.1]: [2021-10-17 13:32:59 +0000] [19] [CRITICAL] WORKER TIMEOUT (pid:23)
2021-10-17T13:32:59.778326+00:00 app[web.1]: [2021-10-17 13:32:59 +0000] [23] [INFO] Worker exiting (pid: 23)

My test route:

@main_bp.route('/test_route')
def test_route():
    celery_tasks.test_task.apply_async()

    return render_template('main/test.html')

Celery task:

@celery_app.task()
def test_task():
    print('Test task succeeded')

    return 1

The worker looks fine before a task is fired:

2021-10-17T13:54:28.080039+00:00 app[worker.1]: [2021-10-17 13:54:28,079: INFO/MainProcess] mingle: all alone
2021-10-17T13:54:28.080111+00:00 app[worker.1]: [2021-10-17 13:54:28,080: DEBUG/MainProcess] ^-- substep ok
2021-10-17T13:54:28.080178+00:00 app[worker.1]: [2021-10-17 13:54:28,080: DEBUG/MainProcess] | Consumer: Starting Tasks
2021-10-17T13:54:28.086858+00:00 app[worker.1]: [2021-10-17 13:54:28,086: DEBUG/MainProcess] ^-- substep ok
2021-10-17T13:54:28.086956+00:00 app[worker.1]: [2021-10-17 13:54:28,086: DEBUG/MainProcess] | Consumer: Starting Control
2021-10-17T13:54:28.094411+00:00 app[worker.1]: [2021-10-17 13:54:28,094: DEBUG/MainProcess] ^-- substep ok
2021-10-17T13:54:28.094481+00:00 app[worker.1]: [2021-10-17 13:54:28,094: DEBUG/MainProcess] | Consumer: Starting Gossip
2021-10-17T13:54:28.100571+00:00 app[worker.1]: [2021-10-17 13:54:28,100: DEBUG/MainProcess] ^-- substep ok
2021-10-17T13:54:28.100633+00:00 app[worker.1]: [2021-10-17 13:54:28,100: DEBUG/MainProcess] | Consumer: Starting event loop
2021-10-17T13:54:28.100745+00:00 app[worker.1]: [2021-10-17 13:54:28,100: DEBUG/MainProcess] | Worker: Hub.register Pool...
2021-10-17T13:54:28.101184+00:00 app[worker.1]: [2021-10-17 13:54:28,101: INFO/MainProcess] celery@82da4ee3-bdcc-46c6-b65b-d6221ba1693e ready.
2021-10-17T13:54:28.101315+00:00 app[worker.1]: [2021-10-17 13:54:28,101: DEBUG/MainProcess] basic.qos: prefetch_count->32

• I removed the celery task (.apply_async) from the route and confirmed the page loads without it
• Locally, I added time.sleep(30) to the celery task to confirm that the task offloads to the worker, and the page loads while the task is still running. This works as expected

I'm not sure why page loading is stalled in production, given the point of asynchronous processing is to not block requests. I'm also unsure how to dig deeper and debug this error- haven't been able to find much in the documentation.

I found this in the docs: Debugging Request Timeouts, but it only points to long-running tasks and infinite loops, neither of which are the case here

Solution

This is resolved by changing the celery app used in celery_tasks.py.

This doesn't work:

#celery_tasks.py

from celery import Celery

celery_app = Celery()

@celery_app.task()
def test_task():
    ...

This works:

#celery_tasks.py

from extensions import celery

celery_app = celery

@celery_app.task()
def test_task():
    ...

Where:

#extensions.py
import flask
from celery import Celery

class FlaskCelery(Celery):
    def __init__(self, *args, **kwargs):

        super(FlaskCelery, self).__init__(*args, **kwargs)
        self.patch_task()

        if 'app' in kwargs:
            self.init_app(kwargs['app'])

    def patch_task(self):
        TaskBase = self.Task
        _celery = self

        class ContextTask(TaskBase):
            abstract = True

            def __call__(self, *args, **kwargs):
                if flask.has_app_context():
                    return TaskBase.__call__(self, *args, **kwargs)
                else:
                    with _celery.app.app_context():
                        return TaskBase.__call__(self, *args, **kwargs)

        self.Task = ContextTask

    def init_app(self, app):
        self.app = app
        self.config_from_object(app.config)


celery = FlaskCelery()

and __init__.py uses the same class

Answered By - Dr. Funkenstein