[FIXED] How to pass broker_url from Django settings.py to a Celery service

Issue

I have Celery running as a service on Ubuntu 20.04 with RabbitMQ as a broker.

Celery repeatedly restarts because it cannot access the RabbitMQ url (RABBITMQ_BROKER), a variable held in a settings.py outside of the Django root directory.

The same happens if I try to initiate celery via command line.

I have confirmed that the variable is accessible from within Django from a views.py print statement.

If I place the RABBITMQ_BROKER variable inside the settings.py within the Django root celery works.

My question is, how do I get celery to recognise the variable RABBITMQ_BROKER when it is placed in /etc/opt/mydjangoproject/settings.py?

My celery.py file:

import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mydjangoproject.settings')

app = Celery('mydjangoproject')

default_config = 'mydjangoproject.celery_config'

app.config_from_object(default_config)
app.autodiscover_tasks()

My celery_config.py file:

from django.conf import settings

broker_url = settings.RABBITMQ_BROKER
etc...

The settings.py in /etc/opt/mydjangoproject/ (non relevant stuff deleted):

from mydangoproject.settings import *

RABBITMQ_BROKER = 'amqp://rabbitadmin:somepassword@somepassword@webserver:5672/mydangoproject'
etc...

My /etc/systemd/system/celery.service file:

[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=DJANGO_USER
Group=DJANGO_USER
EnvironmentFile=/etc/conf.d/celery
WorkingDirectory=/opt/mydjangoproject

ExecStart=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi start $CELERYD_NODES \
   --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
   --loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait $CELERYD_NODES \
    --pidfile=${CELERYD_PID_FILE} --loglevel="${CELERYD_LOG_LEVEL}"'
ExecReload=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi restart $CELERYD_NODES \
    --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
    --loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
Restart=always

[Install]
WantedBy=multi-user.target

My /etc/conf.d/celery file:

CELERYD_NODES="worker"
CELERY_BIN="/opt/mydjangoproject/venv/bin/celery"
CELERY_APP="mydjangoproject"
CELERYD_CHDIR="/opt/mydjangoproject/"
CELERYD_MULTI="multi"
CELERYD_OPTS="--time-limit=300 --without-heartbeat --without-gossip --without-mingle"
CELERYD_PID_FILE="/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"

Solution

I solved this by adding the following to /etc/systemd/system/celery.service

Environment="PYTHONPATH=/etc/opt/mydjangoproject:/opt/mydjangoproject"
Environment="DJANGO_SETTINGS_MODULE=settings"

Answered By - user12231454