[FIXED] How to configure celery for concurrent execution with multi-process?

Issue

I have a task that talks to an external API, the json response is quite large and I have to make this call multiple times followed by further python processing. To make this less time-consuming, I initially tried:

def make_call(*args, **kwargs):
    pass

def make_another(*args, **kwargs):
    pass

def get_calls():
    return make_call, make_another

def task(*args, **kwargs):
    procs = [Process(target=get_calls()[i], args=(,), 
    kwargs={}) for i in range(3)]
    _start = [proc.start() for proc in procs]
    _join = [proc.join() for proc in procs]

# 
transaction.on_commit(lambda: task.delay()) 

However, I ran into an AssertionError: daemonic processes are not allowed to have children. What would be my best approach to speed up a celery task with additional processes?

Solution

A Celery worker already creates many processes. Take advantage of the many worker processes instead of creating child processes. You can delegate work amongst the celery workers instead. This will result in a more stable/reliable execution.

You could either just create many tasks from your client code or make use of celery's primitives like chains or chords to parallelize the work. These can also be composed with other primitives like groups, etc.

For example, in your scenario, you may have two tasks: one to make the API call(s) make_api_call and another to parse the response parse_response. You can chain these together.

# chain another task when a task completes successfully
res = make_api_call.apply_async((0,), link=parse_response.s())

# chain syntax 1
result_1 = chain(make_api_call.s(1), parse_response.s())
# syntax 2 with | operator
result_b = make_api_call.s(2) | parse_response.s()

# can group chains
job = group([
   chain(make_api_call.s(i), parse_response.s()) 
   for i in range(3)
   ]
)
result = job.apply_async()

This is just a generic example. You can create task(s) and compose them to however your workflow needs. See: Canvas: Designing Work-flows for more information.

Answered By - sytech