[FIXED] celery - will --max-memory-per-child restart workers instantly or allow them to finish the task?

Issue

Once a worker process crosses the --max-memory-per-child limit, does it get killed and restarted instantly? or does celery allow it to finish the current task (even if it has to consume some more memory while finishing) and then restart it?

Solution

According to my tests, it always lets the task finish even if the task uses much more memory than the limit. This is my test:

• set the memory limit to 15 MB
• created a task that:
  • consumes 50 MB sleeps
  • prints memory use, sleeps 30 seconds
  • consumes 50 MB more
  • prints memory use, sleeps 30 seconds
  • prints memory use, exit

I added the sleeps just in case to give time for celery to kill the task.

In all cases it always allows the task to finish and then the next task starts with zero (If I change the task to consume less memory than the limit, the next task does not start at zero).

Answered By - Martin Massera