[FIXED] RQ Job Terminated Unexpectedly

Issue

I have defined the rq task on module task.py as:

import django_rq
import logging
import requests

log = logging.getLogger(__name__)


def req_call():
    url = 'https://jsonplaceholder.typicode.com/posts'
    response = requests.get(url)
    return response


@django_rq.job('default')
def test_request_call_on_rq_job():
    response = req_call()
    log.warning(f"REQUEST's RESPONSE {response.json()}")

When I've offloaded a task in another module as:

if __name__ == '__main__':  # rq job test
    log.setLevel(logging.DEBUG)
    test_post_request_on_rq_job.delay()

I am getting error as:

[INFO] *** Listening on default...  [worker:702]
[INFO] default: rq_test_module.tasks.test_request_call_on_rq_job() (676c1945-9e05-4245-aeb2-65100cdb4169)  [worker:875]
[WARNING] Moving job to FailedJobRegistry (Work-horse terminated unexpectedly; waitpid returned 11 (signal 11); )  [worker:1114]

I then started doing debugging, then I saw error encounters as soon as the job task tried executing the request call i.e. requests.get(url) And if I remove the request call from the job, then it executed gracefully without any errors.

Signal 11 suggests a segmentation fault, I suspect something related to memory but not quite sure about it.

So anyone here encountered similar issue and workaround for this :)

Solution

I think I found the cause of the request being not executed gracefully on rq. I just guess it was due to flow at https://github.com/python/cpython/blob/main/Lib/urllib/request.py#L2619 not being able to pick the proxy setting in the RQ context. So I just skipped the way to reach there by setting NO_PROXY env to the URL I am trying to request.

Now I can run the request on RQ, as it should be.

But without disabling the proxy it still gives an error, maybe I need to dig more into this.

More context on this:

This seems to problem with MAC OS only, because of the import issue I highlighted above.

OS I am facing this issue: Apple M2 Pro, with OS 13.5 (22G74)

I will post more on this once I find concrete ideas for why it is working by setting `NO_PROXY.

I think I got it :),

Let's set up the this task on task.py module This task imports, underlying MAC os function as I outlined above and you can see exact same call in above link.

def test_mac_import_issue_on_request_lib_task():
    import sys
    if sys.platform == 'darwin':
        from _scproxy import _get_proxy_settings, _get_proxies
        _get_proxy_settings()
        _get_proxies()

Then enqueue them from any module, for example

if __name__ == '__main__':  # rq job test
    from tasks test_mac_import_issue_on_request_lib_task_on_rq_job
    test_mac_import_issue_on_request_lib_task_on_rq_job.delay()

Now run worker & execute your module which trigger this enqueue. Now check, whether it pops the error, for e.g [WARNING] Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 11) [worker:874]

That call was executed normally when you run independently (not in the RQ context), For e.g in you pyhon shell, if you try running:

from _scproxy import _get_proxy_settings, _get_proxies
get_proxy_settings() # This outputs dict
_get_proxy_settings() # This also outputs dict

So my next goal is to find out what's going on in the RQ context.

I will further update this upon finding more meaningful ideas on this.

Answered By - Ramchandra Shahi Thakuri