[FIXED] Asyncio subprocess is not being cancelled properly on interrupt

Issue

I have the following code that doesn't work the way I expected it to. I wanted to be able to interrupt my program with SIGINT (Ctrl-C). It should then cancel tasks that run subprocesses and run some code upon cancellation, but I can't get that section to run. It's seems like the signal is being intercepted by something other than the event handler I set up. This isn't what I expected.

Run on ubuntu 22.04 python 3.10.6 and also a docker image for python 3.11.2

import asyncio
import signal

jobs = set()

async def create_worker():
    task = asyncio.create_task(worker())
    jobs.add(task)
    task.add_done_callback(jobs.discard)

async def worker():
    try:
        proc = await asyncio.create_subprocess_shell('sleep 10')
        await proc.communicate()
        print("done")
    except asyncio.CancelledError:
        print("my task is getting cancelled :\\") # should run this section, but doesn't
        raise

async def main() -> None:
    loop = asyncio.get_event_loop()
    loop_hold = asyncio.Event()

    async def graceful_shutdown():
        unfinished_jobs = list(jobs)

        for job in unfinished_jobs:
            job.cancel()

        await asyncio.wait(unfinished_jobs)

        loop_hold.set()

    loop.add_signal_handler(
        signal.SIGINT,
        lambda: asyncio.create_task(graceful_shutdown())
    )

    for _ in range(10):
        await create_worker()
    
    await loop_hold.wait()

if __name__ == "__main__":
    asyncio.run(main())

When I interrupt the program with Ctrl-C before command completion, I was expecting to run some code on worker cancellation. Maybe something like the following:

my task is getting cancelled :\
my task is getting cancelled :\
...
my task is getting cancelled :\

but I'm getting this instead:

done
done
...
done

I've also tried this with asyncio.create_subprocess_exec but I'm getting the same problem.

Solution

It turns out that the SIGINT signal I was sending was being intercepted by the child processes before the event loop could handle the signal. The signal gets sent to all of the children because they're in the same process group. So changing the process group in the subprocess call seems to solve the problem I was having.

Here's the updated worker code:

async def worker():
    try:
        proc = await asyncio.create_subprocess_shell(
            'sleep 10',
            preexec_fn=os.setpgrp
        )
        await proc.wait()
        print("done")
    except asyncio.CancelledError:
        print("my task is getting cancelled :\\")
        raise

I discovered this answer through a slightly different question Terminate external program run through asyncio with specific signal

More information here for whoever is interested:

https://en.wikipedia.org/wiki/Process_group#Applications

https://www.baeldung.com/linux/signal-propagation

Answered By - brent-mercer