[FIXED] Why does asyncio.CancelledError need to be re-raised?

Issue

I have the following aiohttp WebSocket handler:

async def websocket_handler(request):
  ws = None
  
  if 'my-websocket-clients' not in request.app:
    request.app['my-websocket-clients'] = []
  
  print('Websocket connection starting', request.path, request.query)
  
  try:
    ws = aiohttp.web.WebSocketResponse(autoping=True, heartbeat=10.0, compress=True)
    await ws.prepare(request)
    request.app['my-websocket-clients'].append(ws)
    print('Websocket connection ready', len(request.app['my-websocket-clients']))
    async for msg in ws:
      await websocket_message(request, ws, msg)
  except asyncio.exceptions.CancelledError as e:
    print('Websocket connection was closed uncleanly ' + str(e))
    # ~~~~~~ re-raise here? ~~~~~~
  except:
    traceback.print_exc()
  finally:
    try:
      await ws.close()
    except:
      traceback.print_exc()
  
  if ws in request.app['my-websocket-clients']:
    request.app['my-websocket-clients'].remove(ws)
  
  print('Websocket connection closed', len(request.app['my-websocket-clients']))
  
  if ws is None:
    ws = aiohttp.web.Response()
  return ws

According to the documentation, "In almost all situations the exception [asyncio.exceptions.CancelledError] must be re-raised"

Do I need to re-raise the exception in the location marked in the code? This would require me to rewrite the code which removes the client from the client list. Would I also need to re-raise the catch-all except block which comes after the asyncio.exceptions.CancelledError block?

Why do I need to re-raise asyncio.exceptions.CancelledError in the first place, if I should need to do that in this case? In which situations wouldn't I need to re-raise that exception?

Solution

In case of catching CancelledError, you need to be very careful.

Prior to Python 3.8, it was easy to unintentionally suppress this error with code like this:

try:
    await operation()
except Exception:
    log.error('Operataion failed. Will retry later.')

Since Python 3.8 CancelledError is subclass of BaseException and it is necessary to always explicitly handle this error:

try:
    await operation()
except CancelledError:
    # cleanup
    raise
except Exception:
    log.error('Opertaion failed. will retry later.')

The main issue here is that you cannot cancel a task that suppress CancelledError.

But, in any case, the recommendation to re-raise is not absolute and is given for the general case. If you know what you are doing, you can handle the CancelledError and finish the coroutine without throwing it again. It should be minded that when a task is cancelled, its cancellation will generally look like a chain of CancelledErrors, which will be thrown from the innermost await call and thrown up the chain of awaits, and in this case, we break this chain, and must correctly handle this situation.

In the case of the aiohttp websocket handler, I think it is acceptable not to re-raise the CancelledError if you ensure that the resources are cleaned up correctly and the handler exits.

Answered By - alex_noname