[FIXED] Asyncios await reader.read() is waiting forever

Issue

I have a small Server written like this:

async def handle_client(reader, writer):
    request = (await reader.read()).decode('utf8') # should read until end of msg
    print(request)
    response = "thx"
    writer.write(response.encode('utf8'))
    await writer.drain()
    writer.close()


loop = asyncio.get_event_loop()
loop.create_task(asyncio.start_server(handle_client, socket.gethostname(), 8886))
loop.run_forever()

and a small client written like this:

async def tcp_echo_client(message):
    reader, writer = await asyncio.open_connection(
        my_ip, 8886)

    print(f'Send: {message!r}')
    writer.write(message.encode())
    await writer.drain()
    data = await reader.read() # should read until end of msg
    print(f'Received: {data.decode()!r}')
    print('Close the connection')
    writer.close()
    await writer.wait_closed()

asyncio.run(tcp_echo_client(f"Hello World!"))

the client and the server start the comunication but never finish it. why does the reader not recognise end of msg?

if i write

request = (await reader.read(1024)).decode('utf8')

instead it works, but i need to recive undefined large amount of data.

i tried to modify the code of the server like this:

    while True:
        request = (await reader.read(1024)).decode('utf8')
        if not request:
            break

it recieves all data blocks but still waits forever after the last block. why? how do i tell the reader from the server to stop listenig and proceed in the code to send the answer?

Solution

TCP connections are stream-based, which means that when you write a "message" to a socket, the bytes will be sent to the peer without including a delimiter between messages. The peer on the other side of the connection can retrieve the bytes, but it needs to figure out on its own how to slice them into "messages". This is why reading the last block appears to hang: read() simply waits for the peer to send more data.

To enable retrieval of individual messages, the sender must frame or delimit each message. For example, the sender could just close the connection after sending a message, which would allow the other side to read the message because it would be followed by the end-of-file indicator. However, that would allow sender to only send one message without the ability to read a response because the socket would be closed.

A better option is for the writer to only close the writing side of the socket, such partial close sometimes being referred to as shutdown). In asyncio this is done with a call to write_eof:

writer.write(message.encode())
await writer.drain()
writer.write_eof()

Sent like this, the message will be followed by end-of-file and the read on the server side won't hang. While the client will be able to read the response, it will still be limited to sending only one message because further writes will be impossible on the socket whose writing end was closed.

To implement communication consisting of an arbitrary number of requests and responses, you need to frame each message. A simple way to do so is by prefixing each message with message length:

writer.write(struct.pack('<L', len(request)))
writer.write(request)

The receiver first reads the message size and then the message itself:

size, = struct.unpack('<L', await reader.readexactly(4))
request = await reader.readexactly(size)

Answered By - user4815162342