Issue
I keep running into the bellow error and can't work out how to fix it or whether it is a bug in asyncio
that I need to report. My program works fine is it basically the same as the example in there docs but it can send lots of messages upto 20 per second to multiple clients when there is a low number of updates it works fine but when the number of writes increases I run into this error.
I am using create_server
from asyncio to make the server listen for new clients and I believe this creates a new connection on each connection. Not all clients stop only one clients stops and the rest keep running it's usually the last client that connected but not always. I have played with the asyncio/sslproto.py
file removing the reference to [0]
and replacing them with popleft
which they got me the not very helpful SSL: BAD_LENGTH
error which I have had before and using asyncio was supposed to be the fix.
When I logged the value of self._write_backlog
it returned the deque as expected and before the error there was clearly data in it and I believe that self._write_backlog[0]
should return the leftmost element in the deque which there appeared to be one.
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f267462e780>
transport: <_SelectorSocketTransport fd=38 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
File "/usr/local/lib/python3.7/asyncio/sslproto.py", line 689, in
_process_write_backlog
del self._write_backlog[0]
IndexError: deque index out of range
On further investigation the same error also seems to happen at a diffrent point in the code but it is the same error like it's trying to access an empty deque.
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f45f802ec88>
transport: <_SelectorSocketTransport fd=29 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
File "/usr/local/lib/python3.7/asyncio/sslproto.py", line 664, in _process_write_backlog
data, offset = self._write_backlog[0]
IndexError: deque index out of range
EDIT
The version of python I am running on is 3.7.1 but I have also tried 3.7.3 which gave the same error.
UPDATE
It is believed that this issue is caused by an incompatibility between asyncio and OpenSSL https://bugs.python.org/issue37226
UPDATE 2
A minimal example is available on GitHub
Solution
The problem here is related to system resources when you connect lots of clients it can cause CPU or Memory to become exhausted which then causes this error as the write fails.
The solution is to implement flow control to limit the amount of data being sent at one time and stop resources becoming exhausted the following article has a good example of simple flow control that blocks while old data is removed then resumes using the high and low water marks. When the data exceeds the high water mark it triggers pause_writing then when the data size reduces to lower than the low water mark writing resumes this can cause a noticeable slow down dependent on load.
As an alternative to adding your own flow control to a protocol the asyncio Streams primitives offer a simple way to write and read data that includes flow control by default but for me made implementation harder so I opted to include my own at the protocol level as shown in the linked article.
Answered By - bobthemac
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.