Issue
I have an issue with my (dockerized) application consuming an ever increasing amount of memory until the Linux kernel kills is. In the container's docker logs
I only got an ominous Killed
message without further context - only after checking the kernel logs (cat /var/log/kern.log
) and docker stats
did I realize what was happening - memory usage was going up by ~10MB per second.
The application is an asyncronous grpc
-server with several concurrent tasks (using return ensure_future(self._continuous_function)
to keep the function's task up and running, async so the server endpoints are not blocked).
Solution
I found out rather quickly that ensure_future()
caused by memory leak issue - or rather the return
I used with it. Apparently the return meant that a reference to the original task was being kept in memory (instead of being garbage collected), and the function only had a very short wait time associated with it (1ms) - so memory kept building up fast.
So after removing the leak my app now consumes about 60MB of very constant memory. I hope this helps somebody else out, I couldn't find a reference to this exact issue which is why I'm sharing it here.
Answered By - wiseboar
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.