[FIXED] How can I organize healthcheck for my workers (Docker with python code)?

Issue

In my project I have a REST API (Flask-server) and workers (python script that work in Docker containers - simple infinite cycle, make request to server and get work). Flask-server send tasks to workers.

But of course something might go wrong and Docker container may fall. And at that moment I want server to know about it. Can you offer a simple way to do that?

Solution

It's quite easy to do this type of thing with Redis which is a very light-weight, lightning fast, in-memory data-structure server. It can serve lists, hashes, sets, ordered sets, integers, atomic integers, strings and so on all across your network with clients for bash, Python, PHP, C/C++ and everything else.

I would use a simple KEY and set an EXPIRE on the key, say 10s. Then each server and each client in your network simply SETs the KEY named by its hostname (or function) at least every 10s which will reset its timeout. If anyone queries the key and it hasn't been set for 10s they know that client/server or host/process has died.

Documentation is here.

---

Another, even simpler way to communicate is via the filesystem. Each worker must "touch" (create) a file called /tmp/WORKERXXX.alive every N seconds. The process that wants to check the workers are alive then checks the files exist every N+1 seconds and deletes them. If a file doesn't exist, the worker is restarted.

Touching the file doesn't have to be intrusive within your worker's code. It can simply create an extra thread at start-up which runs an infinite loop, sleeping N seconds and then touching its keep-alive file.

Answered By - Mark Setchell