[FIXED] Python multiprocessing No file descriptors available error inside docker alpine

Issue

OSError: [Errno 24] No file descriptors available while creating multiprocessing queue. It only occurs when using alpine image (FROM python:alpine3.19) normal image (FROM python) works fine. I have tried incressing ulimit. Default docker setting:

"default-ulimits": {
    "nofile": {
      "Hard": 4096,
      "Name": "nofile",
      "Soft": 4096
    },
    "nproc": {
      "Hard": 4096,
      "Name": "nproc",
      "Soft": 4096
    }
  }

And also during 'run' command see below. Ulimit setting seems to work fine (it is visible in /proc/1/limits). Is there some other limit that I am not aware of?

Steps to reproduce:

Dockerfile:

# FROM python
FROM python:alpine3.19
WORKDIR test
COPY . .
CMD [ "python", "test.py"]

test.py:

import multiprocessing
import time
import subprocess

if __name__ == '__main__':
    queues = []
    print((subprocess.check_output(['cat', '/proc/1/limits'])).decode())
    for i in range(1000):
        print(f"Appending multiprocessing.Queue() n. {i} ", end="")
        queues.append(multiprocessing.Queue())
        print("ok")
        time.sleep(0.1)
    print("all ok")

(docker build -t test .)

Output (FROM python):

• Test low nofile limit
• works as expected
• hitting limit at about 128/2 queues

docker run -it --rm --ulimit nofile=128 test

Appending multiprocessing.Queue() n. 62 Traceback (most recent call last):
  File "/test/test.py", line 10, in <module>
    queues.append(multiprocessing.Queue())
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/multiprocessing/context.py", line 103, in Queue
    return Queue(maxsize, ctx=self.get_context())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/multiprocessing/queues.py", line 42, in __init__
    self._reader, self._writer = connection.Pipe(duplex=False)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/multiprocessing/connection.py", line 543, in Pipe
    fd1, fd2 = os.pipe()
               ^^^^^^^^^
OSError: [Errno 24] Too many open files

• Test high nofile limit
• works beautifully

docker run -it --rm --ulimit nofile=4096 test

Appending multiprocessing.Queue() n. 998 ok
Appending multiprocessing.Queue() n. 999 ok
all ok

Output (FROM python:alpine3.19):

• Test low nofile limit
• works as expected
• hitting limit at about 128/2 queues

docker run -it --rm --ulimit nofile=128 test

Appending multiprocessing.Queue() n. 62 Traceback (most recent call last):
  File "/test/test.py", line 10, in <module>
    queues.append(multiprocessing.Queue())
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/multiprocessing/context.py", line 103, in Queue
    return Queue(maxsize, ctx=self.get_context())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/multiprocessing/queues.py", line 42, in __init__
    self._reader, self._writer = connection.Pipe(duplex=False)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/multiprocessing/connection.py", line 543, in Pipe
    fd1, fd2 = os.pipe()
               ^^^^^^^^^
OSError: [Errno 24] No file descriptors available

• Test high nofile limit
• Also fails
• What's wrong here?

docker run -it --rm --ulimit nofile=4096 test

Appending multiprocessing.Queue() n. 85 Traceback (most recent call last):
  File "/test/test.py", line 10, in <module>
    queues.append(multiprocessing.Queue())
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/multiprocessing/context.py", line 103, in Queue
    return Queue(maxsize, ctx=self.get_context())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/multiprocessing/queues.py", line 48, in __init__
    self._wlock = ctx.Lock()
                  ^^^^^^^^^^
  File "/usr/local/lib/python3.12/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/multiprocessing/synchronize.py", line 169, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/usr/local/lib/python3.12/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 24] No file descriptors available

Edit:

• I am running Docker Engine v24.0.7 from Windows 11
• It behaves the same on Centos 8 (Docker version 24.0.7, build afdd53b)
• I also tried:
• multiprocessing.Value(c_int, 0) -> Fails at 256. iteration.
• multiprocessing.Lock() -> Fails at 256. iteration.
• multiprocessing.Semaphore(1) -> Fails at 256. iteration.
• threading.Semaphore(1) -> Does not fail.
• (multiprocessing.Pipe()) -> Does not fail.
• multiprocessing.SimpleQueue() -> Fails at 128. iteration.
• multiprocessing.Condition() -> Fails at 64. iteration.

Solution

The issue is seems related to alpine linux using musl, while debian uses glibc: It is not actually file descriptors lacking, but POSIX-sem_open can raise EMFILE too if the limit of semaphores is exceeded.

As all listed multiprocessing-classes use semaphores internally, this limit is eventually reached (earlier for classes using multiple semaphores).

The limit is given as SEM_NSEMS_MAX and specified as 256. The code actually raising EMFILE is in sem_open.

Huge thanks to @psykose from #musl for pointing this out to someone new to C like me.

So why is it working on glibc / debian then?

Generally, the glibc-implementation of sem_open does not allow for EMFILE to happen, so the issue simply cannot occur using glibc (unless of course system resources are exhausted). Some further information is available on the libc-help mailinglist.

While I posted about the problem on the musl-mailinglist, I really do not know how much work a change would be and what the priority would be. So for the moment, the most sensible way is probably to use a different distribution which is using glibc for now.

Edit: Another solution would of course be python changing the semaphore-mechanism to in-memory-semaphores, if this is possible. I added a discussion point at the Python Discourse.

Answered By - Patrick Rauscher