[FIXED] python csvwriter gives "BrokenPipeError" when used in pipe

Issue

Here's an example of what I'm seeing:

❯ cat /tmp/test.py
#!/usr/bin/env python3

import csv

with open("/dev/stdout", "w") as f:
    csvwriter = csv.writer(f)
    csvwriter.writerows((_,) for _ in range(10000))

This runs just fine when redirected to a file or to sent to stdout.

❯ /tmp/test.py > /dev/null
❯ echo $?
0

However, when sent to head, and some other tools likely, it raises an exception.

❯ /tmp/test.py | head > /dev/null
Traceback (most recent call last):
  File "/tmp/test.py", line 7, in <module>
    csvwriter.writerows((_,) for _ in range(10000))
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/test.py", line 5, in <module>
    with open("/dev/stdout", "w") as f:
BrokenPipeError: [Errno 32] Broken pipe

If I have something like range(15) then no problem. Question is, is there some setting I can use in python to not have this behaviour?

Solution

From the python docs for SIGPIPE (https://docs.python.org/3/library/signal.html#note-on-sigpipe):

Piping output of your program to tools like head(1) will cause a SIGPIPE signal to be sent to your process when the receiver of its standard output closes early. This results in an exception like BrokenPipeError: [Errno 32] Broken pipe. To handle this case, wrap your entry point to catch this exception as follows:

Adapting their example to your code, you can handle this exception like so:

import os, sys, csv

try:
    with open("/dev/stdout", "w") as f:
        csvwriter = csv.writer(f)
        csvwriter.writerows((_,) for _ in range(10000))
    sys.stdout.flush()
except BrokenPipeError:
    # Python flushes standard streams on exit; redirect remaining output
    # to devnull to avoid another BrokenPipeError at shutdown
    devnull = os.open(os.devnull, os.O_WRONLY)
    os.dup2(devnull, sys.stdout.fileno())
    sys.exit(1)  # Python exits with error code 1 on EPIPE

Answered By - ajc2000