Issue
Backround
I am calling an executable from Python and need to pass a variable to the executable. The executable however expects a file and does not read from stdin.
I circumvented that problem previously when using the subprocess module by simply calling the executable to read from /dev/stdin
along the lines of:
# with executable 'foo'
cmd = ['foo', '/dev/stdin']
input_variable = 'bar'
with subprocess.Popen(
cmd,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
) as process:
stdout, stderr = process.communicate(input_variable)
print(f"{process.returncode}, {stdout}, {stderr}")
This worked fine so far. In order to add concurrency, I am now implementing asyncio and as such need to replace the subprocess module with the asyncio subprocess module.
Problem
Calling asyncio subprocess for a program using /dev/stdin
fails. Using the following async function:
import asyncio
async def invoke_subprocess(cmd, args, input_variable):
process = await asyncio.create_subprocess_exec(
cmd,
args,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
stdin=asyncio.subprocess.PIPE,
)
stdout, stderr = await process.communicate(input=bytes(input_variable, 'utf-8'))
print(f"{process.returncode}, {stdout.decode()}, {stderr.decode()}")
This generally works for files, but fails for /dev/stdin
:
# 'cat' can be used for 'foo' to test the behavior
asyncio.run(invoke_subprocess('foo', '/path/to/file/containing/bar', 'not used')) # works
asyncio.run(invoke_subprocess('foo', '/dev/stdin', 'bar')) # fails with "No such device or address"
How can I call asyncio.create_subprocess_exec on /dev/stdin
?
Note: I have already tried and failed via asyncio.create_subprocess_shell and writing a temporary file is not an option as the file system is readonly.
Minimal example using 'cat'
Script main.py
:
import subprocess
import asyncio
def invoke_subprocess(cmd, arg, input_variable):
with subprocess.Popen(
[cmd, arg],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
) as process:
stdout, stderr = process.communicate(input_variable)
print(f"{process.returncode}, {stdout}, {stderr}")
async def invoke_async_subprocess(cmd, arg, input_variable):
process = await asyncio.create_subprocess_exec(
cmd,
arg,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
stdin=asyncio.subprocess.PIPE,
)
stdout, stderr = await process.communicate(input=input_variable)
print(f"{process.returncode}, {stdout.decode()}, {stderr.decode()}")
cmd = 'cat'
arg = '/dev/stdin'
input_variable = b'hello world'
# normal subprocess
invoke_subprocess(cmd, arg, input_variable)
asyncio.run(invoke_async_subprocess(cmd, arg, input_variable))
Returns:
> python3 main.py
0, b'hello world', b''
1, , cat: /dev/stdin: No such device or address
Tested on:
- Ubuntu 21.10, Python 3.9.7
- Linux Mint 20.2, Python 3.8.10
- Docker image: python:3-alpine
Solution
I'll briefly wrap up the question and summarize the outcome of the discussion.
In short: The problem is related to a bug in Python's asyncio library that has been fixed by now. It should no longer occur in upcoming versions.
Bug Details: In contrast to the Python subprocess
library, asyncio
uses a socket.socketpair()
and not a pipe to communicate with the subprocess.
This was introduced in order to support the AIX platform.
However, it breaks when re-opening /dev/stdin
that doesn't work with a socket.
It was fixed by only using sockets on AIX platform.
Answered By - xoph
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.