[FIXED] Flask hangs when run as a subprocess alongside pytest

Issue

I've spent the last hour and a half trying and failing to debug this test and I am utterly stumped. To simplify the process of testing the Flask server I am building, I have made a relatively simple script which starts the server, then runs pytest, kills the server, writes the outputs to files, and exits with Pytest's exit code. This code was working perfectly until today, and I haven't modified it since (aside from debugging this issue).

Here's the problem: when it gets to a certain point in the tests, it hangs. The weird thing is that this does not happen if I run my tests in any other way.

• Debugging my server in VS Code, and running tests in the terminal: works
• Running my server using the same code used in the test script and running pytest manually: works
• Running pytest using the test script and running the server through the start server script (which uses the same code for running the server as the test script does) in a second terminal: works

Here's the other interesting thing: the tests always hang in the same place, part way through the setup fixture. It sends the clear command, and an echo request to the server (which prints the name of the current test). The database clears successfully, and the server echoes the correct information, but the echo route never exits - my tests never get a response. This echo route behaves perfectly for the 50 or so tests that happen before this point. If I comment out the test that is causing it to fail, it fails on the next test. If I comment out the call to the echo then it hangs on a later test on a completely different request to a different route. When it hangs, the server cannot be killed using a SIGTERM, but instead requires a SIGKILL.

Here is my echo route:

@debug.get('/echo')
def echo() -> IEcho:
    """
    Echo an input. This returns the given value, but also prints it to stdout
    on the server. Useful for debugging tests.

    ## Params:
    * `value` (`str`): value to echo
    """
    try:
        value = request.args['value']
    except KeyError:
        raise http_errors.BadRequest('echo route requires a `value` argument')

    to_print = f'{Fore.MAGENTA}[ECHO]\t\t{value}{Fore.RESET}'
    # Print it to both stdout and stderr to ensure it is seen across all logs
    # Otherwise it could be more difficult to figure out what's up with server
    # output
    print(to_print)
    print(to_print, file=sys.stderr)
    return {'value': value}

And here is my code that sends the requests:

def get(token: JWT | None, url: str, params: dict) -> dict:
    """
    Returns the response to a GET web request

    This also parses the response to help with error checking

    ### Args:
    * `url` (`str`): URL to request to

    * `params` (`dict`): parameters to send

    ### Returns:
    * `dict`: response data
    """
    return handle_response(requests.get(
        url,
        params=params,
        headers=encode_headers(token),
        timeout=3
    ))

def echo(value: str) -> IEcho:
    """
    Echo an input. This returns the given value, but also prints it to stdout
    on the server. Useful for debugging tests.

    ## Params:
    * `value` (`str`): value to echo
    """
    return cast(IEcho, get(None, f"{URL}/echo", {"value": value}))

@pytest.fixture(autouse=True)
def before_each(request: pytest.FixtureRequest):
    """Clear the database between tests"""
    clear()
    echo(f"{request.module.__name__}.{request.function.__name__}")
    print("After echo")  # This never prints

Here is my code for running Pytest in my test script

def pytest():
    pytest = subprocess.Popen(
        [sys.executable, '-u', '-m', 'pytest', '-v', '-s'],
    )

    # Wait for tests to finish
    print("🔨 Running tests...")
    try:
        ret = pytest.wait()
    except KeyboardInterrupt:
        print("❗ Testing cancelled")
        pytest.terminate()
        # write_outputs(pytest, None)
        # write_outputs(pytest, "pytest")
        raise
    # write_outputs(pytest, "pytest")
    if ret == 0:
        print("✅ It works!")
    else:
        print("❌ Tests failed")
    return bool(ret)

And here is my code for running my server in my test script:

def backend(debug=False, live_output=False):
    env = os.environ.copy()
    if debug:
        env.update({"ENSEMBLE_DEBUG": "TRUE"})
        debug_flag = ["--debug"]
    else:
        debug_flag = []
    if live_output is False:
        outputs = subprocess.PIPE
    else:
        outputs = None
    flask = subprocess.Popen(
        [sys.executable, '-u', '-m', 'flask'] + debug_flag + ['run'],
        env=env,
        stderr=outputs,
        stdout=outputs,
    )
    if outputs is not None and (flask.stderr is None or flask.stdout is None):
        print("❗ Can't read flask output", file=sys.stderr)
        flask.kill()
        sys.exit(1)

    # Request until we get a success, but crash if we failed to start in 10
    # seconds
    start_time = time.time()
    started = False
    while time.time() - start_time < 10:
        try:
            requests.get(
                f'http://localhost:{os.getenv("FLASK_RUN_PORT")}/debug/echo',
                params={'value': 'Test script startup...'},
            )
        except requests.ConnectionError:
            continue
        started = True
        break

    if not started:
        print("❗ Server failed to start in time")
        flask.kill()
        if outputs is not None:
            write_outputs(flask, None)
        sys.exit(1)
    else:
        if flask.poll() is not None:
            print("❗ Server crashed during startup")
            if outputs is not None:
                write_outputs(flask, None)
            sys.exit(1)
        print("✅ Server started")
        return flask

So in summary, does anyone have any idea what on earth is happening? It freezes on such a simple route that this makes me very concerned. I think I may have found some crazy bug in Flask or in the requests library or something.

Even if you don't know what's happening with this, it'd be really helpful to have any ideas as to how I can debug this further, as I have absolutely no idea what is going on.

Solution

It turns out that my server output was filling up all the buffer space in the pipe, meaning that it would wait for the buffer to empty. The issue is that my test script was waiting for the tests to exit, and the tests could not progress unless the server was active. As such, the code reached a three-way deadlock. I fixed it by redirecting my output through a file (where limited buffer size wasn't a problem).

Answered By - Miguel Guthridge