[FIXED] Nginx/Uwsgi/Flask POST times out if body is too large

Issue

I'm using a dockerimage based on https://github.com/tiangolo/uwsgi-nginx-flask-docker/tree/master/python3.6. I am running a python app inside that accepts a POST, does some processing on the json body, and returns a simple json response back. A post like this:

curl -H "Content-Type: application/json" -X POST http://10.4.5.168:5002/test -d '{"test": "test"}'

works fine. If, however, I post a larger json file, I get a 504: Gateway Timeout.

curl -H "Content-Type: application/json" -X POST http://10.4.5.168:5002/test -d @some_6mb_file.json

I have a feeling that there is an issue with the communication between Nginx and Uwsgi, but I'm not sure how to fix it.

EDIT: I jumped inside the docker container and restarted nginx manually to get better logging. I'm receiving the following error:

2018/12/21 20:47:45 [error] 611#611: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.4.3.168, server: , request: "POST /model/refuel_verification_model/predict HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock", host: "10.4.3.168:5002"

From inside the container, I started a second instance of my Flask app, running without Nginx and Uwsgi and it worked fine. The response took approximately 5 seconds to be returned (due to the processing time of the data.)

Configurations:

/etc/nginx/nginx.conf:

user  nginx;
worker_processes 1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}
daemon off;

/etc/nginx/conf.d/nginx.conf:

server {
    listen 80;
    location / {
        try_files $uri @app;
    }
    location @app {
        include uwsgi_params;
        uwsgi_pass unix:///tmp/uwsgi.sock;
    }
    location /static {
        alias /app/static;
    }
}

/etc/nginx/conf.d/upload.conf:

client_max_body_size 128m;
client_body_buffer_size 128m;

Solution

There was an issue with Tensorflow. I had loaded a tensorflow model during the app initialization and then tried to use it later. Because of the threading done by the webserver and the "non-thread-safe" nature of Tensorflow, the processing hung leading to the timeout.

Answered By - Brandon Schabell