[FIXED] AWS Django NGINX timeout and async errors

Issue

I have a view in which I try to upload an excel sheet via a form. The excel that I upload isn't that big but the processing of the data in that excel is causing nginx time out issues resulting in a 502 Bad Gateway error. I do not have this problem when testing locally but it pops up on AWS EC2 when testing it on that environment.

I increased the time out periods in nginx via (as suggested here):

proxy_read_timeout 300s;
proxy_connect_timeout 300s;

proxy_buffer_size          128k;
proxy_buffers              4 256k;
proxy_busy_buffers_size    256k;

And that did improve things somewhat since I can see more results but it was not sufficient. Curiously it timed out again before the 300s were up. Incidentally, if it used all 300s it should complete the process.

It was also suggested I use async to make the processing independent of the upload process and that it run the processing in the background. However in attempting I keep running into this error:

"object HttpResponseRedirect can't be used in 'await' expression"

This error occurs when I try to load the page even before I try to upload the excel file which is the only time I have a redirect, which is meant to redirect to a page where the uploaded documents are listed.

If I remove the @sync_to_async decorator from the upload view I instead get this error

"didn't return an HttpResponse object. It returned an unawaited coroutine instead. You may need to add an 'await' into your view"

How can I resolve this?

@sync_to_async
def process_excel_upload(request, excel_file):
    ** processing of the excel document ***
    return ("complete")

@login_required
@sync_to_async
@user_preapproved(home_url='landing-dashboard', redirect_field_name='', message='You are not authorised to perform this action.')
@user_passes_test(group_check, redirect_field_name=None)
async def ExcelUploadEditView(request):
    user = request.user
    member = ProfileMember.objects.get(user=user)
    if request.method == 'POST':
        form = ExcelUploadForm(request.POST or None, request.FILES or None)
        if form.is_valid():            
            json_payload = {
                "message": "Your file is being processed"
            }
            excel = request.POST.get('excel')
            await asyncio.create_task(process_excel_upload(request,excel))
            form.save()
            return redirect('excel-after',json_payload)
        else:
            print(form.errors)
    else:
        form = ReportUploadForm()
        context = {
            'form_documentupload': form,
            'user': user,
            'member':member,

            }
        return render(request, "excel_upload.html", context)

Solution

This might work for you:

[views.py]

    import threading
    
    def process_excel_upload(request, excel_file):
        ** processing of the excel document ***
        upload_status = UploadStatus.objects.get(user=request.user)
        upload_status.status = 'complete'
        upload_status.save()
    
    def get_upload_status(request):
        upload_status = UploadStatus.objects.get(user=request.user)
        return JsonResponse({'status': upload_status.status})
    
    @login_required
    @user_preapproved(home_url='landing-dashboard', redirect_field_name='', message='You are not authorised to perform this action.')
    @user_passes_test(group_check, redirect_field_name=None)
    def ExcelUploadEditView(request):
        user = request.user
        member = ProfileMember.objects.get(user=user)
        if request.method == 'POST':
            form = ExcelUploadForm(request.POST or None, request.FILES or None)
            if form.is_valid():            
                json_payload = {
                    "message": "Your file is being processed"
                }
                excel = request.POST.get('excel')
                upload_status = UploadStatus.objects.get(user=request.user)
                upload_status.status = 'processing'
                upload_status.save()
                t = threading.Thread(target=process_excel_upload,args=(request,excel))
                t.setDaemon(True)
                t.start()
                form.save()
                return redirect('excel-after',json_payload)
            else:
                print(form.errors)
        else:
            form = ReportUploadForm()
            context = {
                'form_documentupload': form,
                'user': user,
                'member':member,
                }
            return render(request, "excel_upload.html", context)

[template.html]

    //call this function in the upload response:
    
    function checkUploadStatus() {
        $.ajax({
            url: '/get_upload_status/',  // replace with the URL you set to your get_upload_status view
            success: function(data) {
                if (data.stats === 'complete') {
                    //replace the uploading... message with a completion message
                } else {
                    setTimeout(checkUploadStatus, 3000);
                }
            }
        });
    }

[models.py]

    from django.contrib.auth.models import User
    
    class UploadStatus(models.Model):
        user = models.OneToOneField(User, on_delete=models.CASCADE, primary_key=True)
        status = models.CharField(max_length=20, default='pending')

You may also consider using web sockets, tho I believe is an overkill

Answered By - 42WaysToAnswerThat