[FIXED] Python Flask background threading is not working: code is not being executed

Issue

Context:

I have a function in my views command where some varaibales is being send through a background task script after a HTTP post on form. This background script handles a lot of API calls and is converting this to JSON that is gonna be converted Because this can take a long time (because of the way the API Calls) go before the next html page is rendered I had decided to make this one a background task with threading. Eventually if possible I would like a que to be added later on.

---

The problem:

But the thing is even though I clearly set it to a daemon thread and everything. The code is not being executed. I the Python Console log does not even show the code being executed, so what I am doing wrong here.

---

Just look at Order 66 if you want the background task itself.

"""
Routes and views for the flask application.
"""

from datetime import datetime
from flask import render_template, jsonify
from FlaskWebPODTracer import app
import json
from itsdangerous import TimedJSONWebSignatureSerializer as Serializer
from flask import request, redirect, url_for, flash
# The main script that has to happen in the background.
import classified.background_script.backgroundtask as Order66
from threading import Thread
import random
import string
from celery import Celery
from flask_mail import Mail, Message
import requests




API_VERSION = "v1/"
Succes_message = "U ontvangt nog een mail wanneer de data klaar is. Bedankt om PODTracer te gebruiken."
Fail_message_1 = "Wijzig de parameters van uw aanvraag en probeer opnieuw."
Fail_message_2 = "Blijft dit voorkomen contacteer dan support."
Fail_message = Fail_message_1 + '\n' + Fail_message_2

@app.route('/handle_data/')
@app.route('/handle_data/', methods=['POST', 'GET'])
def handle_data():
    #if request.method == 'GET':
    #    return render_template(
    #            "request-completed.html",
    #            title=
    #            "Aanvraag ingediend",
    #            objects = jsonobjects,
    #            year=datetime.now().year,
    #            message='Uw aanvraag is voltooid.'
    #            )
    if request.method == 'POST':
        email = request.form['inputEmail']
        stringpart = request.form['city']
        sper_year = int(request.form["Speryear"])
        urlpart = request.form["api-url-input"]
        url = "Classified" + API_VERSION + urlpart
        print(url)
        response = requests.get(url)
        if response.status_code == 200:
            jsonobjects = len(response.json())
            task = Thread(group=None, target=None,name=Order66.main, args=(stringpart,url,sper_year,email), daemon=True)
            task.start()
            state = "succesvol."
            message_body = Succes_message

        else:
            jsonobjects = 0;
            state = "onsuccesvol."
            message_body = Fail_message
        return render_template(
                "request-posted.html",
                respstate = state,
                body = message_body,
                title=
                "Aanvraag ingediend",
                objects = jsonobjects,
                year=datetime.now().year,
                message='Uw aanvraag is ' + state
                )
        # TODO RUN THIS IN BACKGROUND
        # app.route('/request-completed')

@app.route('/handle_data_fail')
def handle_data_fail():
    jsonobjects = 0
    state = "onsuccesvol."
    message_body = Fail_message
    return render_template(
        "request-posted.html",
        respstate = state,
        body = message_body,
        title= 
        "Aanvraag ingediend",
        objects = jsonobjects,
        year=datetime.now().year,
        message='Uw aanvraag is ' + state
                )

Solution

As discussed in comments, here's an oversimplified example of an event-driven system with RabbitMQ and Flask.

Dependencies you need:

(flask-rabbitmq) ➜  flask-rabbitmq pip freeze
click==8.0.3
Flask==2.0.2
itsdangerous==2.0.1
Jinja2==3.0.3
MarkupSafe==2.0.1
pika==1.2.0
Werkzeug==2.0.2

Try to create a RabbitMQ docker container with the command below:

docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.9-management

your simple flask app would look like this:

from flask import Flask
from flask import request
import json
from RabbitMQPublisher import publish

app = Flask(__name__)

@app.route('/publish', methods=['POST'])
def index():
    if request.method == 'POST':
        publish(json.dumps(request.json))
        return 'Done'
    

if __name__ == '__main__':
      app.run()

your RabbitMQPublisher.py would be like this:

import pika

def publish(data: str):
    connection = pika.BlockingConnection(
        pika.ConnectionParameters(host='0.0.0.0', port=5672))
    channel = connection.channel()
    channel.exchange_declare(exchange='test', exchange_type='fanout')
    channel.queue_declare(queue='', exclusive=True)
    channel.basic_publish(exchange='test', routing_key='', body='{"data": %s}'%(data))
    connection.close() 

and finally your script.py would be like this:

import pika
import json
from time import sleep

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='0.0.0.0', port=5672))
channel = connection.channel()

channel.exchange_declare(exchange='test', exchange_type='fanout')

result = channel.queue_declare(queue='', exclusive=True)

channel.queue_bind(exchange='test', queue='')


def callback(ch, method, properties, body):
    body = json.loads(body.decode().replace("'", '"'))
    print(body)
    


channel.basic_consume(
    queue='', on_message_callback=callback, auto_ack=True)

channel.start_consuming()

inside the callback function in the code above, you can specify you're logic and when your logic finishes you can again use RabbitMQ to call a pattern on flask side or event do http call with requests. that would be your choice.

Answered By - Hossein Heydari