[FIXED] Python method with ProcessPoolExecutor freeze QT gui

Issue

One of buttons in my qt gui starts ProcessPoolExecutor which is making tifs images. I want add process bar but my gui is freezing on time when the event loop is working.

    @pyqtSlot()
    def on_pb_start_clicked(self):
        if self.anaglyph_output != None and self.tmp_output != None:
            progress_value = 1
            self.progress_display(progress_value)
            df = self.mask_df
            df_size = 10
           
            workers = self.cpu_selected
            args = [(i, df[df.fid == i + 1.0], self.anaglyph_output, self.tmp_output) for i in range(df_size)]
            tasks = []

            with ProcessPoolExecutor(max_workers=workers) as executor:
                for arg in args:
                    tasks.append(asyncio.get_event_loop().run_in_executor(executor, create_anaglyph, *arg))

            loop = asyncio.get_event_loop().run_until_complete(asyncio.gather(*tasks))

            print(loop)

        else:
            self.display_no_path_warning()

i dont know what to do

Solution

In this code you are gaining nothing by using async code - and, by using it incorrectly, you are actually blocking the execution that would be parallel in the sub-processes, until all processing is finished.

asyncio in Python allow for concurrent tasks, but then everything ou are running has to be written to be colaborative parallel using the loop. What you are doing is inside a single-threaded callback from Qt, your on_pb_start_clicked, you are starting an asyncio loop, and telling it to wait until all tasks placed in it are ready. This function won't return, and therefore, the Qt UI will block.

Since you are already running a Qt application, you are better of using the Qt mechanisms for concurrency - besides the tasks that offloaded to other processes using a ProcessPoolExecutor.

In this case, since you do not seen to check the return value, you can simply submit all your tasks to the process pool, and return from your function.

If you need the return values, you have to register a callback function with Qt so that it can take the tasks list you create in this function, and call done in the futures inside them, to learn which are ready.

from PyQt5.QtCore import QTimer  # not sure if you are on PyQT5 - just find out how to import QTimer in your bindings

...

    @pyqtSlot()
    def on_pb_start_clicked(self):
        if self.anaglyph_output != None and self.tmp_output != None:
            progress_value = 1
            self.progress_display(progress_value)
            df = self.mask_df
            df_size = 10
           
            workers = self.cpu_selected
            args = [(i, df[df.fid == i + 1.0], self.anaglyph_output, self.tmp_output) for i in range(df_size)]
            # for correctness: check if "self.executor" already exists
            # and is running
            if not hasattr(self, "executor"):
                self.executor = ProcessPoolExecutor(max_workers=workers)
                self.tasks = set()
            for arg in args:
                self.tasks.add(executor.submit(create_anaglyph, *arg))
            self.executor = executor
            QTimer.singleShot(200, self.check_tasks) # callback checking in 200 miliseconds


        else:
            self.display_no_path_warning()
            
    def check_tasks(self):
        for task in self.tasks.copy(): # iterate in a copy of self.tasks to avoid side-effects of modifications on the set while iterating over it
            if task.done():
                result = task.result()  # this is the return value of the off-process execution
                self.tasks.remove(task)
                ... # do things with result
        if not self.tasks:
            self.executor.shutdown()
            del self.executor
        else:
            # if there are tasks in execution, re-schedule the check
            QTimer.singleShot(200, self.check_tasks)

Answered By - jsbueno