[FIXED] True Concurrency in Python

Issue

I am relatively new to concurrency in Python and I am working on some code that has to call functions outside of my code. I cannot edit those functions but I need them to run concurrently. I've tried a few different solutions like Multiprocessing, Threading and AsyncIO. AsyncIO comes the closest to what I want if every function I was calling was defined with it, but they're not.

The functions I'm calling will block. Sometimes for 15-30 minutes. During that time, I need other functions doing other things. The code below illustrates my problem. If you run it you'll see that whether using Threads or Multiprocesses, the tasks always run serially. I need them to run simultaneous to each other. I get that the output blocks until the entire script runs, but the tasks themselves should not.

What am I missing? With so many choices for concurrency or at least apparent concurrency in Python, I would think this is easier than I'm finding it.

#!/usr/bin/python3

from datetime import datetime
from multiprocessing import Process
import sys
from threading import Thread
from time import sleep


def main():

    # Doing it with the multiprocess module
    print("Using MultiProcess:")
    useprocs()

    print("\nUsing Threading:")
    usethreads()


def useprocs():
    procs = []
    
    task1 = Process(target=blockingfunc('Task1'))
    task1.start()
    procs.append(task1)

    task2 = Process(target=blockingfunc('Tast2'))
    task2.start()
    procs.append(task2) 

    task1.join()
    task2.join()
    
    print('All processes completed')


def usethreads():
    threads = []

    task3 = Process(target=blockingfunc('Task3'))
    task3.start()
    threads.append(task3)

    task4 = Process(target=blockingfunc('Task4'))
    task4.start()
    threads.append(task4)   

    task3.join()
    task4.join()

    print('All threads completed')


def blockingfunc(taskname):
    now = datetime.now()
    current_time = now.strftime("%H:%M:%S")
    print(current_time, "Starting task: ", taskname)
    sleep(5)
    now = datetime.now()
    current_time = now.strftime("%H:%M:%S")
    print(current_time, taskname, "completed")


if __name__ == '__main__':
    try:
        main()
    except:
        sys.exit(1)

Solution

Note that the program you posted imports Thread but never uses it.

More importantly, in a line like:

task1 = Process(target=blockingfunc('Task1'))

you're calling blockingfunc('Task1') and passing what it returns (None) as the value of the target argument. Not at all what you intended. What you intended:

task1 = Process(target=blockingfunc, args=['Task1'])

Then, as intended, blockingfunc isn't actually invoked before you call the start() method.

Answered By - Tim Peters