[FIXED] Whats the best way to handle tasks that need to be run concurrently?

Issue

I've been writing a python agent that monitors linux. Originally everything stemmed from a single function that when into a loop collecting metrics etc. But since then, i've added some other features that need to run separate to that loop in its own loop.

For example, i have a main metrics loop, but i also have the agent listening on two ports for different types of messages (although now that i think about it i could bring this down to a single port). I was using multiprocess to run these different loops, but ran into the issue of sharing vars between processes. There are solutions to this, but its just starts to look messy. I also started questioning why i need so many processes (depending on the role, i can have up to 5 processes running). Anyways, you can probably tell im not a seasoned python developer (I'm not an actual developer).

What should i be doing here? should i be using threads instead, since they have shared memory space? Whats the most common/generally accepted way of doing this?

Solution

The most common way or generally accepted way of doing this is by using global queues to share information between threads otherwise. a Very simple way of doing this is that you launch different classes within your main program in different threads after creating global queues for specific info you want to communicate over them. Have a look below:

import queue as Queue
import threading
import time

class part_1:
    def __init__(self) -> None:
        global q_input
        self.program_switch = 4

        while self.program_switch < 5:
            print('Waiting in the while loop')
            input_value = q_input.get()
            if input_value == 1:
                self.my_function()
            #stuff here running on separate thread

    def my_function(self):
            print('Stopped by input from another function')
            self.program_switch = 6
            pass
        
class part_2:
    def __init__(self) -> None:
        global q_input
        global q_output
        #stuff here running on separate thread
        value = input('Enter value 1 to stop the loop')
        q_input.put(int(value))
        print("First function will stop first but the starter function will stop late")
        time.sleep(6)
        print("it must stop late to show it is running independently")
        pass

q_input = Queue.Queue(maxsize=1)
# Put something on it as if some part tries to get them they wouldn't crash
q_input.put(0)

# Start in seperate threads first run part_2 class and then part_1
t_2 = threading.Thread(target=part_2)
t_1 = threading.Thread(target=part_1)
t_2.start()
t_1.start()

It is a good template to start experimenting with. if someone has a better approach to this please post that or link that. I am a beginner too. :) I hope it helps.

Answered By - Fahim Chohan