[FIXED] Multi Threading python 3.7

Issue

I'm trying to optimize a script that connects to some routers based on a list that I provide with IPs.

however, it's taking a quite long time to finish the configuration once starts the configuration once when it finishes the previous router, my question is, would be possible to create a function that doesn't need to wait for it, maybe a multithreading function that can configure a few elements simultaneously.

def configure_routers():

    hosts = create_vrdc_dic()  # -> it's the dictonary that pass the variables

    for i in hosts .keys():
        hostname = hosts [i].get('hostname')
        mgmt_ip = hosts [i].get('mgmt_ip')

        console_config.config_vrdc(hostname, mgmt_ip)  # -> here I'm calling another module that configures the routers using pexpect

Thanks for any help here.

Solution

To configure your routers simultaneously, you first need to make sure it is safe to do so with the module/library you are using. If only one instance should be running at a time, you will just have to wait for each router to be configured one by one. For example, if the process you are using reads/writes to the same file/database while running and you multi-thread that process, you can end up with a corrupted file/database.

If you determine that your operation is safe to multi-thread, try something like the following code:

import threading


# Create a class for multi-threading the configuration of a router
class RouterConfigure(threading.Thread):
    def __init__(self, ip: str, hostname: str):
        threading.Thread.__init__(self)
        self.ip = ip
        self.hostname = hostname

    def run(self):
        console_config.config_vrdc(self.hostname, self.ip)


threads = []
hosts = create_vrdc_dic()

# Create a thread for each IP/hostname in the dict
for i in hosts.keys():
    hostname = hosts[i].get('hostname')
    ip = hosts[i].get('mgmt_ip')
    
    thread = RouterConfigure(ip=ip, hostname=hostname)
    thread.start()
    threads.append(thread)

# Wait for threads to finish before exiting main program
for t in threads:
    t.join()

If there are more hosts to configure than your system can handle with multiple threads, figure out how to split the process up into chunks so that only x threads are running at the same time.

Answered By - Noah Broyles