[FIXED] Why is multiprocess running outside of the target function?

Issue

I have this code:

import multiprocessing
with open('pairs.txt') as f:
    pairs = f.read().splitlines()

print(pairs)


def worker(pairtxt):
    print(pairtxt)

if __name__ == '__main__':
    jobs = []
    for i in pairs:
        p = multiprocessing.Process(target=worker, args=(i,))
        jobs.append(p)
        p.start()

When I run this, it outputs pairs variable(not pairtxt) 3 times, I'm guessing once by itself and twice by the multiprocessing but why is it even running outside the target function?

My output I'm getting with pairs.txt containing 2 lines "1" and "2"

['1', '2']
['1', '2']
1
['1', '2']
2

Solution

Try moving the with open and print(pairs) statements into your if __name__ == '__main__' block.

I suspect that python is running the full script every time the subprocess is called, as it wants to ensure that all of the dependencies are met (imports and such) for the function that you hand it. By having running code in your script that's outside of your main block, it's running that every time your subprocess is run as well.

import multiprocessing
def worker(pairtxt):
    print(pairtxt)

if __name__ == '__main__':
    with open('pairs.txt') as f:
        pairs = f.read().splitlines()

    print(pairs)
    jobs = []
    for i in pairs:
        p = multiprocessing.Process(target=worker, args=(i,))
        jobs.append(p)
        p.start()

Answered By - AetherUnbound